2

We have just changed our website over to using ssl everywhere, but this is resulting in some broken embedded content in browsers.

Is there a tool I can use to crawl the whole site looking for this kind of broken content (mostly video) so I can fix/replace? There's enough content that doing this manually would be a slog. I already use linkchecker occasionally to find 404ing links and such, but this doesn't find broken embeds (afaik).

  • 1
    Can you expand on what you mean by broken embedded content in browsers? – Tom Sep 10 '16 at 22:57
  • Videos for the most part, that the browser won't load due to being from a non ssl url. Ends up leaving a blank area on the page. – feroxy Sep 10 '16 at 23:48
  • Di dyou try Xenu's link sleuth? It has ben around for decades and works well fo rme - home.snafu.de/tilman/xenulink.html – Mawg Sep 14 '16 at 8:26
  • Are you using a CMS to power the website? For example, WordPress, If you are maybe look for a tool written for that CMS to scan the database and change the protocol for you? E.g. https://interconnectit.com/products/search-and-replace-for-wordpress-databases/. – essexboyracer May 14 '17 at 4:34
  • Did you ever find a solution? – Mawg Nov 13 '18 at 12:59
1

I suggest to try crawling the site with Screamingfrog.

1

I'd check your webserver log files, depending on what format (ie, Apache common log format) you should be able to see status code (look for 404), the ask-for-but-unfound resource, and the referring URL (what page teh link/embed was on)

0

If all of your content is supposed to be https or ssl then you should be able to crawl the site with Scrapy and check for links to on site but not the correct type with Beautiful Soup 4.

Just use Scrapy to find all of the pages, parse with BS4 and check the embed elements.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.