4

I am looking for a tool similar to W3C's link checker that will display any broken links BEHIND A LOGIN WALL.

Currently, I login and open Chrome's developer console to inspect the errors. This is both tedious and error prone. I would love to pass some credentials to a service and have it crawl the site regularly.

Does anyone know of such a service? My initial use will be to scan for broken image links on an internal company storefront.

  • It has to be a hosted Web app, or would a desktop client or a self-hosted Web app be okay, too? – unor Dec 14 '15 at 14:03
1

First a cautionary word you should ideally ask the site admin for a user name and password for a test user so as not to require the use of your own username & password in the test scripts. Alternatively, if the site allows registration you should register a test username, password & email.

Once you have done that you can set up walking the site and checking the links using python and either scrapy or possibly Requests. Both can handle working on sites that require logging in.

I would recommend prompting the user for the username and password, (once), and storing them in environment variables for the duration of the session might be a good security practice.

Examples of scraping a site that requires a login

Scrapy

From the documentation.

import scrapy

class LoginSpider(scrapy.Spider):
    name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']

    def parse(self, response):
        return scrapy.FormRequest.from_response(
            response,
            formdata={'username': 'john', 'password': 'secret'},
            callback=self.after_login
        )

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.logger.error("Login failed")
            return

        # continue scraping with authenticated session...

Requests

r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
# You now need to check for a successful login and start to walk the site checking the links.

The other cautionary is that some sites rely heavily on JavaScript to (re-)generate links - in this case you will need to use PhantomJS and Selenium, to generate then parse those links.

  • 1
    Java !== Javascript – a CVn Oct 10 '16 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.