3

I would like to automatically crawl about 1 million URLs (on a CSV file) and check whether the sites provide a Facebook Login button (i.e., users can create accounts by connecting their Facebook account to the site). Then, I would like to record for which sites these buttons were found.

I would also like to check whether the site uses HTTPS and makes use of cookies.

What are good tools?

Improve this question

3 Answers 3

Reset to default
3

I would suggest python possibly plus the scrapy library.

  • Free & Open Source
  • Python comes with a CSV library so you can quickly read and parse your csv file
  • Python has several url and web access libraries that can do what you need
  • Scapy simplifies getting and parsing the web pages.
Improve this answer
6
1

If you need to crawl certain website for links, you may try sitechecker web crawler or screaming frog. They have a convenient interface and many useful functions: 1) website monitoring 2) detecting certain URLs 3) explore internal links and their anchors

Improve this answer
1

Do you need tools for automatically crawling websites, check for certain links, and extract data? You may contact with Essayhub Its a trusted service provider where you will get service.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.