3

I would like to automatically crawl about 1 million URLs (on a CSV file) and check whether the sites provide a Facebook Login button (i.e., users can create accounts by connecting their Facebook account to the site). Then, I would like to record for which sites these buttons were found.

I would also like to check whether the site uses HTTPS and makes use of cookies.

What are good tools?

2 Answers 2

3

I would suggest python possibly plus the scrapy library.

  • Free & Open Source
  • Python comes with a CSV library so you can quickly read and parse your csv file
  • Python has several url and web access libraries that can do what you need
  • Scapy simplifies getting and parsing the web pages.
6
  • 1
    much as I dislike "code it yourself answers" meta.softwarerecs.stackexchange.com/questions/1201/… I have to agree that there is unlikely to be a tool which does what the OP wants. However, if he is looking for a programming solution then he should be asking on programmers.stackexchange.com An interesting question, though.
    – Mawg
    Commented May 18, 2015 at 9:17
  • OP, if you are not a coder and you can spare $168, you might want to look at fminer.com
    – Mawg
    Commented May 18, 2015 at 9:58
  • Or, you might pay someone to develop it cheaply, as it is relatively simple. See fiverr.com/categories/programming-tech and similar
    – Mawg
    Commented May 18, 2015 at 10:00
  • Thanks a lot for the suggestion, guys. I will try out scrapy/python first.
    – You_got_it
    Commented May 18, 2015 at 14:07
  • Of course, if you can code Python, then, in addition to scrapy, there is also Beautiful Soup crummy.com/software/BeautifulSoup You can google for comparisons. Thee is a pretty good comparison at quora.com/… Bottom line, " Scrapy has steeper learning curve than Beautiful Soup, but it does have more features. Beautiful Soup can be limited by your designated URL unless you set up an infinite loop providing certain criteria manually".
    – Mawg
    Commented May 18, 2015 at 15:17
1

If you need to crawl certain website for links, you may try sitechecker web crawler or screaming frog. They have a convenient interface and many useful functions: 1) website monitoring 2) detecting certain URLs 3) explore internal links and their anchors

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.