I'm currently using PHP DOM with CURL and Simple HTML DOM Parser for HTML scraping. I need to scrape large set of contents and it need to stored. I am using PHP as my main language and MySQL as storage DB however I am interested in options in any language and can integrate it into my workflow.

Are there any other HTML scraping packages I should look into? I have heard about web scraping libraries named 'Scrapy' and 'Beautiful Soup' which are based on Python.

Suggestions are highly appreciated.! :)

  • stackoverflow.com/questions/2861/options-for-html-scraping as a useful reference for posters. Mar 5 '14 at 12:40
  • Downvote? Please explain. This question seems like a perfect fit for this new StackExchange site. Mar 5 '14 at 20:06
  • @BasilBourque I didn't downvote, but when I do, I don't always feel the need to explain it. But I can imagine people downvote this because it's not easy to understand what's being asked. How much is a large set? Terabytes? Hundreds, millions or billions of items? Is it for PHP or Python (apparently everything is fine?). What HTML scraping library features are important? Javascript support? Performance?
    – Olli
    Mar 6 '14 at 0:29

jsoup – Java Library for Fetching & Parsing

For fetching and parsing HTML, I use the Java-based library, jsoup. It manages amazingly well with mangled mal-formed HTML code.

Has a variety of ways to query, to answer questions like "Give me the third table in the document".

From the library’s web page:

jsoup is a Java library for working with real-world HTML. It provides a very convenient API for extracting and manipulating data, using the best of DOM, CSS, and jquery-like methods.

  • Which DB are you using ? Mar 5 '14 at 12:47
  • @SurabhilSergy DB? You mean database? Database is irrelevant here. Your question was about scraping tools. My answer was jsoup which is a Java library that (a) fetches a web page, (b) parses that page’s HTML, and (c) enables you to query for pieces of that HTML content in various ways. Read Wikipedia for meaning of "web scraping". Mar 5 '14 at 12:52
  • Thanks the answer. I just asked for database recommendations in case of storing the large set of scraped data. Mar 5 '14 at 12:57
  • 2
    A database recommendation is a separate question. It’s a worthy question, but it’s not this question. You should make a new Question on this site asking about Database for storing large sets of data from web scraping. StackExchange is about specific questions with specific answers, without long-winded discussions or going off on tangents. Mar 5 '14 at 13:00

Java Class: ScheduledExecutorService

If you want to repeatedly and regularly do the web scraping, and you use Java, check out the bundled class ScheduledExecutorService.

An ExecutorService that can schedule commands to run after a given delay, or to execute periodically.

This class is newer, and has advantages over the more commonly cited Timer class. Search StackOverflow.com for more information and discussions.

To use a ScheduledExecutorService, you define a class that implements the Runnable interface. That interface simply means your class includes a method named run. In that method, you fetch your web page, parse it, and handle/store it. That method is where you might call a library such as jsoup. You pass an instance of your Runnable class to the ScheduledExecutorService to be run every minute, hour, whatever time you specify.

Not the answer you're looking for? Browse other questions tagged or ask your own question.