I have a list of 1,000 URLs pointing to news articles. I am looking for a program (CLI if possible) or libraries (C++/Java or interpreted languages preferred) that can take a URL as input and output the main content of a page written in English. It is OK if it is specialized for news articles only.

E.g. if the input URL is http://news.sky.com/story/384359/russia-fake-vodka-kills-25-people, it should return:


A deadly form of fake vodka has killed 25 people in Russia, prompting police to issue a public warning. The batch of illegal alcohol has been tracked down to the far east of the country in Siberia.

Local news agencies reported that 25 people had bought drinks contaminated with methanol, a form of alcohol that can cause blindness or kill.

After it appeared their deaths may be connected, police issued a warning to local residents to avoid buying the vodka substitute.

Four women have since been arrested for running an illegal distillery that may have produced the deadly drinks.

Although vodka is very cheap in Russia, many prefer to buy home-made varieties - known as "samogan" - to save money.

Fake vodka is often made from wood alcohol which is meant for industrial purposes.

Alcohol poisoning kills thousands of Russians every year; 39,000 died in 2004 alone.


Russia: Fake Vodka Kills 25 People

If possible, free and working with Windows 7 or Linux.


First of all, there is no reason to use C++, stick with scripting languages (Python, Ruby, PHP or Javascript will do the trick just fine). Because you will be iterating frequently and also because the bottleneck will never be the parsing but rather the network latency. Also note that you should be working from the CLI.

The concept is the following:

  • Retrieve html page source, curl is a good library for that
  • Deal with any encoding issues (force everything to utf-8 for simplicity)
  • Parse the content to get only what you are after (ideally with a DOM parser, otherwise RegEx)
  • Store the results to a database

There are many frameworks out there that will take care of the mundane tasks, you should only write the scraping functions. The title will probably be between the <title> tags, as for the content you can either do it manually for more precision depending on the number of different domains you are scraping, or automatically if its too diverse. In this case, try looking for <p> tags or big blogs of text without html non-text markup with Regex.

As for the frameworks themselves, you can have a look at:

  • Goutte (PHP)
  • Scrappy (Python)
  • Beautiful Soup (Python) and Rubiful Soup (Ruby)
  • Node + Cheerios (Server-side javascript, very powerful jquery css selectors!)

Good luck


Boilerpipe is written in Java and does exactly that.

You can try their demo. The demo does sometimes go over quota and becomes unavailable; if that happens just try again later.

I found it very useful and easy to implement.

  • Last time I was into this nothing even came close to Boilerpipe. Also somebody had ported it to C#. Sep 30 '14 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.