Looking for a powerful text processing tool

Question

I need to extract some text patterns from the huge HTML.

I can definitely write a precise regular expression with the group that matches exactly what I need.

What tool would be appropriate to extract these groups and get me the list of them back?

So basically I'm looking for a powerful tool that allows you to play with the text/search/find/replace/extract using regular expressions. I assume this tool could also have different helpful functionality like sort/unique, etc.

I understand there are different linux tools that could be used for that purpose, but I don't know of any combination of them that could allow me to easy do what I need.

Here is the example of the problem:

I have an HTML source with lots of different link on it. I need to extract all the URLs from these links, but not all the URLs on the page. I need to parse these URLs and process them further.

Let's say URL is the following: http://example.com/sub/www354.
I need to extract www354 and add it to some pattern, e.g.:
myownstring/id/www354

I would appreciate for any suggestions on what is the best tool/tools to do this.

Answer 1

I would suggest looking at Python and Scrapy.

Python with it's standard libraries includes lots of very powerful text processing tools including regular expressions but scrapy takes it a lot further.

From the web site:

• Built-in support for selecting and extracting data from HTML and XML sources
• Built-in support for cleaning and sanitizing the scraped data using a collection of reusable filters (called Item Loaders) shared between all the spiders.
• Built-in support for generating feed exports in multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP, S3, local filesystem)

Both are free, cross platform and you can test the code interactively which saves a lot of time.

Answer 2

I would recommend JSoup

It is Java-based framework.

There is "Selector" mechanism to find and operate with html elements.

Answer 3

If you know C#, you can process HTML with the help of HtmlAgilityPack which is available as a NuGet package.

It uses XPath-like selectors, which may already be familiar to you from XML processing. The following is an example which I used to extract the product image, name and price from a website. Using it was really straight forward.

using System;
using System.Net;
using System.Text;
using HtmlAgilityPack;

namespace HtmlExtract
{
    class Program
    {
        static void Main(string[] args)
        {
            var doc = new HtmlDocument();
            using (var client = new WebClient())
            {
                doc.Load(client.OpenRead(args[0]), Encoding.UTF8);
            }

            var products = doc.DocumentNode.SelectNodes("//div[@class=\"product-teaser\"]");
            foreach (var product in products)
            {
                var image = product.SelectSingleNode(".//img");
                Console.WriteLine("Image URL: {0}", image.GetAttributeValue("src",""));

                var title = product.SelectSingleNode(".//h2/a");
                Console.WriteLine("Title    : {0}", title.InnerText);

                var price = product.SelectSingleNode(".//span[@class=\"regular-price\"]/span[@class=\"price\"]");
                Console.WriteLine("Price    : {0}", price.InnerText);

                Console.WriteLine();
            }
        }
    }
}

Regarding some of your specific requirements:

text/search/find/replace/extract ...

Finding HTML items is done through XPaths, see the example above. That means you specify element names and attribute names (also in combination) and get back the node. On that node, you can do additional operations, such as string operations.

using regular expressions

Well, we should not use Regex to parse HTML but after you have parsed the HTML with HtmlAgilityPack, you could parse the rest using the C# Regex class.

I assume this tool could also have different helpful functionality like sort/unique, etc.

Of course, e.g. with the SortedList class.

with lots of different link on it

The example above extracts links from a <img src="...">. Of course you can also extract links from <a href="..."> in a similar way. After doing that, you can still decide for other requirements, like if (link.StartsWith("http://www354.") { ... }.