C# library for parsing HTML?

Question

The library should support at least .NET framework version 3.5, and 4.0; support for version 4.5, and future versions, would be great too.

It should handle 'messy' HTML too, if possible.

Answer 1

AngleSharp:

• Actively developed/maintained
• Built-in support for CSS selectors

Answer 2

Html Agility Pack

I've used Html Agility Pack and, although its home page only explicitly mentions version 2.0, it works great with version 4.0 of the .NET framework. I suspect it works fine with version 4.5 too.

Here's some example code using Html Agility Pack with LINQ:

var document = new HtmlDocument();
document.Load(@"C:\Documents and Settings\Kenny\My Documents\project\document.html");

var table = document.GetElementbyId("table5");
var tableRows = table.ChildNodes
                    .Where(cn => cn.NodeType == HtmlNodeType.Element)
                    .Skip(2);

Unfortunately, it doesn't seem to be an active project. The latest change, per the project CodePlex page, is July 2012. It's possible there's not much room for improvement and based on my preliminary usage it seems stable and fast.

Fizzler

Fizzler is built on top of Html Agility Pack and provides support for using CSS selectors to access parsed HTML documents.

Unfortunately, like Html Agility Pack, it also seems inactive; the latest change, per its Google Code source listing, the last change was January 2013. It's also possible that it too is stable and not in need of ongoing development or maintenance.

Other Resources

• .net - What is the best way to parse html in C#? - Stack Overflow
• Parsing HTML with c#.net - Stack Overflow

Answer 3

CsQuery is also very good HTML parser with CSS selectors. It generates same DOM as Gecko based browsers. It has also much better license (MIT) then Html Agility Pack (MS-PL), which is incomatible with GPL.

This library is also very easy to use because it has jQuery like API.

EDIT: Currently (25 Jun 2016) it is not actively maintained. So there is better alternative like AngleSharp.

Answer 4

CefSharp

Why?

• Actively maintained
• You get the might of Chromium
• Let's you run any JavaScript. It's much easier to develop parsing this way. You go to your Chromium-based browser console and develop script you want. When you've written some little C# code basis, your way of development is just in to paste Javascript code from console without need to write C# loops and query.
• Let's you fire C# event from your JavaScript code. It is extremely useful when you want to fire event of AJAX success to get results.

There are three kinds of CefSharp:

• CefSharp.WinForms
• CefSharp.Wpf
• CefSharp.OffScreen

The first two are used like IE-based WebBrowser in Windows.Forms. But it is Chromium based. And for parsing you should use CefSharp.OffScreen.

Install it via Nuget and use it.
Install-Package CefSharp.OffScreen -Version 57.0.0

---

Code

Provided examples are not short as possible but they will make programming using CefSharp easier.

I will using jQuery for Javascript calls for demonstration and example simplicity assuming that target site has this library. You can execute plain JS or chose any that is available on target site.

First of all javascript results are returned througth JavascriptResponse's property Result of object type. Javascript arrays are mapped to List<object>. Other result type mapping are evident: string, int, bool but they all will be stored in object Result property. To make Javascript methods generic I use the following ConvertHelper.

public static class ConvertHelper
{
    public static T[] GetArrayFromObjectList<T>(object obj)
    {
        return ((IEnumerable<object>)obj)
            .Cast<T>()
            .ToArray();
    }

    public static List<T> GetListFromObjectList<T>(object obj)
    {
        return ((IEnumerable<object>)obj)
            .Cast<T>()
            .ToList();
    }

    public static T ToTypedVariable<T>(object obj)
    {
        if (obj == null)
        {
            dynamic dynamicResult = null;
            return dynamicResult;
        }

        Type type = typeof(T);
        if (type.IsArray)
        {
            dynamic dynamicResult = typeof(ConvertHelper).GetMethod(nameof(GetArrayFromObjectList))
                .MakeGenericMethod(type.GetElementType())
                .Invoke(null, new[] { obj });
            return dynamicResult;
        }

        if (type.IsGenericType && type.GetGenericTypeDefinition() == typeof(List<>))
        {
            dynamic dynamicResult = typeof(ConvertHelper).GetMethod(nameof(GetListFromObjectList))
                .MakeGenericMethod(type.GetGenericArguments().Single())
                .Invoke(null, new[] { obj });
            return dynamicResult;
        }

        return (T)obj;
    }
}

I've add class to handle Javascript errors:

public class JavascriptException : Exception
{
    public JavascriptException(string message): base(message) { }
}

Then we need to create our core CefSharpWrapper class to perform all dirty work with browser stuff.

public class CefSharpWrapper
{
    private ChromiumWebBrowser _browser;

    public void InitializeBrowser()
    {
        CefSettings settings = new CefSettings();

        // Perform dependency check to make sure all relevant resources are in our output directory.
        Cef.Initialize(new CefSettings(), performDependencyCheck: true, browserProcessHandler: null);

        _browser = new ChromiumWebBrowser();

        // wait till browser initialised
        AutoResetEvent waitHandle = new AutoResetEvent(false);

        EventHandler onBrowserInitialized = null;

        onBrowserInitialized = (sender, e) =>
        {
            _browser.BrowserInitialized -= onBrowserInitialized;

            waitHandle.Set();
        };

        _browser.BrowserInitialized += onBrowserInitialized;

        waitHandle.WaitOne();
    }

    public void ShutdownBrowser()
    {
        // Clean up Chromium objects
        Cef.Shutdown();
    }

    public Task<T> GetResultAfterPageLoad<T>(string pageUrl, Func<Task<T>> onLoadCallback)
    {
        TaskCompletionSource<T> tcs = new TaskCompletionSource<T>();

        EventHandler<LoadingStateChangedEventArgs> onPageLoaded = null;

        T t = default(T);

        // An event that is fired when the first page is finished loading.
        // This returns to us from another thread.
        onPageLoaded = async (sender, e) =>
        {
            // Check to see if loading is complete - this event is called twice, one when loading starts
            // second time when it's finished
            // (rather than an iframe within the main frame).
            if (!e.IsLoading)
            {
                // Remove the load event handler, because we only want one snapshot of the initial page.
                _browser.LoadingStateChanged -= onPageLoaded;

                t = await onLoadCallback();

                tcs.SetResult(t);
            }
        };

        _browser.LoadingStateChanged += onPageLoaded;

        _browser.Load(pageUrl);

        return tcs.Task;
    }

    // Method to get result via Javascript    
    public async Task<T> EvaluateJavascript<T>(string script)
    {
        JavascriptResponse javascriptResponse = await browser.GetMainFrame().EvaluateScriptAsync(script);

        if (javascriptResponse.Success)
        {
            object scriptResult = javascriptResponse.Result;
            return ConvertHelper.ToTypedVariable<T>(scriptResult);
        }

        throw new JavascriptException(javascriptResponse.Message);
    }
}

Then we call our CefSharpWrapper class from Main method to get all a href's from stackoverflow home page.

public class Program
{
    private static void Main()
    {
        MainAsync().Wait();
    }

    private static async Task MainAsync()
    {
        CefSharpWrapper wrapper = new CefSharpWrapper();

        wrapper.InitializeBrowser();

        string[] urls = await wrapper.GetResultAfterPageLoad("http://stackoverflow.com/", async () =>
            await wrapper.EvaluateJavascript<string[]>("$('a[href]').map((index, element) => $(element).prop('href')).toArray()"));

        wrapper.ShutdownBrowser();
    }
}

Note: this library doesn't distinguish empty array, null and undefined. They are all returned as null. So to avoid NullReferenceException add either corresponding code to CefSharpWrapper (but then you would have to deal with distinguishing is null in C# meant null or empty array in Javascript) or add following code to Main.

if (urls == null) urls = new string[0];

Answer 5

If you want something really fast, look in here: Majestic-12 : Projects : C# HTML parser (.NET)

It won't be the easiest to use but it'll probably be the fastest.