18

The library should support at least .NET framework version 3.5, and 4.0; support for version 4.5, and future versions, would be great too.

It should handle 'messy' HTML too, if possible.

8

AngleSharp:

  • Actively developed/maintained
  • Built-in support for CSS selectors
4

Html Agility Pack

I've used Html Agility Pack and, although its home page only explicitly mentions version 2.0, it works great with version 4.0 of the .NET framework. I suspect it works fine with version 4.5 too.

Here's some example code using Html Agility Pack with LINQ:

var document = new HtmlDocument();
document.Load(@"C:\Documents and Settings\Kenny\My Documents\project\document.html");

var table = document.GetElementbyId("table5");
var tableRows = table.ChildNodes
                    .Where(cn => cn.NodeType == HtmlNodeType.Element)
                    .Skip(2);

Unfortunately, it doesn't seem to be an active project. The latest change, per the project CodePlex page, is July 2012. It's possible there's not much room for improvement and based on my preliminary usage it seems stable and fast.

Fizzler

Fizzler is built on top of Html Agility Pack and provides support for using CSS selectors to access parsed HTML documents.

Unfortunately, like Html Agility Pack, it also seems inactive; the latest change, per its Google Code source listing, the last change was January 2013. It's also possible that it too is stable and not in need of ongoing development or maintenance.

Other Resources

  • 1
    HAP is still popular, but it's buggy quirky garbage and should be avoided in modern code. The bug of incorrect parsing of HTML tags with optional closing tags is still there, after many many years. It still returns nulls instead of empty collections making code verbose. Also HAP is Ms-PL and Fizzler is LGPL, not MIT/BSD, which can be important in some cases. Just do not use it, there're better options. – Athari Jun 11 '17 at 5:08
4

CsQuery is also very good HTML parser with CSS selectors. It generates same DOM as Gecko based browsers. It has also much better license (MIT) then Html Agility Pack (MS-PL), which is incomatible with GPL.

This library is also very easy to use because it has jQuery like API.

EDIT: Currently (25 Jun 2016) it is not actively maintained. So there is better alternative like AngleSharp.

3

If you want something really fast, look in here: Majestic-12 : Projects : C# HTML parser (.NET)

It won't be the easiest to use but it'll probably be the fastest.

3

CefSharp

Why?

  • Actively maintained
  • You get the might of Chromium
  • Let's you run any JavaScript. It's much easier to develop parsing this way. You go to your Chromium-based browser console and develop script you want. When you've written some little C# code basis, your way of development is just in to paste Javascript code from console without need to write C# loops and query.
  • Let's you fire C# event from your JavaScript code. It is extremely useful when you want to fire event of AJAX success to get results.

There are three kinds of CefSharp:

  • CefSharp.WinForms
  • CefSharp.Wpf
  • CefSharp.OffScreen

The first two are used like IE-based WebBrowser in Windows.Forms. But it is Chromium based. And for parsing you should use CefSharp.OffScreen.

Install it via Nuget and use it.
Install-Package CefSharp.OffScreen -Version 57.0.0


Code

Provided examples are not short as possible but they will make programming using CefSharp easier.

I will using jQuery for Javascript calls for demonstration and example simplicity assuming that target site has this library. You can execute plain JS or chose any that is available on target site.

First of all javascript results are returned througth JavascriptResponse's property Result of object type. Javascript arrays are mapped to List<object>. Other result type mapping are evident: string, int, bool but they all will be stored in object Result property. To make Javascript methods generic I use the following ConvertHelper.

public static class ConvertHelper
{
    public static T[] GetArrayFromObjectList<T>(object obj)
    {
        return ((IEnumerable<object>)obj)
            .Cast<T>()
            .ToArray();
    }

    public static List<T> GetListFromObjectList<T>(object obj)
    {
        return ((IEnumerable<object>)obj)
            .Cast<T>()
            .ToList();
    }

    public static T ToTypedVariable<T>(object obj)
    {
        if (obj == null)
        {
            dynamic dynamicResult = null;
            return dynamicResult;
        }

        Type type = typeof(T);
        if (type.IsArray)
        {
            dynamic dynamicResult = typeof(ConvertHelper).GetMethod(nameof(GetArrayFromObjectList))
                .MakeGenericMethod(type.GetElementType())
                .Invoke(null, new[] { obj });
            return dynamicResult;
        }

        if (type.IsGenericType && type.GetGenericTypeDefinition() == typeof(List<>))
        {
            dynamic dynamicResult = typeof(ConvertHelper).GetMethod(nameof(GetListFromObjectList))
                .MakeGenericMethod(type.GetGenericArguments().Single())
                .Invoke(null, new[] { obj });
            return dynamicResult;
        }

        return (T)obj;
    }
}

I've add class to handle Javascript errors:

public class JavascriptException : Exception
{
    public JavascriptException(string message): base(message) { }
}

Then we need to create our core CefSharpWrapper class to perform all dirty work with browser stuff.

public class CefSharpWrapper
{
    private ChromiumWebBrowser _browser;

    public void InitializeBrowser()
    {
        CefSettings settings = new CefSettings();

        // Perform dependency check to make sure all relevant resources are in our output directory.
        Cef.Initialize(new CefSettings(), performDependencyCheck: true, browserProcessHandler: null);

        _browser = new ChromiumWebBrowser();

        // wait till browser initialised
        AutoResetEvent waitHandle = new AutoResetEvent(false);

        EventHandler onBrowserInitialized = null;

        onBrowserInitialized = (sender, e) =>
        {
            _browser.BrowserInitialized -= onBrowserInitialized;

            waitHandle.Set();
        };

        _browser.BrowserInitialized += onBrowserInitialized;

        waitHandle.WaitOne();
    }

    public void ShutdownBrowser()
    {
        // Clean up Chromium objects
        Cef.Shutdown();
    }

    public Task<T> GetResultAfterPageLoad<T>(string pageUrl, Func<Task<T>> onLoadCallback)
    {
        TaskCompletionSource<T> tcs = new TaskCompletionSource<T>();

        EventHandler<LoadingStateChangedEventArgs> onPageLoaded = null;

        T t = default(T);

        // An event that is fired when the first page is finished loading.
        // This returns to us from another thread.
        onPageLoaded = async (sender, e) =>
        {
            // Check to see if loading is complete - this event is called twice, one when loading starts
            // second time when it's finished
            // (rather than an iframe within the main frame).
            if (!e.IsLoading)
            {
                // Remove the load event handler, because we only want one snapshot of the initial page.
                _browser.LoadingStateChanged -= onPageLoaded;

                t = await onLoadCallback();

                tcs.SetResult(t);
            }
        };

        _browser.LoadingStateChanged += onPageLoaded;

        _browser.Load(pageUrl);

        return tcs.Task;
    }

    // Method to get result via Javascript    
    public async Task<T> EvaluateJavascript<T>(string script)
    {
        JavascriptResponse javascriptResponse = await browser.GetMainFrame().EvaluateScriptAsync(script);

        if (javascriptResponse.Success)
        {
            object scriptResult = javascriptResponse.Result;
            return ConvertHelper.ToTypedVariable<T>(scriptResult);
        }

        throw new JavascriptException(javascriptResponse.Message);
    }
}

Then we call our CefSharpWrapper class from Main method to get all a href's from stackoverflow home page.

public class Program
{
    private static void Main()
    {
        MainAsync().Wait();
    }

    private static async Task MainAsync()
    {
        CefSharpWrapper wrapper = new CefSharpWrapper();

        wrapper.InitializeBrowser();

        string[] urls = await wrapper.GetResultAfterPageLoad("http://stackoverflow.com/", async () =>
            await wrapper.EvaluateJavascript<string[]>("$('a[href]').map((index, element) => $(element).prop('href')).toArray()"));

        wrapper.ShutdownBrowser();
    }
}

Note: this library doesn't distinguish empty array, null and undefined. They are all returned as null. So to avoid NullReferenceException add either corresponding code to CefSharpWrapper (but then you would have to deal with distinguishing is null in C# meant null or empty array in Javascript) or add following code to Main.

if (urls == null) urls = new string[0];
  • This question isn't related to rendering HTML. I couldn't find anything pertaining to parsing HTML in the project info on GitHub. – Kenny Evitt Nov 28 '16 at 15:09
  • I know. You can use this library for parsing and it's very comfortable for parsing that's why I'm recommending it. You just call ChromiumWebBrowser's EvaluateScriptAsync method and using Javascript you get results you want. For instance if target site supports jQuery and you need all href of a you can call _browser.EvaluateScriptAsync("$('a[href]').map((index, element) => $(element).prop('href')).toArray()"). I can show you code examples but not sure it will be relevant in this website. I have used current library in production for parsing web sites. – Vadim Ovchinnikov Nov 28 '16 at 19:32
  • Code examples are absolutely relevant; please add a simple example if you can. For instance, how would C# code retrieve data parsed by the JavaScript code? This is a very interesting answer to the question. I was implicitly assuming that the HTML I'd want to parse would already be provided, but being able to use Chromium (or any browser engine) would allow things like single-page-web-apps to also be processed. – Kenny Evitt Nov 28 '16 at 20:35
  • Very interesting (and very hacky) answer! – Kenny Evitt Dec 19 '16 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.