0

I need a headless browser, or similar, pure Java library like HtmlUnit that supports JavaScript to reliably render all web page content just as a manual browser would do.

Although HtmlUnit claims to support JavaScript it is very unreliable. I switched from HtmlUnit to ui4j three years ago because ui4j initially did a much better job rendering JavaScript-dependent web page content.

The trouble is ui4j is an unsupported library not subject to updates. It is getting progressively less reliable as websites continue to evolve and it gets increasingly left behind.

My application isn't the traditional one of web crawling and scraping to collate large amounts of user data. I only need to get web page content from a few specific websites. But the content has to be rendered just as it would be in a manual browser which I can then programmatically edit and save using the library to simulate browser events such as clicks.

Can anyone recommend a pure Java library that would let me do this reliably for all websites?

Improve this question
2
  • I am sorry but "pure Java" + " just as a manual browser would do" + "reliable JavaScript" = impossible. All common JavaScript engines are implemented in a language different than Java. Even among them compatibility isn't easy. If you want a something that mimics a web browser 100% then you have to use a real web browser like Firefox or Chrome. Have you tried Selenium in headless mode?
    – Robert
    Jun 13, 2022 at 16:39
  • I’d heard about Selenium but your comment encouraged me to give it a go. Selenium does not natively support modifying a web page’s DOM, but it can be done by JavaScript through use of its executeScript() method. All the reliability issues faced using libraries like ui4j etc are now resolved because web page handling is performed by a mainstream reliable browser, running in headless mode. The only downside (which I can live with) is that my app now relies on another application running at the same time (ie, the Chrome browser exe). Many thanks Robert, your help was a breakthrough for me!
    – Marty
    Jul 15, 2022 at 4:18

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.