1

I want to scrape webpages with dynamically loaded content. As a user browsing this webpage, I have to click on an HTML element before the content of interest is dynamically loaded into the page.

I know how to retrieve the initial HTML programmatically from the server, but how can I issue a browser-like click programmatically, and execute any appropriate javascript, so as to get access to the dynamic content?

Do I need to host a browser control inside my scraping app somehow?

Any language would work for me but C++/C# preferred.

Thanks.

1

I use Selenium to control a headless Chrome browser. This provides access to full browser functionality, such as javascript and dynamic content. You can programmatically control the browser in exactly the same way as if you were using a mouse, except in code you use XPath elements.

During the development phase, I run a full browser so I can watch it interact and can also easily troubleshoot with Developer Tools if there is an uncaught error.

I use Python. This page seems to indicate that C# is supported.

  • 1
    Since I came here to answer "Selenium". I will upvote that answer :-) I second that, @Toby There are other ways, but Selenium is probably the best way to achieve what you want. It is the industry standard test tool for browsers and browser app, and as such makes it easy to click on that HTML element and scrape the result (or, when testing, compare it to an expected value). – Mawg Jan 11 at 7:48

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.