1

I want to scrape webpages with dynamically loaded content. As a user browsing this webpage, I have to click on an HTML element before the content of interest is dynamically loaded into the page.

I know how to retrieve the initial HTML programmatically from the server, but how can I issue a browser-like click programmatically, and execute any appropriate javascript, so as to get access to the dynamic content?

Do I need to host a browser control inside my scraping app somehow?

Any language would work for me but C++/C# preferred.

Thanks.

Improve this question

1 Answer 1

Reset to default
1

I use Selenium to control a headless Chrome browser. This provides access to full browser functionality, such as javascript and dynamic content. You can programmatically control the browser in exactly the same way as if you were using a mouse, except in code you use XPath elements.

During the development phase, I run a full browser so I can watch it interact and can also easily troubleshoot with Developer Tools if there is an uncaught error.

I use Python. This page seems to indicate that C# is supported.

Improve this answer
1
  • 1
    Since I came here to answer "Selenium". I will upvote that answer :-) I second that, @Toby There are other ways, but Selenium is probably the best way to achieve what you want. It is the industry standard test tool for browsers and browser app, and as such makes it easy to click on that HTML element and scrape the result (or, when testing, compare it to an expected value).
    – Mawg
    Commented Jan 11, 2019 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.