Quite often I solve problems related to getting a lot of information that I need on webpages by opening the browser on the developer mode, clicking around the page (or doing it automatically with javascript), and downloading parts of the webpage with small javascript scripts on the console. That works fine, but I'd like to learn how to do it in a more automated way. I know that Google Chrome has this command-line tool for doing some headless operations and that they even have a node.js library for doing this kind of stuff... But also, I know there're other options, like emacs that also has some automating functionalities for getting information on webpages. I'm willing to dedicate the time learning the details of a tool like this, but I'd like to have a perspective of my options. The Google Chrome solution seems like the obvious solution for me... But I don't know if I'm missing easier ways to achieve this goal.

Functionalities that I need:

  • Allow me to navigate the web in an automated way.
  • Allow me to filter classes and ids that exist on the page, and click around to change the initial state of the page.
  • Save specific parts of the HTML inside a variable or concatenate them on files (an entire div class for example, just so I can filter it properly later using different tools that are not related to this question)
  • Allow me to do all the points above in a headless mode, just so I can also analyze the data collected in a process with a programming language and tools of my choice

Are there tools that easily can do these four points? I'm ok if I end up using some very old command-line tool like a terminal browser. The main issue is modifying the state of the webpage by clicking around on ids and downloading parts of its HTML...


What you are looking for is called automated webpage test framework, take a look at selenium (Java, Python, Ruby & C++) and puppet (only Node.js?).

What's more, userscript is a lighter option, but with far less functions: you can only use JavaScript.

Update: I saw a new one called Playwright from Microsoft, cross-platform.

  • 1
    Thanks, I didn't know this term and tools. After searching for selenium on StackOverflow, I found this question that is useful as a reference for this kind of functionalities too. – raylight Apr 27 at 14:41
  • 1
    @raylight Glad to hear that! What's more, selenium is actually a wrapper for browser drivers (e.g. that Chrome link you mentioned), and it's easier to use. I personally use selenium to automation work very much, and I pretty like it! – user26742873 Apr 27 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.