Quite often I solve problems related to getting a lot of information that I need on webpages by opening the browser on the developer mode, clicking around the page (or doing it automatically with javascript), and downloading parts of the webpage with small javascript scripts on the console. That works fine, but I'd like to learn how to do it in a more automated way. I know that Google Chrome has this command-line tool for doing some headless operations and that they even have a node.js library for doing this kind of stuff... But also, I know there're other options, like emacs that also has some automating functionalities for getting information on webpages. I'm willing to dedicate the time learning the details of a tool like this, but I'd like to have a perspective of my options. The Google Chrome solution seems like the obvious solution for me... But I don't know if I'm missing easier ways to achieve this goal.
Functionalities that I need:
- Allow me to navigate the web in an automated way.
- Allow me to filter classes and ids that exist on the page, and click around to change the initial state of the page.
- Save specific parts of the HTML inside a variable or concatenate them on files (an entire div class for example, just so I can filter it properly later using different tools that are not related to this question)
- Allow me to do all the points above in a headless mode, just so I can also analyze the data collected in a process with a programming language and tools of my choice
Are there tools that easily can do these four points? I'm ok if I end up using some very old command-line tool like a terminal browser. The main issue is modifying the state of the webpage by clicking around on ids and downloading parts of its HTML...
