Functionalities that I need:
- Allow me to navigate the web in an automated way.
- Allow me to filter classes and ids that exist on the page, and click around to change the initial state of the page.
- Save specific parts of the HTML inside a variable or concatenate them on files (an entire div class for example, just so I can filter it properly later using different tools that are not related to this question)
- Allow me to do all the points above in a headless mode, just so I can also analyze the data collected in a process with a programming language and tools of my choice
Are there tools that easily can do these four points? I'm ok if I end up using some very old command-line tool like a terminal browser. The main issue is modifying the state of the webpage by clicking around on ids and downloading parts of its HTML...