I'm trying to make a web recorder. It is easy to get the content of a web page, but when you filter that data, you usually choose "Inspect Element" and assume that the element's Xpath, id, class is known and filtered.
I tried to do it with Selenium, but it's more used for testing. How can I do this with beautifulsoup in Scrapy(I tried to use selector)? Or should I use something else?
Any advice on how to do capture and parse function will be highly appreciated!