I've been struggling to find the best tool for the job when it comes to debugging CSS selectors or XPath expressions for the web-scraping tasks I do daily which require doing a lot of HTML-parsing. Currently, I do some sort of a combination of the following:
- Chrome developer tools (
$x()functions) but this works only if I want to actually render the page. Sometimes, it is not safe for me to render it in a browser
- use Python console and
bs4.BeautifulSoupbut the problem is that:
- that requires me to set up boilerplate code for HTML parsing
- CSS selectors support in
BeautifulSoupis very limited
What could be my better options? The ideal setup for me would be something similar to PyCharm's regular expression search inside a file, but to be able to provide a CSS selector or XPath expression and see the matching nodes/attributes highlighted.
I hoped PyCharm or other IntelliJ products have a community/third-party plugin but I failed to find one. Would really appreciate any recommendations.