I've been struggling to find the best tool for the job when it comes to debugging CSS selectors or XPath expressions for the web-scraping tasks I do daily which require doing a lot of HTML-parsing. Currently, I do some sort of a combination of the following:

  • Chrome developer tools ($(), $$(), $x() functions) but this works only if I want to actually render the page. Sometimes, it is not safe for me to render it in a browser
  • use Python console and lxml.html or bs4.BeautifulSoup but the problem is that:

What could be my better options? The ideal setup for me would be something similar to PyCharm's regular expression search inside a file, but to be able to provide a CSS selector or XPath expression and see the matching nodes/attributes highlighted.

I hoped PyCharm or other IntelliJ products have a community/third-party plugin but I failed to find one. Would really appreciate any recommendations.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.