0

We are recently working on the development of a crawler. One major feature/function is to do transformation from crawled HTML into text. We have surveyed a few python solution such as scrapy. However, we decide to move coding language from python to nodejs (Because of the headless chrome). Is there any recommendation on exist JS framework/library excellent to do the work in transforming html into text? (e.g.: html-to-text)

Ps. It will be great that the package/library itself can have some intelligence to remove the advertisement/footer/header out and easy to do extension if we want customized logic.

Many thanks!

  • 1
    What text do you want to extract? <HEAD> metadata keywords? Page title? – Nicolas Raoul Feb 14 at 5:47
  • Hi Nicolas Raoul, We wish to extract the title/body of page. Take URL=https://www.techrepublic.com/article/10-popular-programming-languages-developers-should-learn-in-2019/ for example, we wish to extract text including title '10 popular programming languages developers should learn in 2019' and the article body. But it is ok that the library/framework not that smart/intelligent. But it is great that it knows to remove header/footer/banner and advertisement will be good enough! – johnklee Feb 14 at 6:03
  • 1
    A classic non-javascript solution would also be "lynx --dump http://example.com" (just to have something to benchmark against). – knb Feb 14 at 9:32
  • Hi knb , Got it! Many thanks for your suggestion. Good to know (lynx as 'a customizable text-based web browser')! – johnklee Feb 15 at 0:28

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.