0

We are recently working on the development of a crawler. One major feature/function is to do transformation from crawled HTML into text. We have surveyed a few python solution such as scrapy. However, we decide to move coding language from python to nodejs (Because of the headless chrome). Is there any recommendation on exist JS framework/library excellent to do the work in transforming html into text? (e.g.: html-to-text)

Ps. It will be great that the package/library itself can have some intelligence to remove the advertisement/footer/header out and easy to do extension if we want customized logic.

Many thanks!

  • 1
    What text do you want to extract? <HEAD> metadata keywords? Page title? – Nicolas Raoul Feb 14 at 5:47
  • Hi Nicolas Raoul, We wish to extract the title/body of page. Take URL=https://www.techrepublic.com/article/10-popular-programming-languages-developers-should-learn-in-2019/ for example, we wish to extract text including title '10 popular programming languages developers should learn in 2019' and the article body. But it is ok that the library/framework not that smart/intelligent. But it is great that it knows to remove header/footer/banner and advertisement will be good enough! – johnklee Feb 14 at 6:03
  • 1
    A classic non-javascript solution would also be "lynx --dump http://example.com" (just to have something to benchmark against). – knb Feb 14 at 9:32
  • Hi knb , Got it! Many thanks for your suggestion. Good to know (lynx as 'a customizable text-based web browser')! – johnklee Feb 15 at 0:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.