0

I am in a situation where I have to analyze text data. I have URLs of numerous websites from where I need to rip text and place it in a file for further analysis.

I tried few crawlers but they are not fulfilling the purpose. Is there any tool available? Otherwise I have to do manually which is lot of time consuming activity. I need less or more similar tool that should operate on a batch of URLs all at once.

App that convert a web page to text: Textise

0

If you have a list of web pages to fetch on a periodic basis, (i.e. no need to follow links, etc), then pandoc can fetch the pages and convert then to any of a number of formats including plain text.

  • Free, gratis & open source
  • Numerous output formats
  • Cross Platform: Windows, Linux & OS-X

Example:

pandoc -s -f html http://www.gnu.org/software/make/ -o example12.text

Will result in a markdown version of the GNU make manual. If you have multiple URLs on a single command line they will be downloaded in the order given and added to a single text, (or other format).

You can easily construct a batch file to fetch several pages one after the other to differing output files.

Example with two sources and https:

C:\>pandoc -s -f html -t plain https://docs.python.org/3/whatsnew/3.5.html https://docs.python.org/2.7/whatsnew/2.7.html -o whatsnew.txt

results in a plain text, (but with some UTF-8 characters), output file with both sets of what's new.

Alternatively copying the following into a batch file called getpynew.bat:

@echo Getting whats new in python 3.5
@pandoc -s -f html -t plain https://docs.python.org/3/whatsnew/3.5.html -o whatsnew3.txt
@echo Getting whats new in python 3.5
@pandoc -s -f html -t plain https://docs.python.org/2.7/whatsnew/2.7.html -o whatsnew2.txt
@echo Done

And running it by either typing the name or double clicking will result in 2 files.

N.B. Take care when trying to use lots of URLs on a single line as Windows has a character limit of about 250 length, if you are doing a lot of URLs and need them to be in a single file then do them one at a time and then concatenate the files.

| improve this answer | |
  • It worked. But when I increase URLs in single command even 2 URLs then it gives pandoc ResponseTimeout Error. Secondly, the resultant text document also contain div tags, how can I have plain text. – usra Apr 11 '16 at 9:18
  • To get rid of the div tags try -t plain not sure why you are getting timeout errors - you do need to be sure that a) the URLs all work & b) you only have 1 -o followed by the name of the output file. – Steve Barnes Apr 11 '16 at 10:09
  • Worked.last help needed on a) how to handle https as pandoc does not recognize it b) separator for multiple URLs in one command? c) any link to build batch file for above command. PS: I use windows – usra Apr 11 '16 at 10:43
  • Updated answer, the only one not immediately obvious is that the separator is a simple space. – Steve Barnes Apr 11 '16 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.