0

Do we have a program that takes all text in a website? I wanted to know because I want to see a whole website in text and change it a little.

  • Do you mean the actual readable text or a text representation of a webpage (HTML)? – Mr Public Mar 24 '17 at 16:25
  • 1
    Yes - a web crawler such as scrapy can crawl the site asking for all of the content. – Steve Barnes Apr 6 '17 at 18:03
  • 1
    @user6779864: There is a nice walk-though at doc.scrapy.org/en/latest/intro/tutorial.html – Steve Barnes Apr 10 '17 at 11:09
  • 1
    @user6779864 - doc.scrapy.org/en/latest/intro/tutorial.html saves all of the content of pages, (as html), in the first spider section and demonstrates using the scrapy shell to get the text from a page. – Steve Barnes Apr 23 '17 at 10:06
  • 1
    @user6779864 - Yes: Did you TRY following the walk through which shows you exactly how to do what you are asking for. – Steve Barnes Apr 27 '17 at 7:10
1

Pandoc - "a universal document converter" should be in just about everybody's tool kit.

It is:

  • Free, gratis & open source
  • Just about invaluable

With it you can convert from one format to another, including from online web pages to plain text.

For an example:

pandoc https://en.wikipedia.org/wiki/Pandoc -f html -t plain -o soq.txt

Is asking pandoc to download the hmtl page about itself from Wikipedia and convert it to plain text outputting to a file called soq.txt The result looks like:

PANDOC

From Wikipedia, the free encyclopedia

Jump to: navigation, search

Pandoc Original author(s) John MacFarlane Initial release 10 August 2006 (10 years ago) (2006-08-10) Stable release 1.19 / 1 December 2016 (3 months ago) (2016-12-01)

Repository github.com/jgm/pandoc Development status Active Written in Haskell Operating system Unix-like, OS X, Windows License GNU GPLv2 Website pandoc.org - []Free software portal

PANDOC is a free and open-source software document converter, widely used as a writing tool (especially by scholars)1[3][4] and as a basis for publishing workflows.[5][6][7] It was originally created by John MacFarlane, a philosophy professor at the University of California, Berkeley.[8]

Contents

  • 1 Supported file formats
  • 2 Integration with reference managers
  • 3 References
  • 4 External links

Supported file formats[edit]

Pandoc's most thoroughly supported file format is an extended version of Markdown, but it can also read many other forms of lightweight markup language, HTML, ReStructuredText, LaTeX, OPML, Org-mode, DocBook, and Office Open XML (Microsoft Word .docx).

It can be used to create files in many more formats, including Office Open XML, OpenDocument, HTML, Wiki markup, InDesign ICML, web-based slideshows,[9] ebooks,[10] OPML, and various TeX formats (through which it can produce a PDF). It has built-in support for converting LaTeX mathematical equations to MathML and MathJax, among other formats.

Plug-ins for custom formats can also be written in Lua, which has been used to create an exporting tool for the Journal Article Tag Suite.[11]

Integration with reference managers[edit]

An included module, pandoc-citeproc, allows the program to use data from reference management software such as BibTeX, EndNote, Mendeley, or Papers. It has the ability to integrate directly with Zotero.[12] The information is automatically transformed into a citation in various styles (such as APA, Chicago, or MLA) using an implementation of the Citation Style Language. This allows the program to serve as a simpler alternative to LaTeX for producing academic writing.[13]

References[edit]

  1. ^ Mullen, Lincoln (2012-02-23). "Pandoc Converts All Your (Text) Documents". The Chronicle of Higher Education Blogs: ProfHacker. Retrieved 2014-06-27. 

  2. ^ McDaniel, W. Caleb (2012-09-28). "Why (and How) I Wrote My Academic Book in Plain Text". W. Caleb McDaniel at Rice University. Retrieved 2014-06-27. 

  3. ^ Healy, Kieran (2014-01-23). "Plain Text, Papers, Pandoc". Retrieved 2014-06-27. 

  4. ^ Ovadia, Steven (2014). "Markdown for Librarians and Academics". Behavioral & Social Sciences Librarian. 33 (2): 120–124. doi:10.1080/01639269.2014.904696. ISSN 0163-9269. 

  5. ^ Till, Kaitlyn; Shed Simas; Velma Larkai (2014-04-14). "The Flying Narwhal: Small mag workflow". Publishing @ SFU. Retrieved 2014-06-27. 

  6. ^ Maxwell, John (2013-11-01). "Building Publishing Workflows with Pandoc and Git". Publishing @ SFU. Retrieved 2014-06-27. 

  7. ^ Maxwell, John (2014-02-26). "On Pandoc". eBound Canada: Digital Production Workshop, Vancouver, BC. Retrieved 2014-06-27. 

  8. ^ "John MacFarlane". Department of Philosophy. University of California, Berkeley. Retrieved 25 July 2014. 

  9. ^ See as an example MacFarlane, John (2014-05-17). "Pandoc for Haskell Hackers". BayHac 2014, Mountain View, CA. Retrieved 2014-06-27.  The source file is written in Markdown.

  10. ^ Mullen, Lincoln (2012-03-20). "Make Your Own E-Books with Pandoc". The Chronicle of Higher Education Blogs: ProfHacker. Retrieved 2014-06-27. 

  11. ^ Fenner, Martin (2013-12-12). "From Markdown to JATS XML in one Step". Gobbledygook. Retrieved 2014-06-27. 

  12. ^ Hetzner, Erik (2014-06-25). "zotxt". Retrieved 2014-06-27. 

  13. ^ Tenen, Dennis; Grant Wythoff (2014-03-19). "Sustainable Authorship in Plain Text using Pandoc and Markdown". The Programming Historian. Retrieved 2014-06-27. 

External links[edit]

  • Official website
  • PanDoc and Wikiversity – PanDocElectron

[]

Retrieved from "https://en.wikipedia.org/w/index.php?title=Pandoc&oldid=756329870"

Categories: - 2006 software - Free software programmed in Haskell - Technical communication tools - Workflow software

Hidden categories: - Official website different in Wikidata and Wikipedia

Navigation menu

Personal tools

  • Not logged in

Talk

Contributions

Create account

Log in

Namespaces

  • Article

    Talk

Variants

Views

  • Read

Edit

View history

More

Search

Navigation

  • Main page

Contents

Featured content

Current events

Random article

Donate to Wikipedia

Wikipedia store

Interaction

  • Help

About Wikipedia

Community portal

Recent changes

Contact page

Tools

  • What links here

Related changes

Upload file

Special pages

Permanent link

Page information

Wikidata item

Cite this page

Print/export

  • Create a book

Download as PDF

Printable version

Languages

  • Deutsch
  • Español
  • Français
  • Русский
  • 中文

Edit links

  • This page was last modified on 23 December 2016, at 14:17.

    Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization.

 

Privacy policy

About Wikipedia

Disclaimers

Contact Wikipedia

Developers

Cookie statement

Mobile view

 

[Wikimedia Foundation]

[Powered by MediaWiki]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.