4

I want to create an offline app to show drugs information. The information I need to use is already available in wikipedia. But at this page, there are so many subpages (1000+) to be saved and in my opinion doing this task manually is not feasible.

Also note that I should save each drug in a formatted element in XML (with certain schema). Now I'm looking for a tool to facilitate this task. Does such a tool exist at all or not?

My Requirements

  1. Gets a HTML page(s) and creates an XML document
  2. Preferably free
  3. Windows or Linux based
Improve this question
2
  • (a) So the tool does not have to download/select all the drug Wikipedia articles, correct? (b) Which XML schema should be used, or should it use the HTML elements with the HTML namespace? (c) Should the tool work with a list of URLs, with only one URL, with a local file, with full text input?
    – unor
    Jan 7, 2015 at 0:23
  • @unor (a) If doing so it's better (b) A simple XSD schema, I think this is not matter (c) Working with a list of URLs is better but supporting files and other documents is also acceptable (d) Thanks for your response
    – frogatto
    Jan 7, 2015 at 8:33

2 Answers 2

Reset to default
3

I doubt you will find an off-the-shelf tool for this purpose, this is IMHO a too specific requirement. But there a lots of frameworks for each major programming language which help you to implement a web "scraper" or "crawler" by yourself.

For example, googling for "python web crawler" immediately showed up http://scrapy.org/, looking for "java web crawler" gave a link to crawler4j. Using such a framework, for someone with >4K points on stackoverflow it should be a matter of a few days at most to implement what you are looking for.

Improve this answer
2
  • Thanks dear Doc Brown and @unor, your answers are really helpful and valuable but I can't unfortunately accept both of them. In my opinion experiencing some programming for this task might be more enjoyable! so I've accepted this answer, however I really appreciate unor's answer
    – frogatto
    Jan 8, 2015 at 5:26
  • @abforce: maybe you can combine the two suggestions - write a crawler (using one of the suggested frameworks), which makes use of the Special:Export feature.
    – Doc Brown
    Jan 8, 2015 at 7:21
4

Not a general tool for this job, but a solution that converts Wikipedia articles to XML documents:

  1. Go to http://en.wikipedia.org/wiki/Special:Export
  2. Enter article name(s)
  3. Click at "Export"

(By entering a category name into the "Add pages from category" field, you can automatically export all pages that belong to this category.)

On http://www.mediawiki.org/wiki/Manual:Parameters_to_Special:Export you can read about more features of the export function that can be controlled by manipulating the URL.

The actual article content (included in the XML element text) is exported in MediaWiki syntax. On http://www.mediawiki.org/wiki/Alternative_parsers you can find various tools that convert this markup to something else, e.g., XML.

For example, Pandoc can import MediaWiki syntax and export to XHTML (which is XML).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.