5

Wikidata is an online database which contains many details about many countries, politicians, paintings, etc. For instance, for each country you have name, flag, map, which are all strings or URLs to online images.

How can I simply generate a CSV file containing the item type and properties I want?

For instance, if I say I want all countrys and their name and capital, then it would generate a CSV file like this:

country;capital
India;New Delhi
Brazil;Brasília
...

Any OS/webapp/app is OK. Preferably open source.

I don't want to download the whole Wikidata database locally, so the tool would have to make requests to the official live server.

  • 1
    Not a full-up tool, but in Perl you can use Catmandu with Catmandu::Wikidata module to import Wikidata; and then Text::CSV to export from whatever DB you used for Catmandu storage – DVK Feb 11 '14 at 1:15
2

You can query Wikidata with SPARQL at https://query.wikidata.org/. I created the command line tool wdq to facilitate querying from command line. You task can be solved as following (please use the most recent version 0.4.4). First find out the item-identifiers for country

$ wdq country
Q6256: country
  distinct region in geography; a broad term that can include political divisio…

and the property-identifiers for capital, flag etc.

$ wdq psearch capital
P36: capital
  location (city, municipality) of governmental seat of the country, or adminis…
...

Another important property is P31: instance of. You can also find these identifiers via the web interface at https://www.wikidata.org/.

Then formulate and run a query in abbreviated SPARQL syntax:

$ wdq "?country wdt:P31 wd:Q6256 ; wdt:P36 ?capital" 

To get labels in addition to identifiers and to emit CSV, use

$ wdq -fcsv -lcountry,capital "?country wdt:P31 wd:Q6256 ; wdt:P36 ?capital" 

If you only want the labels, the easiest way to go is this (unless countries contain a semicolon in its name):

$ wdq --format "{countryLabel};{capitalLabel}" -lcountry,capital "?country wdt:P31 wd:Q6256 ; wdt:P36 ?capital" 

By the way you can choose the output language with option -g/--language. You can also get the full SPARQL query to reuse for instance at https://query.wikidata.org/:

$ wdq -lcountry,capital "?country wdt:P31 wd:Q6256 ; wdt:P36 ?capital" -n
PREFIX bd: <http://www.bigdata.com/rdf#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX schema: <http://schema.org/>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX wikibase: <http://wikiba.se/ontology#>
SELECT * WHERE {
    ?country wdt:P31 wd:Q6256 .
    ?country wdt:P36 ?capital .
    SERVICE wikibase:label {
        bd:serviceParam wikibase:language "en" .
        ?capital rdfs:label ?capitalLabel .
    }
    SERVICE wikibase:label {
        bd:serviceParam wikibase:language "en" .
        ?country rdfs:label ?countryLabel .
    }
}

If you add ?capital wdt:P625 ?coordinate . you get a nice map of capitals of the world.

If you also install perl module Catmandu, wdq can directly import the result set into a database and/or apply more postprocessing and output formatting.

1

You should be able to do this quite quickly by using python and scrapy.

Scrapy is a python web scraping program and python has default libraries to write to csv files so between the two you are a lot of the way there.

Both are Free - both Libre and FOSS.

  • I would prefer to avoid screen-scraping since they provide an API and are a non-profit. Still waiting for a solution usable by non-programmers :-) – Nicolas Raoul Jan 5 '15 at 1:04
  • @NicolasRaoul if you need the level of flexibility that you indicate - any query, selective output - for the non-programmer about the only option is a web browser + pencil & paper! – Steve Barnes Jan 5 '15 at 5:23
1

There are many tools to accomplish your goals and the advantages and disadvantages largely depend on your current skill set. Therefore, I will simply list the tools that I know about you will have to examine which tools match the languages your know and the platforms you have access to. Furthermore, my experiences is that all of the tools are imperfect and that you will have to improve them to get exactly what you need.

Official Wikimedia information

  1. Manual:Using content from Wikipedia
  2. Alternative parsers (an excellent list of many different types of parsers but many of them are out of date
  3. Manual:Pywikibot/Scripts Official Wikimedia python-based scripts to accomplish tasks

More tools

  1. Ways to process and use Wikipedia dumps A very old blog post listing some tools
  2. DBpedia is a community focused on extracting structured data from Wikipedia
  3. Scrapy was already mentioned
  4. import.io is not specific to Wikipedia, but it has the power to accomplish your goals

Another thought The information you are looking for almost certainly already exists somewhere. CIA world fact book, UN databases, and open data sources certainly have this information.

Good luck!

  • All of these tools are not specifically for Wikidata. I want a tool designed to extract data from Wikidata. country/capital is just an example, I need CSV files for a wide array of things, under a reusable license like the one Wikidata provide. – Nicolas Raoul Jan 5 '15 at 1:00
1

1) Go to https://query.wikidata.org

2) Enter this SPARQL query:

SELECT ?item ?itemLabel ?capitalLabel ?flag ?map
WHERE
{
    ?item wdt:P31 wd:Q6256 .
    OPTIONAL {?item wdt:P36 ?capital}
    OPTIONAL {?item wdt:P41 ?flag}
    OPTIONAL {?item wdt:P242 ?map}
    SERVICE wikibase:label { bd:serviceParam wikibase:language "en" }
}

This query might look complicated, but it is actually easy. It could be translated to:

Show the item, its human-readable label, and the variables called ?capital, ?flag, ?map  
The item is a country.  
The item's capital is ?capital.  
The item's flag is ?flag.  
The item's map is ?map.  
Use English for human-readable labels.

There are many properties such as P:36, the best way to find other is to look at a real item, the properties are the grey column in the Statements section.

3) Press "Download" then "CSV"

Result:

item,itemLabel,capitalLabel,flag,map
http://www.wikidata.org/entity/Q16,Canada,Ottawa,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20Canada.svg,http://commons.wikimedia.org/wiki/Special:FilePath/Canada%20%28orthographic%20projection%29.svg
http://www.wikidata.org/entity/Q17,Japan,Tokyo,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20Japan.svg,http://commons.wikimedia.org/wiki/Special:FilePath/Japan%20%28orthographic%20projection%29.svg
http://www.wikidata.org/entity/Q20,Norway,Oslo,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20Norway.svg,http://commons.wikimedia.org/wiki/Special:FilePath/Europe-Norway.svg
http://www.wikidata.org/entity/Q27,Ireland,Dublin,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20Ireland.svg,http://commons.wikimedia.org/wiki/Special:FilePath/EU-Ireland.svg
http://www.wikidata.org/entity/Q28,Hungary,Budapest,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20Hungary.svg,http://commons.wikimedia.org/wiki/Special:FilePath/EU-Hungary.svg
http://www.wikidata.org/entity/Q29,Spain,Madrid,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20Spain.svg,http://commons.wikimedia.org/wiki/Special:FilePath/EU-Spain.svg
http://www.wikidata.org/entity/Q30,United States of America,Washington,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20the%20United%20States.svg,http://commons.wikimedia.org/wiki/Special:FilePath/USA%20orthographic.svg
http://www.wikidata.org/entity/Q31,Belgium,Brussels,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20Belgium%20%28civil%29.svg,http://commons.wikimedia.org/wiki/Special:FilePath/Europe%20location%20BEL.png
http://www.wikidata.org/entity/Q32,Luxembourg,Luxembourg,http://commons.wikimedia.org/wiki/Special:FilePath/Flag%20of%20Luxembourg.svg,http://commons.wikimedia.org/wiki/Special:FilePath/EU-Luxembourg.svg
[...]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.