I'm looking for a way that I can convert several thousand .xml files into .csv. I've had no problems doing so in small quantities, but the data that I am dealing with is within several thousand folders. More specifically, I am looking at MLB game data over the course of a single season. Each game has its own separate folder within other folders for day and month. Within each game folder are several files, however I am only interested in one .xml file called inning_all.xml for each game. I need to convert these ~2,500 files into .csv files in order to be interpreted by Stata (stata has a built in xml import function but is incompatible with these specific files; they are able to imported after converting to .csv).
You haven't said anything about your technology constraints, but
(a) converting XML to CSV is easily done with any XSLT processor
(b) XSLT 2.0 processors such as Saxon typically have the ability to process multiple files in directory structures using the collection() or uri-collection() functions.
The number of files isn't a problem - I've processed much larger quantities than this in a minute or two, and Saxon's implementation of the collection() function is multi-threaded so it scales well.
- Free, gratis & Open Source
- Cross Platform including Windows
- Quick and easy to learn
- os.walk will "walk" a directory tree to find all of the files of interest.
- there are multiple xml parsers but you might even be better off using the super powerful regular expression processing of the re library.