Software Recommendations Stack Exchange is a question and answer site for people seeking specific software recommendations. It only takes a minute to sign up.
Sign up to join this community
I need to parse a very large XML file of 30 GB into CSV. I have 15 GB of RAM available. I have been looking at some alternatives, for instance xmltodict, which has some sort of Streaming option, but this creates a dictionary, which then I am not able to load in order to save it as a CSV.
What tools are available (preferably in Python) to parse such big XML files to CSV, where it is not possible to load nor process the file because of RAM limits?
One possibility is a streaming XSLT 3.0 processor, which given your constraints means in practice Saxon/C Enterprise Edition (this has a Python language binding).
There is actually a CSV-to-XML stylesheet published as a worked example in the XSLT 3.0 specification, but sadly no counterpart to do the reverse. However, you can see the principle in some of the answers here:
https://stackoverflow.com/questions/365312/xml-to-csv-using-xslt
or here:
https://stackoverflow.com/questions/15226194/xml-to-csv-using-xslt
To make the code streamable, the key constraint is that any template rule or for-each instruction that processes a particular element can only make one traversal of the element's children. That means you can't, for example, do one pass of the source XML to discover the field names and then another pass to process the values.
Note: Saxon-EE is a commercial product and I have a commercial interest in it.
The XML Utilities library is worth a try, assuming valid & flat XML structure - it even comes with a command line xml2csv utility.
It specifically states:
xmlutils.py is a set of Python utilities for processing xml files serially for converting them to various formats (SQL, CSV, JSON). The scripts use ElementTree.iterparse() to iterate through nodes in an XML document, thus not needing to load the entire DOM into memory. The scripts can be used to churn through large XML files (albeit taking long :P) without memory hiccups.
