I am looking for the fastest Python library to read a CSV file (if that matters, 1 or 3 columns, all integers or floats, example) into a Python array (or some object that I can access in a similar fashion, with a similar access time). It should be free, work on Windows 7 and Ubuntu 12.04, and with Python 2.7 x64.

CSV with 1 column:

350
750
252
138
125
125
125
112
95
196
105
101
101
101
102
101
101
102
202
104

CSV with 3 columns:

9,52,1
52,91,0
91,135,0
135,174,0
174,218,0
218,260,0
260,301,0
301,341,0
341,383,0
383,423,0
423,466,0
466,503,0
503,547,0
547,583,0
583,629,0
629,667,0
667,713,0
713,754,0
754,796,0
796,839,1
  • 4
    Coincidentally, there is a very similar question posted 2 hours ago on Super User: superuser.com/q/775893/137286 The first answer suggests a fast library. – ComFreek Jul 2 '14 at 9:34
  • Have you tested with fastcsv, as I suggested below, lately? I would be very interested to hear how it performs with your data. Cheers, Daniel – hexerei software Mar 13 '16 at 18:51
up vote 31 down vote accepted

So I eventually wrote a small benchmark using the libraries Steve Barnes had pointed at. I had found the same when looking for it as I was writing the question so I guess that's the main ones. Some other ideas that haven't tried yet: HDF5 for Python, PyTables, IOPro (non-free).

In short, pandas.io.parsers.read_csv beats everybody else, NumPy's loadtxt is impressively slow and NumPy's from_file and load impressively fast.

Data (I should have generated them in the benchmark but I am running out of time right now)

Code:

import csv
import os
import cProfile
import time
import numpy
import pandas
import warnings

# Make sure those files in the same folder as benchmark_python.py
# As the name indicates:
# - '1col.csv' is a CSV file with 1 column
# - '3col.csv' is a CSV file with 3 column
filename1 = '1col.csv'
filename3 = '3col.csv'
csv_delimiter = ' '
debug = False

def open_with_python_csv(filename):
    '''
    https://docs.python.org/2/library/csv.html
    '''
    data =[]
    with open(filename, 'rb') as csvfile:
        csvreader = csv.reader(csvfile, delimiter=csv_delimiter, quotechar='|')
        for row in csvreader:
            data.append(row)    
    return data

def open_with_python_csv_cast_as_float(filename):
    '''
    https://docs.python.org/2/library/csv.html
    '''
    data =[]
    with open(filename, 'rb') as csvfile:
        csvreader = csv.reader(csvfile, delimiter=csv_delimiter, quotechar='|')
        for row in csvreader:
            data.append(map(float, row))    
    return data

def open_with_python_csv_list(filename):
    '''
    https://docs.python.org/2/library/csv.html
    '''
    data =[]
    with open(filename, 'rb') as csvfile:
        csvreader = csv.reader(csvfile, delimiter=csv_delimiter, quotechar='|')
        data = list(csvreader)    
    return data


def open_with_numpy_loadtxt(filename):
    '''
    http://stackoverflow.com/questions/4315506/load-csv-into-2d-matrix-with-numpy-for-plotting
    '''
    data = numpy.loadtxt(open(filename,'rb'),delimiter=csv_delimiter,skiprows=0)
    return data

def open_with_pandas_read_csv(filename):
    df = pandas.read_csv(filename, sep=csv_delimiter)
    data = df.values
    return data    


def benchmark(function_name):  
    start_time = time.clock()
    data = function_name(filename1)       
    if debug: print data[0] 
    data = function_name(filename3)
    if debug: print data[0]
    print function_name.__name__ + ': ' + str(time.clock() - start_time), "seconds"


def benchmark_numpy_fromfile():
    '''
    http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromfile.html
    Do not rely on the combination of tofile and fromfile for data storage, 
    as the binary files generated are are not platform independent.
    In particular, no byte-order or data-type information is saved.
    Data can be stored in the platform independent .npy format using
    save and load instead.

    Note that fromfile will create a one-dimensional array containing your data,
    so you might need to reshape it afterward.
    '''
    #ignore the 'tmpnam is a potential security risk to your program' warning
    with warnings.catch_warnings():
        warnings.simplefilter('ignore', RuntimeWarning)
        fname1 = os.tmpnam()
        fname3 = os.tmpnam()

    data = open_with_numpy_loadtxt(filename1)
    if debug: print data[0]
    data.tofile(fname1)
    data = open_with_numpy_loadtxt(filename3)
    if debug: print data[0]
    data.tofile(fname3)
    if debug: print data.shape
    fname3shape = data.shape
    start_time = time.clock()
    data = numpy.fromfile(fname1, dtype=numpy.float64) # you might need to switch to float32. List of types: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
    if debug: print len(data), data[0], data.shape
    data = numpy.fromfile(fname3, dtype=numpy.float64)
    data = data.reshape(fname3shape)
    if debug: print len(data), data[0], data.shape    
    print 'Numpy fromfile: ' + str(time.clock() - start_time), "seconds"

def benchmark_numpy_save_load():
    '''
    http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromfile.html
    Do not rely on the combination of tofile and fromfile for data storage, 
    as the binary files generated are are not platform independent.
    In particular, no byte-order or data-type information is saved.
    Data can be stored in the platform independent .npy format using
    save and load instead.

    Note that fromfile will create a one-dimensional array containing your data,
    so you might need to reshape it afterward.
    '''
    #ignore the 'tmpnam is a potential security risk to your program' warning
    with warnings.catch_warnings():
        warnings.simplefilter('ignore', RuntimeWarning)
        fname1 = os.tmpnam()
        fname3 = os.tmpnam()

    data = open_with_numpy_loadtxt(filename1)
    if debug: print data[0]    
    numpy.save(fname1, data)    
    data = open_with_numpy_loadtxt(filename3)
    if debug: print data[0]    
    numpy.save(fname3, data)    
    if debug: print data.shape
    fname3shape = data.shape
    start_time = time.clock()
    data = numpy.load(fname1 + '.npy')
    if debug: print len(data), data[0], data.shape
    data = numpy.load(fname3 + '.npy')
    #data = data.reshape(fname3shape)
    if debug: print len(data), data[0], data.shape    
    print 'Numpy load: ' + str(time.clock() - start_time), "seconds"


def main():
    number_of_runs = 20
    results = []

    benchmark_functions = ['benchmark(open_with_python_csv)', 
                           'benchmark(open_with_python_csv_list)',
                           'benchmark(open_with_python_csv_cast_as_float)',
                           'benchmark(open_with_numpy_loadtxt)',
                           'benchmark(open_with_pandas_read_csv)',
                           'benchmark_numpy_fromfile()',
                           'benchmark_numpy_save_load()']
    # Compute benchmark
    for run_number in range(number_of_runs):
        run_results = []
        for benchmark_function in benchmark_functions:
            run_results.append(eval(benchmark_function))
            results.append(run_results)

    # Display benchmark's results
    print results
    results = numpy.array(results)
    numpy.set_printoptions(precision=10) # http://stackoverflow.com/questions/2891790/pretty-printing-of-numpy-array
    numpy.set_printoptions(suppress=True)  # suppress suppresses the use of scientific notation for small numbers:
    print numpy.mean(results, axis=0)
    print numpy.std(results, axis=0)    

    #Another library, but not free: https://store.continuum.io/cshop/iopro/

if __name__ == "__main__":
    #cProfile.run('main()') # if you want to do some profiling
    main()  

Windows 7:

Output:

open_with_python_csv: 1.57318865672 seconds
open_with_python_csv_list: 1.35567931732 seconds
open_with_python_csv_cast_as_float: 3.0801260484 seconds
open_with_numpy_loadtxt: 14.4942111801 seconds
open_with_pandas_read_csv: 0.371965476805 seconds
Numpy fromfile: 0.0130216095713 seconds
Numpy load: 0.0245501650124 seconds

To install all libraries: Unofficial Windows Binaries for Python Extension Packages

Windows configuration:


Ubuntu 12.04:

Output:

open_with_python_csv: 1.93 seconds
open_with_python_csv_list: 1.52 seconds
open_with_python_csv_cast_as_float: 3.19 seconds
open_with_numpy_loadtxt: 7.47 seconds
open_with_pandas_read_csv: 0.35 seconds
Numpy fromfile: 0.01 seconds
Numpy load: 0.02 seconds

To install all libraries:

sudo apt-get install python-pip
sudo pip install numpy
sudo pip install pandas

If libraries are already installed but need to be upgraded:

sudo apt-get install python-pip
sudo pip install numpy --upgrade
sudo pip install pandas --upgrade

Ubuntu configuration:

  • Ubuntu 12.04 x64
  • Python 2.7.3
  • NumPy 1.8.1 (import numpy; numpy.version.version)
  • Pandas 0.14.0 (import pandas as pd; pd.__version__)

Obviously feel free to improve the benchmark by commenting/editing/etc, I'm sure about that there are plenty of things to enhance:

  • Making sure that the current loading functions are well optimized
  • Try new functions / libraries such as HDF5 for Python, PyTables, IOPro (non-free).
  • Generate the CSV in the benchmark (so that one doesn't have to download the CSV files)
  • 1
    An interesting comparison of results. – Steve Barnes Jul 3 '14 at 18:17
  • 3
    you saved me from the dark and lonely world of numpy.loadtxt – zkurtz Jan 5 '15 at 16:32
  • 2
    Great answer. FYI I ran your benchmark on a 2gB cvs file: open_with_python_csv: 153.789445 seconds open_with_python_csv_list: 146.709768 seconds open_with_python_csv_cast_as_float: 86.957046 seconds open_with_numpy_loadtxt: 157.669637 seconds open_with_pandas_read_csv: 77.612668 seconds – M.R. Jun 15 '15 at 19:38
  • This is great and I've learned a lot this evening from this post. However your np.fromfile seems awfully fast. What I found in my tests is that it just works on the first line, so to do the whole file you have to do a loop which slows things down again to 60s for my 20k line csv file, vs. 0.75s for Pandas. Are you reading the entire file with np.fromfile? I was also able to use np.fromstring by loading the entire csv file, replaceing the '\n with '' then running np.fromstring. The string manipulations were fast, but converting to numbers was slow. This method took 2.6s. – Sonicsmooth Jun 1 '17 at 6:50
  • Trying this in 2018 on windows 10 with python 2.7.13 with a 100000 row file with 19 columns just testing the open_with_python_csv, open_with_python_csv_list and open_with_pandas_read_csv and the pandas method is not faster. Built in csv means are ~0.35 and pandas ~0.43. – Davos Mar 19 at 13:24

I would like to contribute another library here, that I stumbled over looking for similar question. I tested it with Franck Dernoncourts benchmark code and it beats Pythons standard csv and Pandas by miles. I could not test with numpy, since i tested with a 24.000 line csv with number and string values.

This speedy library is actually based on the default csv implementation, just using TextIO which makes it faster AND handles unicode strings correctly.

It is named fastcsv and was developed by Masaya Suzuki. You can close it in GitHub or use Pypi to install. Simplest is:

pip install fastcsv

On http://pythonhosted.org/fastcsv/ you can see more Benchmark results, but for just reading csv let me repeat their results here:

Reading Benchmark using fastcsv

Would be interesting to know, how this performs with your data.

  • 2
    "TLDR: This is deprecated. Use Python3 and the standard csv module. So, I don't recommend using this library." - Author's note? – Sebastian Palma Mar 10 at 13:39
  • @SebastianPalma Original question specifies python 2.7 – Davos Mar 19 at 12:56

You have a wide choice depending on data size and complexity and what you are going to do with the resulting data:

  1. The csv library that comes with Python by default.
  2. NumPy - numpy.from_file function - Reads to a NumPy array so it is very powerful.
  3. Pandas - pandas.io.parsers.read_csv function - reads to a pandas data frame, is very powerful, and can handle huge data sets.

The first will probably be faster to import while the others are more powerful. All are free & cross platform. The first is already part of your Python installation if you have a default one.

There is a new pydatatable package which has very fast csv reader based on R data.table fread implementation.
Read more at https://github.com/h2oai/datatable If you want to have pandas object loaded you can simply run

pandas_dataframe = dt.fread(srcfile).to_pandas()

There is a new Python package for data mining which names DaPy. Which has a simple I/O API and fast enough for you. According to the performance test from the author, DaPy spent 12.5 seconds on loading a csv file with more than 2 million records, while pandas spent 4 seconds. However, DaPy is based on some python native data structures and more easily to use.

cmd: pip install DaPy 
>>> import DaPy as dp
>>> data = dp.read('file.csv')
>>> data.show()

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.