I'm working with a huge CSV dataset and I'm now make some pre-processing on it. The pre-processing consists of removing lines with "not a numbers", change columns positions or remove lines. Is there any tools package or library in any programming language that performs this kind of pre-processing?

Many thanks


  • This isn't a statistical question. It is off topic here. That said, any software that can read & write individual lines (or small sets of lines) at a time could do this. That is, you would read a manageable number of lines, write those w/o NaN, & itterate. – gung Jul 26 '16 at 16:08

The ideal tool for this is a python tool called Pandas.

  • Free, Gratis & Open Source
  • Cross Platform
  • Can handle huge data sets
  • Intelligent label-based slicing, fancy indexing, and subsetting of large data sets

You could try datamash


Also importing your records into mysql.

importing into sqlite.

sqlite> .mode csv
sqlite> .import test.csv foo

if the first line of your csv file contains the column names, then you can omit then you don't have to create the table first.

Not a number SELECT * FROM myTable WHERE col1 NOT REGEXP '[0-9]+';

You can and together as complex as where statement as you need.

select *

You can use to select column names in any order, and omit or move columns at will.

select a,b,z,t,w,u from table;

Notice they are not in any order, and columns could be missing.

You can physically move columns around, but because of the way SQL works it is completely unnecessary.

ALTER TABLE Employees MODIFY COLUMN empName VARCHAR(50) AFTER department;

If you prefer a good PHP library, try Spout: a PHP library "to read and write CSV or Excel files in a fast and scalable way" (supports CSV, XLSX and ODS file formats). It's "capable of processing very large files while keeping the memory usage really low".


you can use Acho Studio for this.

  1. remove lines by using the "filter" function
  2. change the column position by using the "move column" feature
  3. then you can download the csv files for further analytics

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy