3

I'm working with a huge CSV dataset and I'm now make some pre-processing on it. The pre-processing consists of removing lines with "not a numbers", change columns positions or remove lines. Is there any tools package or library in any programming language that performs this kind of pre-processing?

Many thanks

Richard

migrated from stats.stackexchange.com Jul 26 '16 at 17:39

This question came from our site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.

  • This isn't a statistical question. It is off topic here. That said, any software that can read & write individual lines (or small sets of lines) at a time could do this. That is, you would read a manageable number of lines, write those w/o NaN, & itterate. – gung Jul 26 '16 at 16:08
1

The ideal tool for this is a python tool called Pandas.

  • Free, Gratis & Open Source
  • Cross Platform
  • Can handle huge data sets
  • Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
1

You could try datamash

https://www.gnu.org/software/datamash/manual/datamash.html

Also importing your records into mysql.

importing into sqlite.

sqlite> .mode csv
sqlite> .import test.csv foo

if the first line of your csv file contains the column names, then you can omit then you don't have to create the table first.

Not a number SELECT * FROM myTable WHERE col1 NOT REGEXP '[0-9]+';

You can and together as complex as where statement as you need.

select *

You can use to select column names in any order, and omit or move columns at will.

select a,b,z,t,w,u from table;

Notice they are not in any order, and columns could be missing.

You can physically move columns around, but because of the way SQL works it is completely unnecessary.

ALTER TABLE Employees MODIFY COLUMN empName VARCHAR(50) AFTER department;
0

If you prefer a good PHP library, try Spout: a PHP library "to read and write CSV or Excel files in a fast and scalable way" (supports CSV, XLSX and ODS file formats). It's "capable of processing very large files while keeping the memory usage really low".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy