33
So I eventually wrote a small benchmark using the libraries Steve Barnes had pointed at. I had found the same when looking for it as I was writing the question so I guess that's the main ones. Some other ideas that haven't tried yet: HDF5 for Python, PyTables, IOPro (non-free).
In short, pandas.io.parsers.read_csv beats everybody else, NumPy's loadtxt is ...
answered Jul 3 '14 at 16:46
Franck Dernoncourt
36.2k29 gold badges108 silver badges221 bronze badges
9
The 'problem' with a CSV file is that the single 'table' doesn't have a name. Therefore, you'll need to define a table (including columns) and import the CSV file before doing any SQL queries on it.
Almost all database browsers offer import from CSV; if you want a lightweight one, I can recommend SQLite Browser. It's open-source, cross platform (Windows, ...
8
I would like to contribute another library here, that I stumbled over looking for similar question. I tested it with Franck Dernoncourts benchmark code and it beats Pythons standard csv and Pandas by miles. I could not test with numpy, since i tested with a 24.000 line csv with number and string values.
This speedy library is actually based on the default ...
8
Have a look at q
q is a command line tool that allows direct execution of SQL-like queries on CSVs/TSVs (and any other tabular text files).
https://github.com/harelba/q
7
I think your mentioning of a spreadsheet is a red herring. Any data can be represented and edited as a bunch of tables, but they are almost always the wrong format; and in this case IMO limiting your solutions to tabular entry seems like it is going to be a crutch.
I believe what you actually want is a form builder and data collection app, perhaps like:
...
6
You can use OpenOffice (or LibreOffice, which is pretty much the same):
free and open source
Windows and Linux
specify the delimiter when importing the CSV:
answered Jun 9 '14 at 15:12
Franck Dernoncourt
36.2k29 gold badges108 silver badges221 bronze badges
6
For Windows, if you have Microsoft Office, you can use Excel. You have the ability to set a custom delimiter when importing text files:
5
It is a little difficult to exactly get your needs, but maybe Google docs provides what you want?
You can make a pretty elaborate form in this. You can then sent employees or clients the link to the form and they will be able to provide answers, which will be collected in a spreadsheet.
This spreadsheet has a separate access rights and you can edit is as ...
5
You have a wide choice depending on data size and complexity and what you are going to do with the resulting data:
The csv library that comes with Python by default.
NumPy - numpy.from_file function - Reads to a NumPy array so it is very powerful.
Pandas - pandas.io.parsers.read_csv function - reads to a pandas data frame, is very powerful, and can handle ...
5
I would suggest downloading the wxPython, (it is quite likely to be already installed on both Mac and Linux), and the wxPython demo and taking a look at the SimpleGrid.py in there, adding file load and save for csv is trivial given that Python comes with a comprehensive csv library and you will find how to add copy/paste/pop-up menu, etc. elsewhere in the ...
5
You can use Google Drive (free, web-based). Create a new spreadsheet, go to File > import, then specify the delimiter:
answered Jun 9 '14 at 15:10
Franck Dernoncourt
36.2k29 gold badges108 silver badges221 bronze badges
5
You could try reCsvEditor, a CSV file viewer/editor which supports a wide variety of field delimiters, very large files and Unicode Files. Files are displayed in a table format.
It seems to match all your requirements, it's written in Java, and it has a SourceForge project portal too (where the program can be downloaded from).
Here is a list of features, ...
5
I have found something amazing http://tabula.technology/ this is the best tool we have! It's also Free. It works really well with PDF files but even works fairly well with well formed tables like above that are images.
Awesome interface and great to use.
It is open source (MIT License) and the source code is available at https://github.com/tabulapdf/...
5
Python is:
Free, Gratis & Open Source including commercial use
Available for windows, along with just about everything else, although recent versions (post 2.5.4) have dropped Windows XP support and Windows 7 support has gone as of 3.9.
Can definitely be command line driven
Has a built in csv module/library that can output various dialects of csv file.
...
4
Osmo has been around for years and is pretty good - the GUI's not so flashy, but it's pretty lightweight with minimal dependencies. It has basic import/export functionality, as well as everything you'd expect from a simple address book / contact manager. http://clayo.org/osmo/
You should be able to install it via sudo aptitude install osmo on Ubuntu/Debian ...
4
This is really a job for a scripting language. The task you describe is so specific that you're very unlikely to find a tool that does exactly that. On the other hand, it can be written in a few simple lines of code.
You didn't specify an operating system, so I'm going to assume that it is POSIX-compliant. That includes Linux, OS X, FreeBSD, etc. On Windows,...
answered Mar 16 '15 at 1:16
Gilles 'SO- stop being evil'
4,8483 gold badges26 silver badges42 bronze badges
4
Take a look at the Dask library. (http://dask.pydata.org/en/latest/). It extends pandas' DataFrames and Numpy arrays for larger-than-memory computation.
Here's a blog post about using Dask with scikit-learn: https://www.continuum.io/blog/developer-blog/dask-and-scikit-learn-3-part-tutorial
4
Two libraries that fits the requirements and several of desired features:
Rapidcsv
CSVstream
Fast C++ CSV Parser
I start to test them few days ago, so I can not confirm all the features, however I can point to detais that are defining my choice:
Rapidcsv is what I'm currently using, it allows to load data as vectors, and you can obtains row-vectors, that ...
4
If you have the flexibility to use some light scripting languages try -
R script - It has sqldf library which can be used to run sql queries on dataframes. You just have to read your csv into a dataframe.
Python - pandasql library has the same functionality over pandas dataframes.
4
Using PostgreSQL and DBeaver is another option.
PostgreSQL is a sophisticated, open source database system. It allows for importing CSV files through the COPY command and through graphical UIs such as DBeaver.
DBeaver is a free and multi-platform database management tool.
In DBeaver:
Create the database table for the CSV files. Make sure the table column ...
3
You could always look at Pyspread which reports that while row & col sizes depend on memory size, etc. according to the FAQ: For standard size on GTK platforms, 80 000 000 rows can be displayed. - I would expect similar sizes in columns to be available - as a test for your use case I resized the grid to 100,000x100,000 without problems.
Price: Free
...
3
Take a look at csvkit, a python library for working with CSV data with accompanying command line scripts. It's designed to be smart about character encoding (and can be used to convert or normalize your encodings), and since it uses Python's csv module, it's pretty reliable in terms of reading and writing according to spec.
It's designed with UNIX ...
3
It's somewhat unclear how far your requirements stretch, but in general most scripting languages have a CSV parser of their own, and can be used from the command line.
If there's an error in your CSV file, a parser should find it.
For example:
echo "one, two" | ruby -r csv -e 'CSV.parse(STDIN.read)'
results in the script producing no output and an exit ...
3
I'm not entirely sure if I understand your intention correctly, but at least on Linux that should be easily done using core command line utilities like sed or awk. sed is a stream editor, which usually works line-based, and understands regular expressions, while awk is a "structured text processor".
But you don't even need those, as the Bash shell can deal ...
3
I would suggest using python with the xlrd library, the conversion could be as simple as:
from xlrd import open_workbook
import os
import sys
def xl2csv(filename):
""" Convert an excel file to csv."""
basename, ext = os.path.splitext(filename)
wb = open_workbook(filename)
for s in wb.sheets():
print 'Sheet:',s.name
outfile = ...
3
ABBYY FineReader does very good job when it comes to optical recognition. Probably the best on the market. And it can also export to *.csv, among many other formats. The downside is, it is not free and this particular version works on Windows only.
A different version for macOS exists, but it lacks many features and overall slower (subjectively) that a ...
3
Since you are in a Windows environment have you considered PowerShell? It meets all of your criteria and you already have it available (no need to install anything). This command will do what you want:
# With headers
(Get-Content -Path $pathToJsonFile | ConvertFrom-Json) `
| ConvertTo-Csv -NoTypeInformation `
| Set-Content $pathToOutputFile
# Without ...
3
You can try out the Proposal Review template on Simitless.
you can either use it as is or take it as inspiration and modify it by deleting or adding new columns (or creating new things altogether),
it takes .csv documents, so you can import and export them at any time,
for reviewers to access and make leave reviews, you (if you are the one creating the app)...
3
SQLite is a nice lightweight tool that is supported by multiple languages. If you use Python, there are several tools in the Python Package Index (pypi.org) that make it easy to use (including https://pypi.org/project/querycsv/, which I wrote). If you use R, you can use read.csv to import the data to a data frame, and use the sqldf package to query it ...
3
Try csvsql. It is part of csvsuite, a suite of command line tools written in python, since March 2019 tested against Python 3.7.
The csvkit tools are described in the book Data Science at the command line. If you pass standard input to csvsql, then the table is named stdin.
csvsql --help
Generate SQL statements for one or more CSV files, or execute ...
Only top voted, non community-wiki answers of a minimum length are eligible