I'm using windows and can't change that. Novice at programming.

Something that'd let me make pivots easily to analyze would help. Basically I have about 20gb data of userids, spend, country and so on which I want to analyze- say total spend by region and other metrics.

  • csvkit but it really depends on how comfortable you are with command line – albert Jul 10 '20 at 13:52
  • Perl or Python would seem to be a fit ... well, given you're a novice perhaps Python is the better fit, unless you're into linguistics in which case you'll want to pick Perl 😉 – 0xC0000022L Jul 13 '20 at 9:29

if you need to process these in your intranet- get a server or PC with lots of memory. Command line tools such as grep, sort and uniq -c are a good first start to do simple analyses, assuming that the data is reasonably clean.

Alternatively process datafiles in the cloud. New customers get a free tier there. Upload files to, say, Google BigQuery and process them with their other tools (e.g. Cloud Dataprep which would calculate summaries for each column, during upload).

However, you probably shouldn't do this with confidential user data where data protection laws apply.


My approach doesn't really target a "novice" user, but it it could be learned by a novice.

  1. My suggestion would be to try getting a sql database (https://www.postgresql.org/).
  2. Load the data into the database. Example: https://www.postgresqltutorial.com/import-csv-file-into-posgresql-table/
  3. Learn enough SQL to be dangerous. (https://www.w3schools.com/sql/)

Rationale: postgresql is capable of handling disk storage, compression, indexing, RAM usage, and other problems that many other software packages do not.

It's not quite programming, but just on the border of it. SQL can be scary, but still readable for simple questions. example:

FROM spend_info
GROUP BY region

It is a capability that has been around for decades, and will continue to be around, so it's a valuable skill to learn. And there's lots of tutorials out there to help along the way.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy