1

I've a huge file in the size of around 600 MB, I've to write some python codes and generate new columns, for now, I'm using python, but unfortunately, no luck as I'm getting memory error to get through, what you'd suggest to go through all the process?

3
  • 3
    600 mb is not a big amount of file. if you use the python 'pandas' library, I'm sure you can handle all kinds of data manipulation. If you search for python pandas quick start guide, you can learn how you can load your data and make manipulations on it
    – naïveRSA
    Jul 23 '19 at 13:20
  • 1
    Now I'm getting different error, I've done some search update it, but no luck yet. UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte Jul 23 '19 at 13:55
  • 1
    import pandas as pd baseYear = pd.read_csv('2018_jun.csv') I ran this code Jul 23 '19 at 14:00
1

Maybe you need to take in account rows that are just header information. Use pd.read_csv('2018_jun.csv',skiprows=N, sep='\t') where N would be the number of rows to skip and sep='\t' if values are separated by tabs instead of commas.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.