I've a huge file in the size of around 600 MB, I've to write some python codes and generate new columns, for now, I'm using python, but unfortunately, no luck as I'm getting memory error to get through, what you'd suggest to go through all the process?
3
-
3600 mb is not a big amount of file. if you use the python 'pandas' library, I'm sure you can handle all kinds of data manipulation. If you search for python pandas quick start guide, you can learn how you can load your data and make manipulations on it– naïveRSACommented Jul 23, 2019 at 13:20
-
1Now I'm getting different error, I've done some search update it, but no luck yet. UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte– Elmir TürkəlCommented Jul 23, 2019 at 13:55
-
1import pandas as pd baseYear = pd.read_csv('2018_jun.csv') I ran this code– Elmir TürkəlCommented Jul 23, 2019 at 14:00
Add a comment
|
1 Answer
Maybe you need to take in account rows that are just header information. Use pd.read_csv('2018_jun.csv',skiprows=N, sep='\t') where N would be the number of rows to skip and sep='\t' if values are separated by tabs instead of commas.
