1

I want to download a portion of data from Google Ngram's corpora and run tests. The data will likely approach 500 gigs (which would almost fill up my hard drive...)

I normally use Excel VBA for this kind of thing, but the data I normally test is about 1.5 gigs or less. I start my tests by adding each word from the data into an array, and then I loop through the array to find frequencies.

But this process takes up a lot of RAM. Excel often exceeds 1 gig of ram during these processes, which would make testing 500 gigs of data impossible (500 gigs of ram!?).

I'm not at all familiar with SQL or Python, or much else, but I'd certainly be willing to learn if I need to.

What programming language would I need to generate word lists of Google's mega corpus? (And generally speaking, how can I approach this?)

2

Python is quick and easy to learn plus there are domain specific tools for what you are trying to do as a part of the sci-kit learn package including being able to extract a corpus from an on-line source filtering as you go.

It may also be worth looking at using an AWS instance to do the processing - thereby avoiding a full hard disk but read the charges page carefully. Python is also available in AWS.

| improve this answer | |
  • 1
    I also use python with sci-kit learn for such simple matters. – StefanS May 3 '16 at 13:57
0

The raw data from Google Ngram lists each ngram along with the years the ngrams occurred. Since I don't need to know the years of occurrence, I was able to shorten the file size by rewriting the file to only include the ngram and its corresponding freq.

I am doing this using google-ngram-downloader and Python (and with google-ngram-downloader, I don't even have to download any of the full files myself because it streams them on the fly).

The modified files are about 40 times less than their original size!

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.