I want to download a portion of data from Google Ngram's corpora and run tests. The data will likely approach 500 gigs (which would almost fill up my hard drive...)
I normally use Excel VBA for this kind of thing, but the data I normally test is about 1.5 gigs or less. I start my tests by adding each word from the data into an array, and then I loop through the array to find frequencies.
But this process takes up a lot of RAM. Excel often exceeds 1 gig of ram during these processes, which would make testing 500 gigs of data impossible (500 gigs of ram!?).
I'm not at all familiar with SQL or Python, or much else, but I'd certainly be willing to learn if I need to.
What programming language would I need to generate word lists of Google's mega corpus? (And generally speaking, how can I approach this?)