3

There are many algorithms which are based on comparative word frequency used in clustering, keyword analysis, tf-idf, etc.

Usually you need to calculate your own word frequencies from your own corpus. Very large corpora are better but of course this takes a lot of work, space, time, etc, and distracts from the task at hand.

I'm wondering if there are any Web API providers that have done all this for you and provide programmatic access to frequency data via the web.

Requirements:

  • English is a must, other languages are a big plus.
  • Gratis is better than paid, open is better than closed.
  • Optional stemming and/or lemmatization would be a plus but not required.
  • Any requirements for registration, throttling, daily limits, etc are OK.
  • Any format is OK but urlencoded and JSON are expected.
  • Unicode support is a very strong preference. (Should not blow up on words like café, naïve, etc.)
  • 1
    Sorry to say so, but this is definitely outside the scope of this side. "Software recommendations" is about recommending software, not recommending providers/services. I'm afraid that applies to your other question as well, though that might be border-line. – Izzy Feb 28 '14 at 12:40
  • 1
    If all else fails, you could still download the data for Google n-gram viewer (books.google.com/ngrams) and roll you own interface. The corpus is large. – user416 Feb 28 '14 at 12:41
  • 1
    @Izzy That's a bummer, because asking for APIs is also off-topic on stackoverflow (and maybe programmers.se and superuser.se as well, I have not checked) – user416 Feb 28 '14 at 12:42
  • 4
    Please differentiate between "asking for an API" (on-topic here) and "asking for a provider/website" (off-topic). See my answer to your question on Meta for details. Your question on Meta was searching clarification for the API part – but this question here rather belongs to the other: I'm wondering if there are any providers which maintain... – Izzy Feb 28 '14 at 12:57
  • 1
    So this site is specifically for software which includes little or no data?? That seems like quite a bizarre and unnatural distinction to me. Some kind of nitpicky way to force a view or interpretation. That would make make an awful lot of types of software off-topic including much mapping, spell checking, IMEs, flashcards, just off the top of my head. I've gotta run but this calls for a meta topic IMHO... – hippietrail Mar 1 '14 at 8:35
3

To return the relative frequency in 1 million words of the word "smartass", query:

https://api.datamuse.com/words?sp=smartass&md=f&max=1

and extract the result from the json returned, e.g. with python like (the score is NOT the count):

import requests

_wait = 0.5

def get_freq(term):
    response = None
    while True:
        try:
            response = requests.get('https://api.datamuse.com/words?sp='+term+'&md=f&max=1').json()
        except:
            print 'Could not get response. Sleep and retry...'
            time.sleep(_wait)
            continue
        break;
    freq = 0.0 if len(response)==0 else float(response[0]['tags'][0][2:])
    return freq

You can call this 100,000 times a day. It seems that this is automatically maintained if you run a single process as the response has a delay such that it comes to roughly 100k responses per day.

The counts are from the google n-gram corpus.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.