4

There are many algorithms which are based on comparative word frequency used in clustering, keyword analysis, tf-idf, etc.

Usually you need to calculate your own word frequencies from your own corpus. Very large corpora are better but of course this takes a lot of work, space, time, etc, and distracts from the task at hand.

I'm wondering if there are any Web API providers that have done all this for you and provide programmatic access to frequency data via the web.

Requirements:

  • English is a must, other languages are a big plus.
  • Gratis is better than paid, open is better than closed.
  • Optional stemming and/or lemmatization would be a plus but not required.
  • Any requirements for registration, throttling, daily limits, etc are OK.
  • Any format is OK but urlencoded and JSON are expected.
  • Unicode support is a very strong preference. (Should not blow up on words like café, naïve, etc.)
Improve this question
10
  • 1
    Sorry to say so, but this is definitely outside the scope of this side. "Software recommendations" is about recommending software, not recommending providers/services. I'm afraid that applies to your other question as well, though that might be border-line.
    – Izzy ♦
    Feb 28, 2014 at 12:40
  • 1
    If all else fails, you could still download the data for Google n-gram viewer (books.google.com/ngrams) and roll you own interface. The corpus is large.
    – user416
    Feb 28, 2014 at 12:41
  • 1
    @Izzy That's a bummer, because asking for APIs is also off-topic on stackoverflow (and maybe programmers.se and superuser.se as well, I have not checked)
    – user416
    Feb 28, 2014 at 12:42
  • 4
    Please differentiate between "asking for an API" (on-topic here) and "asking for a provider/website" (off-topic). See my answer to your question on Meta for details. Your question on Meta was searching clarification for the API part – but this question here rather belongs to the other: I'm wondering if there are any providers which maintain...
    – Izzy ♦
    Feb 28, 2014 at 12:57
  • 1
    So this site is specifically for software which includes little or no data?? That seems like quite a bizarre and unnatural distinction to me. Some kind of nitpicky way to force a view or interpretation. That would make make an awful lot of types of software off-topic including much mapping, spell checking, IMEs, flashcards, just off the top of my head. I've gotta run but this calls for a meta topic IMHO...
    – hippietrail
    Mar 1, 2014 at 8:35

1 Answer 1

Reset to default
4

To return the relative frequency in 1 million words of the word "smartass", query:

https://api.datamuse.com/words?sp=smartass&md=f&max=1

It outputs:

[{"word":"smartass","score":129630,"tags":["f:0.067229"]}]

Extract the result from the json returned, e.g. with python like (the score is NOT the count):

import requests

_wait = 0.5

def get_freq(term):
    response = None
    while True:
        try:
            response = requests.get('https://api.datamuse.com/words?sp='+term+'&md=f&max=1').json()
        except:
            print 'Could not get response. Sleep and retry...'
            time.sleep(_wait)
            continue
        break;
    freq = 0.0 if len(response)==0 else float(response[0]['tags'][0][2:])
    return freq

You can call this 100,000 times a day. It seems that this is automatically maintained if you run a single process as the response has a delay such that it comes to roughly 100k responses per day.

The counts are from the google n-gram corpus.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.