0

I have a big list of known phrases. Given a body of text (Say a tweet) I'd like to see if it contains a known phrase from a large list. I'm sure something like this must exist (perhaps from the new cognitive suits from IBM Watson/Microsoft Cortana or maybe something like Apache lucene). Unfortunately I don't know the name for this problem to identify a product that solves it :)

1

What you are trying to do is a part of Natural Language processing and one of the significant tools for this is the Python Natural Language Toolkit (nltk).

A good starting point may be found in an article & matching Jupyter notebook fuzzy-sentence-matching-python which discusses tokenization, case insensitive, stopwording, stemming, lemmatization and points towards partial matching.

The process that you will need to follow is to:

  1. Take your entries in the list and for each entry produce a tokenized, fixed case, stopworded, stemmed, lemmatized finger print from that entry.
  2. Take your tweet, or whatever, and do the same.
  3. Search your list of fingerprints scoring each for the number of matches to the tweets fingerprint.
  4. Any that score over a given threshold can then be analysed further possibly to see if shortening the tweet results in a near enough match to be considered a quote.

A book on the use of nltk is available online but it is only one of thousands of books and papers on the subject.

0

This is an ordinary search problem. You just need to combine all your known phrases into a single search term first. With regular expressions, this can be most straightforwardly accomplished with the alternation operator |. For search engines with Boolean-style operators, use OR.

To recommend a particular program (since this is Software Recommendations), try grep. You can construct a regular expression programmatically and then pass it to grep with xargs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.