I have a big list of known phrases. Given a body of text (Say a tweet) I'd like to see if it contains a known phrase from a large list. I'm sure something like this must exist (perhaps from the new cognitive suits from IBM Watson/Microsoft Cortana or maybe something like Apache lucene). Unfortunately I don't know the name for this problem to identify a product that solves it :)
A good starting point may be found in an article & matching Jupyter notebook fuzzy-sentence-matching-python which discusses tokenization, case insensitive, stopwording, stemming, lemmatization and points towards partial matching.
The process that you will need to follow is to:
- Take your entries in the list and for each entry produce a tokenized, fixed case, stopworded, stemmed, lemmatized finger print from that entry.
- Take your tweet, or whatever, and do the same.
- Search your list of fingerprints scoring each for the number of matches to the tweets fingerprint.
- Any that score over a given threshold can then be analysed further possibly to see if shortening the tweet results in a near enough match to be considered a quote.
A book on the use of nltk is available online but it is only one of thousands of books and papers on the subject.
This is an ordinary search problem. You just need to combine all your known phrases into a single search term first. With regular expressions, this can be most straightforwardly accomplished with the alternation operator
|. For search engines with Boolean-style operators, use
To recommend a particular program (since this is Software Recommendations), try
grep. You can construct a regular expression programmatically and then pass it to