1

For a project that searches textual files (mostly unstructured), I would like to use a library that can accomplish the following criteria:

  • Fast search of thousands to hundreds of thousands keywords, returning their start and end position
  • Limited search for regular expressions (potentially thousands in parallel) (It's hard to describe limited here, I'm mostly using repetitions, wildcards, optional characters)
  • Support for searching words

The search results are then processed again:

  • Arbitrary logical operations (AND, OR, NOT, combinations)
  • Distance checks (e.g. the two keywords are ANDed, but need to occur mostly N characters apart)

Optional, but would be really great:

  • Phonetic search
  • Resillience to typos

Also, I want to be able to associate certain actions with search hits.

C++ or Java would work for me. Open-source is a must.

0

The python re & regex libraries are very powerful for this sort of thing and can be combined with multiprocessing to parallelise the work.

Can I also suggest taking a look at Pythons Natural Language Toolkit nltk.

Not C++/Jave but definitely Open Source.

  • 1
    Maybe I should have been more precise: The documents themselves don't need to be processed in parallel, but the keyword search needs to be (eg. using Aho-Corasick instead of Knuth-Morris-Pratt). – a.ilchinger Jan 27 at 14:00

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.