1

For a project that searches textual files (mostly unstructured), I would like to use a library that can accomplish the following criteria:

  • Fast search of thousands to hundreds of thousands keywords, returning their start and end position
  • Limited search for regular expressions (potentially thousands in parallel) (It's hard to describe limited here, I'm mostly using repetitions, wildcards, optional characters)
  • Support for searching words

The search results are then processed again:

  • Arbitrary logical operations (AND, OR, NOT, combinations)
  • Distance checks (e.g. the two keywords are ANDed, but need to occur mostly N characters apart)

Optional, but would be really great:

  • Phonetic search
  • Resillience to typos

Also, I want to be able to associate certain actions with search hits.

C++ or Java would work for me. Open-source is a must.

0

The python re & regex libraries are very powerful for this sort of thing and can be combined with multiprocessing to parallelise the work.

Can I also suggest taking a look at Pythons Natural Language Toolkit nltk.

Not C++/Jave but definitely Open Source.

  • 1
    Maybe I should have been more precise: The documents themselves don't need to be processed in parallel, but the keyword search needs to be (eg. using Aho-Corasick instead of Knuth-Morris-Pratt). – a.ilchinger Jan 27 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.