1

I'm creating a full text search engine for a series of approximately 10 public domain books. The corpus will rarely be updated. Proximity search will be the core feature, e.g. locating "hello" within 5 words of "world".

I'm looking at both Haystack for Django/Python and Ruby solutions. I'm aware of: Solr, Elasticsearch, Whoosh, Xapian, and Sphinx.

With such a small corpus I'm focused on ease of implementation, and right now I'm leaning toward Whoosh (pure Python) or Sphinx (with Haystack or maybe Ruby on Rails). Thoughts?

5
  • Do you have a definition for "word A near word B"? Does "near" mean within 3 words before or after word B? Within 5 words? Is the "nearness" definable by the user?
    – Bulrush
    May 24 '16 at 18:19
  • It would be ideal if it were definable by the user. Even just "in the same chapter" would be sufficient though. Same paragraph or a specific # of words apart would be even better but not strictly required. May 24 '16 at 19:47
  • 1
    I think the feature you mean is called proximity search.
    – unor
    May 25 '16 at 2:27
  • 1
    @unor that has narrowed me down quite a bit. I might try out Whoosh after noting in its documentation that it supports proximity search, and I already have a Django setup. Thanks! May 25 '16 at 22:36
  • Actually realized Ruby might be better suited for my needs, and noted Sphinx has paragraph, sentence, and just "nearby" proximity searches, awesome! sphinxsearch.com/docs/current/extended-syntax.html Might use that. I'll answer my own question here once I really start implementing. May 27 '16 at 14:59
1

I went with Python, Flask, Whoosh, and gunicorn. It was an absolute cinch, I had it running almost immediately, and the Python nativity of Whoosh has been quite nice. Very advanced search, easy additions. Perfect for my small corpus.

I've completed and published the code for my engine. Here is a live demonstration. Read my introductory article for more details, and if I write any further articles for this project they will be listed on my site.

Here's a reference to proximity search with Whoosh's query language.

2
  • this is amazing. What files would I be looking at if I wanted to repurpose your script with a different corpus? Nov 28 '21 at 4:38
  • @StefanoMorello Thanks for asking! I actually have a lot of unpublished improvements to the repo, and just recently re-setup my dev environment for this project. I have been on a streak of publishing everything that has languished on my drive. I've updated the link in the repo About section to a Discord invite for this specific project where I may be of some help; alternatively "Watch" the repo for a future guide. 👍 Nov 29 '21 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.