I have no experience (yet) working with or implementing a custom search engine for a large set of web pages, so I've come here for some recommendations. This seemed more appropriate than Computer Science or Server Fault, but feel free to migrate this if another venue would be better.

Here's the context:

I'm in the process of improving an open-source online Old Norse Dictionary into a more useful/usable form for my students in a course I teach every other year. Original version here, my modified version here.

The superficial changes were finished some time ago and implementing a useful search page is the next major step forward. I quickly abandoned custom Google Search after realizing that it just takes me to the top of the target page, not the specific location on the page and that simply doesn't work given the size of the pages (and no, I'm not interested in chopping the pages up - that's precisely what I disliked about the arrangement/interface of the original version).

In preparation for taking this next step, I've spent several months tagging all the entries with semantic tags to designate different parts of each entry as headwords, definitions, quotes, translations of quotes, citations, etymology notes, grammar information, and general discussion. Every line also has a unique id so search engine hits can go directly to the entry of interest. Full disclosure, I'm still cleaning up tags in areas that confused my automated tagging, but I have enough entries properly tagged that I want to begin implementing/testing the search engine in parallel with the arduous task of proof-reading the tags.

Now that I have a set of fully tagged html files, I need to process these into some kind of database, create a page with an interface to help users search specific types of data (e.g., only search headwords or only search quotes or only search definitions & translations, etc.) and then dynamically generate a webpage containing a list of hits with previews that link back to the specific entries.

Conceptually, I have a general idea of scripts I could write to process the html files into DB entries, etc., but I have no interest (or time or experience) to write an entire search engine from scratch. My brief search for open source search engines turned up a number of options and, frankly, I'm overwhelmed by the number of options out there because I don't have experience with implementing a search engine.

What caught my eye so far:

  • Lucene: active community, supports UTF8, presumably lots of help available if I run into problems; but, I'm afraid this may be heavier and more complex than I really need for a pile of static html pages.

  • Sphinx: active community, supports UTF8, supports morphology/synonym processing (sounds appealing for a dictionary); again, this feels like a pretty heavy/complex tool for my simple application.

  • Indri: supports UTF8, but the homepage hasn't been updated in over 7 years so I'm assuming there won't be much if any support available... The fact that it could parse HTML directly sounded appealing.

  • Zettair: light-weight, specifically for HTML, but the homepage hasn't been updated in over a decade so I'm assuming there's no support.

I'm sure there are many others, but these are the ones that caught my attention enough to put in my notes.

So, with my limited understanding, I believe my requirements are:

  1. UTF8 support - my data is all UTF8 and contains a lot of special characters, accented vowels, the occasional rune, etc. Searches will usually contain Icelandic characters like þ, ð, æ, accented vowels, etc.
  2. Alternate character mapping would be a plus - some editions use ǫ instead of ö and there's much confusion between æ and œ in the texts, etc.
  3. I like the idea of including morphology and synonyms to broaden the search (e.g., the headwords tend to be infinitives or nouns in nom. sing., but the student may be searching for some inflected form of the word). That would be a huge pain to create from scratch, so that's a very minor consideration right now, but might be useful in the future. If a particular solution already contains such files for modern Icelandic, that would be a great starting point!
  4. The source is literally just 44 html pages (about 20 MB of static, unchanging, tagged text). The text in dictionary format, so users need to be able to search one or several types of tags; search terms could be in English or Old Norse or even Latin, German, Greek, etc. for the etymological stuff.
  5. Most searches will be single words, but the ability to search for short phrases (e.g., looking up fragments of specific quotes) would be ideal.
  6. I need to be able to configure list of hits to point to the anchor id of the specific text block containing the hit (vs. simply linking to the entire page containing the hit).
  7. Ideally, I could configure a parser to directly index the html pages, but if I have to write some scripts to chew through my tagged file and regurgitate things in some specific format, I can certainly do that.
  8. Light-weight would be ideal. The server currently hosting my webpage is not something I have admin access to, so I'm stuck with whatever services are offered or whatever I can install in my local account. I can get more details on that as needed. If push comes to shove, the Scandinavian Studies dpt should be able to host my website and then I might have more flexibility.
  9. Open Source is what I'm looking. This is a course I teach outside my area, so it's basically my hobby and is being done in my spare time - it's not part of my academic responsibilities so I have no resources to throw at the project.
  10. I'm capable of learning things and digging down into the details, but I simply have no experience with this kind of project. I would vastly prefer C, C++, python to java (at least, for anything I have to fiddle with/compile myself). I have some programming background, but it's quite mixed: some VMS-Vax stuff in the 90s (that was my last time interfacing with a database...), lots of C++, a smattering of C, Python, PERL, BASH, etc. I love regex but am by no means a master of it. For my job over the past 15 years, I've really just created/modified minor scripts for processing spectroscopic data, but I took two years of CS foundation courses back in the day before changing majors and I wrote a giant pile of simulation code in C++ over the course of 2 years for a research project.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.