1

I only have 8 MB of data and 55000 items which I want to use for fuzzy autocompletion. The user base is very small and it is a research project so I don't need 100% uptime and replication. Even with debouncing, Fuse.js brings Firefox to a halt, even on an Intel i7-9700. So I want to add a backend search engine that can easily be set up as a docker container. The industry standard seems to be ElasticSearch but our single server only has 1 GB of RAM left over and it seems like total overkill. Should I still use ElasticSearch or is there something available on DockerHub that is easy to setup, scales better than Fuse.js and is not as heavy weight as ElasticSearch? The data is already on a Virtuoso SPARQL endpoint, which has a "bif:contains" index however that doesn't seem to support text that includes spaces and fuzzy search. A client side library may be an option too if it is fast enough for real time fuzzy search and does not block the main thread.

Improve this question
6
  • Maybe Qt
    – Basile Starynkevitch
    Jan 13 at 17:33
  • @BasileStarynkevitch: I followed your link but that seems to be a user interface framework, not a backend search engine. Did you mean something else?
    – Konrad Höffner
    Jan 14 at 8:17
  • 1
    "[...] our single server only has 1 GB of RAM left over..." . That is your problem.
    – Francesco Mantovani
    Jan 14 at 8:52
  • @KonradHöffner: I actually don't understand your question, and what you call "fuzzy autocompletion". Feel free to contact me by email to basile@starynkevitch.net (near Paris in France). I also don't understand what you mean by "real time"? Do you need to "autocomplete" in µs or in deciseconds?
    – Basile Starynkevitch
    Jan 14 at 9:37
  • @FrancescoMantovani: More resources would certainly help but unfortunately those are the conditions under which we we operate at the moment.
    – Konrad Höffner
    Jan 14 at 9:39

1 Answer 1

Reset to default
0

If you are a software developer and can afford developing software on Linux for several weeks (or at least days), you might consider the following approach:

Use some open source HTTP server library or framework.

In C, that would be libonion. In C++, try Wt. In Ocaml, try Ocsigen. And all are developed in Europe, so you could contact their developers (or pay support) easily.

8 megabytes of data is tiny. If you have a lot of simultaneous connections (C10K problem), you may want to store it in some database like PostGreSQL (or perhaps sqlite).

A simple backup search engine can easily be coded, using ordinary memoization techniques. Reading books like the garbage collection handbook could be helpful. You could code something which "removes" or "forgets" old queries (e.g. after a few seconds).

If you want a research prototype for a dozen of simultaneous connections, I guess that a RaspberryPi hardware is enough.

Feel free to contact me by email to basile@starynkevitch.net (near Paris in France)

As I commented, in many cases, buying more RAM is cost effective.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.