I only have 8 MB of data and 55000 items which I want to use for fuzzy autocompletion. The user base is very small and it is a research project so I don't need 100% uptime and replication. Even with debouncing, Fuse.js brings Firefox to a halt, even on an Intel i7-9700. So I want to add a backend search engine that can easily be set up as a docker container. The industry standard seems to be ElasticSearch but our single server only has 1 GB of RAM left over and it seems like total overkill. Should I still use ElasticSearch or is there something available on DockerHub that is easy to setup, scales better than Fuse.js and is not as heavy weight as ElasticSearch? The data is already on a Virtuoso SPARQL endpoint, which has a "bif:contains" index however that doesn't seem to support text that includes spaces and fuzzy search. A client side library may be an option too if it is fast enough for real time fuzzy search and does not block the main thread.
1 Answer
If you are a software developer and can afford developing software on Linux for several weeks (or at least days), you might consider the following approach:
Use some open source HTTP server library or framework.
In C, that would be libonion. In C++, try Wt. In Ocaml, try Ocsigen. And all are developed in Europe, so you could contact their developers (or pay support) easily.
8 megabytes of data is tiny. If you have a lot of simultaneous connections (C10K problem), you may want to store it in some database like PostGreSQL (or perhaps sqlite).
A simple backup search engine can easily be coded, using ordinary memoization techniques. Reading books like the garbage collection handbook could be helpful. You could code something which "removes" or "forgets" old queries (e.g. after a few seconds).
If you want a research prototype for a dozen of simultaneous connections, I guess that a RaspberryPi hardware is enough.
Feel free to contact me by email to basile@starynkevitch.net
(near Paris in France)
As I commented, in many cases, buying more RAM is cost effective.
basile@starynkevitch.net
(near Paris in France). I also don't understand what you mean by "real time"? Do you need to "autocomplete" in µs or in deciseconds?