7

I want to self-host a search engine that indexes my static website(s).

I don’t want to use a CMS (which typically have a search engine built-in) for these sites, and I don’t want to refer my visitors to a third-party search engine service.

It doesn’t have to be a user-friendly search engine (like Google, Bing, etc.), it’s mostly for power users, so a complex search syntax may be required.

Requirements

Formalities:

  • The search engine must be FLOSS.

  • It must run on a GNU/Linux server.

  • It must not use Java (I know there are several good search engine projects in Java, but, unfortunately, my host doesn’t support it).

Backend:

  • The search engine must index (X)HTML5. Support for other formats is not needed.

  • I don’t want the crawler to find my pages on its own. Instead, I want to provide a list of URLs which should be crawled (ideally supporting one of the sitemaps.org formats).

  • I don’t want to add metadata about the documents somewhere else than in the documents themselves.

Frontend:

  • The search engine must not require JavaScript (except for optional features).
  • It must not set cookies (unless the user explicitly submits a settings form or similar).

My wish list

  • Indexing: In addition to the full text, it should index as many signals as possible (e.g., in the form of name-value pairs), like meta tags, RDFa/JSON-LD, semantic elements, etc.

  • SERP: I want to be able to define how the results should look like, depending on the indexed data. Similar to Google’s Rich Snippets. For example: show an image, a list, a short table, etc.

  • Ranking: I’d like to be able to tweak the ranking algorithm, e.g., giving each field a certain score/priority.

  • Search operators: It should, at least, support the Boolean AND/OR/NOT and brackets (e.g., (laptop OR notebook) (review OR reviews) -netbook). The more operators, the better (phrase/range/proximity search; field-based search; special characters, case-sensitivity; etc.).

Improve this question
2
  • No Java? No JavaScript? How is it going to run? Am I missing something? Sorry if that sounds rude, it isn't meant to. I am probably just being dumb, as usual :-) I presume you checked SourceForge, etc ... P.s it is supposed to run in a browser, isn't it?
    – Mawg says reinstate Monica
    Jun 2, 2017 at 7:59
  • 1
    @Mawg: Backend: Instead of Java it can use any other programming language commonly available on Web servers, e.g., PHP, Python, Ruby, or Perl. --- Frontend: There should be no need for JavaScript -- it’s just a plain HTML form for the search field, and a plain HTML list for the results. --- And yes, it’s supposed to be used in the browser -- I’ll integrate it in my website.
    – unor
    Jun 2, 2017 at 15:08

2 Answers 2

Reset to default
0

What about good old ht:dig ?

The last release is from 2004, so I'm not sure how it indexes the new elements introduced by HTML5.

Support for non-html text formats (PDF, DOCX...) is also suboptimal, but that was not on your list of requirements.

Improve this answer
0

I also recommend sphinx - see sphinxsearch.com

Sphinx is an open source full text search server, designed from the ground up with performance, relevance (aka search quality), and integration simplicity in mind. It's written in C++ and works on Linux (RedHat, Ubuntu, etc), Windows, MacOS, Solaris, FreeBSD, and a few other systems.

Sphinx lets you either batch index and search data stored in an SQL database, NoSQL storage, or just files quickly and easily — or index and search data on the fly, working with Sphinx pretty much as with a database server.

I have only good experiences uning it on Linux and Windows.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.