12

I am looking for any really free alternatives for implementing an intranet web-search engine.

I know that Google Search Appliance (GSA) would be probably the best but it's extremely expensive and I do want to be able to crawl millions of pages.

I tried SearchBlox and in addition to the fact that I found it poorly documented and counter-intuive it also have a limit of 25.000 documents, which is almost nothing compared to the level I want to scale it to. Maybe if it was better I would have considered upgrading to commercial license but based on the experience, I wouldn't pay for it.

So, now I am looking for other approaches

  • Have you seen Sphinx? – leventov Feb 11 '14 at 19:57
  • 1
    Any specific requirements? See this meta for what is considered good question. – danijelc Feb 11 '14 at 19:59
  • What sources do you want to crawl from? CMIS? WebDAV (network shared drive)? Local disk? Intranet websites? – Nicolas Raoul Feb 12 '14 at 9:34
8

Solr can do this. With it you can define a datasource to crawl it and it interprets the data. Solr is gratis and open source.

Solr has an extremely fast search engine (Lucene) and can import a lot of data. It is optimized for "field like" data like XML, JSON or HTML but there are converters of everything that contains text (like Word, PDF, etc.)

If you just need a few intranet sites (<100.000) to be indexed AND you can access the database you should just set up the Data Import Handler to do the work for you.

Else you will need to code some own thing that sends the data to the server.

Keep in mind that this is a development tool and not an end user program. You will need to create some interface. (It is quite easy, in my experience)

If you store the information in various fields (like meta, title, url, content, language, filetype...) you can search through this fields specifically, thous having the possibility to narrow your search down. If for example all your sites in the intranet have an author and you can access that field and index it, you can search for all documents that are from this author, but ignore all that are about him. It also supports fuzzy search ("seach" finding "search")

I used Solr in one (and the underling search engine Lucene in many) projects and was very impressed by it. The high flexibility in the data processing engines are incredible. The searching part is so fast that I have it on my list to one day understand how it works :)

If all you need is a search crawler & search interface, then the configuration overhead of Solr might not be what you need. But if you need a tool that chews through 30.000.000 documents than this is the tool to go. In the project I used it (with said amount of documents) we had more trouble with network latency then with the Solr search time. You can replicate the index and use a load balancing Solr instance that distributives the search requests to the other ones. And and and... the amount of different optimizations of this tool are staggering. That of course comes with a bit of necessary configuration that might not be super intuitive.

As hinted above, Solr is a wrapper around Lucene so if you already have a CMS to do the site creation for you there might be already a Lucene plugin for it that you can tap into.

  • 2
    Do you have any experience with Solr/Lucene? Care to elaborate? – Olli Feb 12 '14 at 13:34
  • @Olli Yes. I updated the answer. Thanks for the reminder :) – Angelo Fuchs Feb 12 '14 at 17:59
2

Sphider or (better) Sphider Plus

So far I've found them both great. They are pretty much the same except that Sphider Plus is a thousand times better - think of the original as a 'lite' version. This is really a rec. for Sphider plus and that's the feature set I'll be primarily talking about.

  • Requires PHP 5 or 6.
  • Database (MySQL - IIRC can also use a few other DB Engines)

Features to match your requirements:

  • no maximum number of files to index
  • Indexes HTML, and through built in converters can index PDF, DOCX, XLSX, ODT, ODS, CSV and XLS files.
  • Quite reasonably priced (25 EURO)
  • powerful admin interface
  • quite a bit of automation (ie auto-reindexing via CRON etc.)
  • Can it index HTML? The op looked for something that indexed his intranet. – Angelo Fuchs Feb 12 '14 at 11:34
  • 1
    Yeah that's what it is designed for. It appears that I didn't explicitly state that. – Nick Wilde Feb 12 '14 at 15:49
  • Sorry, but the latest release was more than an year ago and the website looks like was made in '90s... – sorin Jan 6 '15 at 13:05
  • @sorin: I have to agree the website is a little ugly. However there was a release today of Sphider Plus - which as I said was the version/fork I was mostly talking about. – Nick Wilde Jan 6 '15 at 15:34
2

Duckduckgo is an Internet crawler and search engine.

Here is its source code: https://github.com/duckduckgo

You can compile it, configure it to crawl only your intranet websites, and host it on a server in your intranet.

That being said, crawling web interfaces is not very efficient, you can do much better (with Solr for instance) if you have back access to where the content is actually stored. This will also allow you to show different results to people who have different access rights (for instance, only HR people can search resumes).

  • 3
    Did you do that? Have you used it? Whats your experience? Could you tell us a bit more about it? (I didn't know you could do that with Duckduckgo, so I'm quite interested) – Angelo Fuchs Feb 12 '14 at 11:35
  • A link to an article that would briefly document how to install your own instance would be more helpful. There are so many repositories there that it makes impossible to know where to start evaluating it as an intranet search engine. – sorin Jan 6 '15 at 13:12
0

It sounds like you'll be needing to crawl over the available sites as well? A free web crawler I've set up is Heritrix. It is open source and is what the Internet Archive's "Wayback Machine" runs on.

https://webarchive.jira.com/wiki/display/Heritrix/Heritrix

You would then need I suppose need to combine it with something like Solr to do the actual indexing for the searches.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.