12

I am looking for any really free alternatives for implementing an intranet web-search engine.

I know that Google Search Appliance (GSA) would be probably the best but it's extremely expensive and I do want to be able to crawl millions of pages.

I tried SearchBlox and in addition to the fact that I found it poorly documented and counter-intuive it also have a limit of 25.000 documents, which is almost nothing compared to the level I want to scale it to. Maybe if it was better I would have considered upgrading to commercial license but based on the experience, I wouldn't pay for it.

So, now I am looking for other approaches

3
  • Have you seen Sphinx?
    – leventov
    Commented Feb 11, 2014 at 19:57
  • 1
    Any specific requirements? See this meta for what is considered good question.
    – danijelc
    Commented Feb 11, 2014 at 19:59
  • What sources do you want to crawl from? CMIS? WebDAV (network shared drive)? Local disk? Intranet websites?
    – Nicolas Raoul
    Commented Feb 12, 2014 at 9:34

4 Answers 4

8

Solr can do this. With it you can define a datasource to crawl it and it interprets the data. Solr is gratis and open source.

Solr has an extremely fast search engine (Lucene) and can import a lot of data. It is optimized for "field like" data like XML, JSON or HTML but there are converters of everything that contains text (like Word, PDF, etc.)

If you just need a few intranet sites (<100.000) to be indexed AND you can access the database you should just set up the Data Import Handler to do the work for you.

Else you will need to code some own thing that sends the data to the server.

Keep in mind that this is a development tool and not an end user program. You will need to create some interface. (It is quite easy, in my experience)

If you store the information in various fields (like meta, title, url, content, language, filetype...) you can search through this fields specifically, thous having the possibility to narrow your search down. If for example all your sites in the intranet have an author and you can access that field and index it, you can search for all documents that are from this author, but ignore all that are about him. It also supports fuzzy search ("seach" finding "search")

I used Solr in one (and the underling search engine Lucene in many) projects and was very impressed by it. The high flexibility in the data processing engines are incredible. The searching part is so fast that I have it on my list to one day understand how it works :)

If all you need is a search crawler & search interface, then the configuration overhead of Solr might not be what you need. But if you need a tool that chews through 30.000.000 documents than this is the tool to go. In the project I used it (with said amount of documents) we had more trouble with network latency then with the Solr search time. You can replicate the index and use a load balancing Solr instance that distributives the search requests to the other ones. And and and... the amount of different optimizations of this tool are staggering. That of course comes with a bit of necessary configuration that might not be super intuitive.

As hinted above, Solr is a wrapper around Lucene so if you already have a CMS to do the site creation for you there might be already a Lucene plugin for it that you can tap into.

2
  • 2
    Do you have any experience with Solr/Lucene? Care to elaborate?
    – Olli
    Commented Feb 12, 2014 at 13:34
  • @Olli Yes. I updated the answer. Thanks for the reminder :) Commented Feb 12, 2014 at 17:59
4

Duckduckgo is an Internet crawler and search engine.

Here is its source code: https://github.com/duckduckgo

You can compile it, configure it to crawl only your intranet websites, and host it on a server in your intranet.

That being said, crawling web interfaces is not very efficient, you can do much better (with Solr for instance) if you have back access to where the content is actually stored. This will also allow you to show different results to people who have different access rights (for instance, only HR people can search resumes).

3
  • 3
    Did you do that? Have you used it? Whats your experience? Could you tell us a bit more about it? (I didn't know you could do that with Duckduckgo, so I'm quite interested) Commented Feb 12, 2014 at 11:35
  • A link to an article that would briefly document how to install your own instance would be more helpful. There are so many repositories there that it makes impossible to know where to start evaluating it as an intranet search engine.
    – sorin
    Commented Jan 6, 2015 at 13:12
  • I could not find the source code for the search engine on the Github profile you provided. There are dozens of random repositories for extensions and stuff, but nothing about the actual DuckDuckGo website.
    – Kotlopou
    Commented Jul 15, 2023 at 12:27
2

Sphider or (better) Sphider Plus

So far I've found them both great. They are pretty much the same except that Sphider Plus is a thousand times better - think of the original as a 'lite' version. This is really a rec. for Sphider plus and that's the feature set I'll be primarily talking about.

  • Requires PHP 5 or 6.
  • Database (MySQL - IIRC can also use a few other DB Engines)

Features to match your requirements:

  • no maximum number of files to index
  • Indexes HTML, and through built in converters can index PDF, DOCX, XLSX, ODT, ODS, CSV and XLS files.
  • Quite reasonably priced (25 EURO)
  • powerful admin interface
  • quite a bit of automation (ie auto-reindexing via CRON etc.)
4
  • Can it index HTML? The op looked for something that indexed his intranet. Commented Feb 12, 2014 at 11:34
  • 1
    Yeah that's what it is designed for. It appears that I didn't explicitly state that. Commented Feb 12, 2014 at 15:49
  • Sorry, but the latest release was more than an year ago and the website looks like was made in '90s...
    – sorin
    Commented Jan 6, 2015 at 13:05
  • @sorin: I have to agree the website is a little ugly. However there was a release today of Sphider Plus - which as I said was the version/fork I was mostly talking about. Commented Jan 6, 2015 at 15:34
0

It sounds like you'll be needing to crawl over the available sites as well? A free web crawler I've set up is Heritrix. It is open source and is what the Internet Archive's "Wayback Machine" runs on.

https://webarchive.jira.com/wiki/display/Heritrix/Heritrix

You would then need I suppose need to combine it with something like Solr to do the actual indexing for the searches.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.