I have a website with public and private sections. The private sections are available to signed-in users, and some only to paying users. I would like to index all pages and PDF assets and provide a search box. I could run that crawler on the local or development server, with all assets and pages open, and create an index. A search box would let a visitor search for a keyword and see if they'd get access to it by registering or buying.

I have seen Algolia on some websites and it does provide a crawler, but it is only for the public section of a website. I could also create the records myself and push them to Algolia, but doing so in my case is a lot of work because of the amount of information on the website and in different forms, and if I do that I might as well do a little more and code a simple search engine based on TF-IDF.

Does Algolia or a competitor provide a crawler to create an index on public and private sections of a website?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.