I have a semi-structured data in JSON format. I say semi-structured because some fields may or may not be present or maybe nested based on the availability of some data.

Say the data format is,

title: "Dummy",
date: "13-10-2019",
a: string-value,
x: int-value,
y: string-value,
z: int-value,

I have millions of such records and each record's size varies from 10Kb to 10Mb. I cannot store them in a nosql or a sql database since some of the string values are unstructured logs of about 10-20 lines and some other fields are some fixed integers within a given range.

I need to be able to search and retrieve any field as fast as possible. Say If I give the value of x and I should be able to get all the dates with that x value or values of all the y's with that corresponding x's value.

Note: I don't want to get entire JSON record, I just want to get the value of one field given the value of another field. But the data is huge so indexing each and every field may not be possible. And also the software needs to have a good Java API.

So I need a software recommendation for a search and retrieval engine. I am currently considering solr and ELK stack(elasticsearch). I am open to other suggestions as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.