I am doing a concept for a bigdata DB and would love to hear your advise to solve the following issue.
- fast async write calls to the db mandatory
- high horizontal scalability
- no issues with backpressure
- data are json objects
- possible to create reports / analysis over the data
To achieve these things it would be great to use the following Architecture:
Kafka -> Spark -> HDFS <- Something like Hive (SQL like language)
What does the data exactly look like which we want to store?
JSON objects with a structured static attributes and unstructured attributes:
Dynamic: the data array can contain lots of different attributes from object to object
The problem should I store the data as JSON object to HDFS? How am I able to aftwards use something like HIVE to do some SQL querries?
I know that kafka has the ability to serialize JSON objects to byte data. I do also know that HDFS is not good at it but that it is capable to store data as JSON files. But how do I use something like HIVE then to analyse the data? HIVE is not able to deserialize JSON files afaik? I have read an article ( https://blog.wikimedia.org/2017/01/13/json-hadoop-kafka/ ) about JSON being stored in HDFS - but I would not like to use a not any longer supported piece of software. Also it seems to be necessary to have at least structured JSON objects which is not the case for my data. Further I found something about HAWQ from Apache which is capable of querrying JSON data in HDFS. Can anybody recommend that setup?
Also it would be necessary for my solution to be able to query data considering the unstructured attributes in the data payload.
Should I convert the JSON objetcs in something like Avro / Parquet as these types seem to suit HDFS the best? But how to I query afterwards the data with considering the unstructured attributes of the JSON objects?
I am very thankful for any tips / hints / advises how to solve this issue! Great community! Thanks in advance and for taking time to read this post.