I am trying to figure out a method or software for updating data into HDFS real time. My requirement is as follows: I will have an Excel in a server. That Excel data needs to be loaded into HDFS. IF some data in Excel is updated, same needs to be updated in HDFS real time.
The method I have come up with: Use a Unix script to pull the Excel into HDFS, and create a hive table on top of it. Schedule a job that runs in small intervals that does the pull and replace the existing Excel each time it loads.
This method unnecessarily runs frequently and I don't feel like it's a very efficient process.
Is there any software like Kafka, Nifi that I can use here?