I am trying to figure out a method or software for updating data into HDFS real time. My requirement is as follows: I will have an Excel in a server. That Excel data needs to be loaded into HDFS. IF some data in Excel is updated, same needs to be updated in HDFS real time.

The method I have come up with: Use a Unix script to pull the Excel into HDFS, and create a hive table on top of it. Schedule a job that runs in small intervals that does the pull and replace the existing Excel each time it loads.

This method unnecessarily runs frequently and I don't feel like it's a very efficient process.

Is there any software like Kafka, Nifi that I can use here?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.