I have a web application that will be used to trigger Spark jobs. I am considering of using Apache Livy to do this. However, I also know that there are other options like SparkLauncher, Spark REST API, Hydrosphere Mist, and just using spark-submit. Which one do you recommend? I expect multiple users (10+) to submit Spark jobs simultaneously using the web application. Based on my research so far, it seems like there's not a real advantage to any of these. I can just pick one and use it. I just want to make sure.

I will be running pyspark code using a Spark cluster in a RedHat Linux environment.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.