I have a web application that will be used to trigger Spark jobs. I am considering of using Apache Livy to do this. However, I also know that there are other options like SparkLauncher, Spark REST API, Hydrosphere Mist, and just using spark-submit. Which one do you recommend? I expect multiple users (10+) to submit Spark jobs simultaneously using the web application. Based on my research so far, it seems like there's not a real advantage to any of these. I can just pick one and use it. I just want to make sure.
I will be running pyspark code using a Spark cluster in a RedHat Linux environment.