I have multiple Spark clusters that I maintain in case any one starts failing. I want to have something load balance job submission across the clusters depending on the health and current load of the cluster. I would also like to be able to isolate a cluster for either specific workloads or in case of the cluster having an unplanned problem.

I am working in AWS, but I am open to open source solutions to this problem.

Are there frameworks that will do this for me?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.