I have multiple Spark clusters that I maintain in case any one starts failing. I want to have something load balance job submission across the clusters depending on the health and current load of the cluster. I would also like to be able to isolate a cluster for either specific workloads or in case of the cluster having an unplanned problem.
I am working in AWS, but I am open to open source solutions to this problem.
Are there frameworks that will do this for me?