I am trying to evaluate the performance of Spark running on bare-metal against one that runs in a container. I am trying to evaluate metrics like response time, CPU usage, memory usage, additional-resource-requirements etc.

I want to know whether there exist standard sizable SparkJobs written to create enough load to test its performance (in the sense of research environments) out there.

Also, what are the standard and appropriate methods to measure the above metrics? Is it via simple terminal tools like top or MAC Activity-monitor to see how much memory and CPU it's consuming? And hardcoding calculations of timing differences to measure response-time etc.

Or are there standard ways of doing this?


  • Perhaps github.com/databricks/spark-perf? But, why do you think there would be a performance difference? Containers share the kernel with the host OS, so there isn't much of a difference between a process running on the host and the same program running in a container. – Emmanuel Rosa May 7 '17 at 19:19
  • @EmmanuelRosa - Thanks for the link. Well true that there isn't much difference. But still, there is a difference. Need to evaluate it in order to verify how much of a throughput loss to expect when containerizing applications similar to Spark. – Shabirmean May 8 '17 at 2:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.