I am trying to evaluate the performance of Spark running on bare-metal against one that runs in a container. I am trying to evaluate metrics like response time, CPU usage, memory usage, additional-resource-requirements etc.
I want to know whether there exist standard sizable SparkJobs written to create enough load to test its performance (in the sense of research environments) out there.
Also, what are the standard and appropriate methods to measure the above metrics? Is it via simple terminal tools like top or MAC Activity-monitor to see how much memory and CPU it's consuming? And hardcoding calculations of timing differences to measure response-time etc.
Or are there standard ways of doing this?