Library for deep learning on images with multiple GPUs

Question

There are many libraries available for deep learning these days:

• Keras (A wrapper around Tensorflow, CNTK, and Theano)
• TensorFlow (from Google)
• CNTK (from Microsoft)
• Theano (discontinued late 2017)
• Caffe2
• MXNet

Which of these libraries supports multiple GPUs the best? In particular, we are looking to train and run deep neural networks like VGG16.

In order of importance the features are:

1. Supports deep learning and application of existing deep networks
2. Maximizes GPU usage of multiple GPUs
3. Minimizes developer time when developing new deep networks with typical architectures (convolutional layers, dropout layers, etc.)