Speedup a TensorFlow Serving container on Azure

I have deployed a TensorFlow model on Microsoft Azure using a TensorFlow Serving container with Docker. I also have a script that starting from an image send the request to the container for the prediction. The script that send the request is running on my local PC, if I host the container in local it takes 0.7 seconds, on Azure it takes 5-12 seconds. I have to speedup the process on Azure and definitely is not only a problem of network. If you think it's necessary you can also change the script or the architecture in order to process the request directly on the container.