Consider adjusting OMP_NUM_THREADS environment variable for containerized deployments
OMP_NUM_THREADS environment variable for containerized
deployments, one or more libraries use Intel’s MKL that does not detect
correctly resource allocation in the cluster.
OpenMKL detects number of CPUs present on a machine where the application runs. This generally does not need to correspond to number of CPUs allocated for containerized applications. Refer to Kubernetes documentation to get more information about assigning CPU requests to a pod.
Consider also adjusting additional environment variables such as (see tensorflow/tensorflow#23238):
export TF_ENABLE_MKL_NATIVE_FORMAT=1 export TF_NUM_INTEROP_THREADS=1 export TF_NUM_INTRAOP_THREADS=4 export OMP_NUM_THREADS=4 export KMP_BLOCKTIME=1 export KMP_AFFINITY=granularity=fine,compact,1,0
Make sure the
OMP_NUM_THREADS environment variable is set and corresponds
to number of CPUs allocated for the container.
Advised manifest changes
Patch manifest for the deployed component (e.g.
DeploymentConfig or any other) so that
OMP_NUM_THREADS is set to number
of CPUs requested for the container run.