Consider adjusting OMP_NUM_THREADS environment variable for containerized deployments

Consider adjusting OMP_NUM_THREADS environment variable for containerized deployments, one or more libraries use Intel’s MKL that does not detect correctly resource allocation in the cluster.

Issue description

OpenMKL detects number of CPUs present on a machine where the application runs. This generally does not need to correspond to number of CPUs allocated for containerized applications. Refer to Kubernetes documentation to get more information about assigning CPU requests to a pod.

Consider also adjusting additional environment variables such as (see tensorflow/tensorflow#23238):

export TF_ENABLE_MKL_NATIVE_FORMAT=1  
export TF_NUM_INTEROP_THREADS=1
export TF_NUM_INTRAOP_THREADS=4
export OMP_NUM_THREADS=4
export KMP_BLOCKTIME=1
export KMP_AFFINITY=granularity=fine,compact,1,0

Affected packages

intel-tensorflow
pytorch

Severity

WARNING

Issue fix

Make sure the OMP_NUM_THREADS environment variable is set and corresponds to number of CPUs allocated for the container.

Pipeline units

MKLThreadsWrap

Recommendation types

latest
performance
secure
stable
testing

See this document that describes recommendation types listed.

Advised manifest changes

Patch manifest for the deployed component (e.g. Job, Deployment, DeploymentConfig or any other) so that OMP_NUM_THREADS is set to number of CPUs requested for the container run.