TensorFlow in version 2.3 has a bug that prevents from running if multiple TensorFlow processes are active

A TensorFlow process running in the background can prevent other TensorFlow processes from running when GPU is used.

Issue description

TensorFlow in version 2.3 acquires GPU and does not release it properly for other TensorFlow processes. This can happen in Jupyter Notebooks when TensorFlow is running in multiple tabs or when a TensorFlow process is in a hanging state.

If this issue happens, TensorFlow crashes with the following error message:

tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse

Affected packages

• tensorflow==2.3
• tensorflow-gpu==2.3

Severity

• WARNING

Issue fix

Make sure only one TensorFlow process is running at a time. Close any Jupyter Notebooks that can block TensorFlow processes from GPU. If running in a console, make sure just one TensorFlow process is active at a time.

Pipeline units

• TensorFlowMultipleProcessesGPUBug

Recommendation types

• latest
• performance
• security
• stable
• testing

See this document that describes recommendation types listed.

Issues associated

• tensorflow/tensorflow#38518

Related

• tensorflow/tensorflow#38518