Keras fails after upgrade

post by bgold on Nov 2, 2020

Hi, I recently reinstalled everything on a Lambda workstation, to upgrade to Ubuntu 20.04 and installed the latest lambda-stack. But I’ve discovered that my Keras models no longer work, failing with

AttributeError: module 'tensorflow.python.framework.ops' has no attribute '_TensorLike' .

Since this seemed similar to other problems mentioned here, I’ve tried both upgrading protobuf and adding the lines

for device in tf.config.experimental.list_physical_devices('GPU'):
    tf.config.experimental.set_memory_growth(device, True)

as suggested elsewhere. Neither have had any effect.

I’ve attached the full error output in case there’s anything useful in there

2020-11-02 10:39:36.652480: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-11-02 10:39:36.655510: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Invalid MIT-MAGIC-COOKIE-1 keyUsing TensorFlow backend.
2020-11-02 10:39:38.028832: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2020-11-02 10:39:38.038570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:1a:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2020-11-02 10:39:38.039058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:68:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2020-11-02 10:39:38.039076: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-11-02 10:39:38.041038: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-11-02 10:39:38.041774: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-11-02 10:39:38.041984: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-11-02 10:39:38.044095: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2020-11-02 10:39:38.044564: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-11-02 10:39:38.044652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-11-02 10:39:38.046506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
Traceback (most recent call last):
  File "cats.py", line 78, in <module>
    x = Dense(intermediate_dim//32, activation='relu',
  File "/usr/lib/python3/dist-packages/keras/backend/tensorflow_backend.py", line 75, in symbolic_fn_wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/keras/engine/base_layer.py", line 446, in __call__
    self.assert_input_compatibility(inputs)
  File "/usr/lib/python3/dist-packages/keras/engine/base_layer.py", line 310, in assert_input_compatibility
    K.is_keras_tensor(x)
  File "/usr/lib/python3/dist-packages/keras/backend/tensorflow_backend.py", line 695, in is_keras_tensor
    if not is_tensor(x):
  File "/usr/lib/python3/dist-packages/keras/backend/tensorflow_backend.py", line 703, in is_tensor
    return isinstance(x, tf_ops._TensorLike) or tf_ops.is_dense_tensor_like(x)
AttributeError: module 'tensorflow.python.framework.ops' has no attribute '_TensorLike'

post by bgold on Nov 9, 2020

Since this might be helpful to others, I finally chased down the source of this error.

There’s an incompatibility between the lambda-stack versions of tensorflow and Keras, but tensorflow includes it’s own keras module which can be used instead. So I changed all my imports that originally looked like

from Keras import backend as K

to instead look like

from tensorflow.keras import backend as K

and things started working again.

post by sabalaba on Nov 13, 2020

Good find — it’s because if you do import keras it’s version 2.3.1 at the time of writing and if you do import tensorflow.keras it’s version 2.4.0 which aren’t compatible. It looks like your models were 2.4.0 compatible?