Cannot import protobuf

post by automata on Jan 18, 2019

We’re planning to use our Lambda Quad for training as part of one of our Google Cloud Dataflow pipelines. In this way, we have google-cloud as dependency, to be able to pull/push image blobs to GCS, together with apache-beam, to be able to actually run our TensorFlow-based models as Dataflow pipelines.

However, we are not able to import protobuf (in a Python 2.7 environment, because apache-beam unfortunately doesn’t support Python 3 yet):

Traceback (most recent call last):
  File "floorplan.py", line 10, in <module>
    import apache_beam as beam
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/__init__.py", line 91, in <module>
    from apache_beam import coders
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/coders/__init__.py", line 19, in <module>
    from apache_beam.coders.coders import *
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/coders/coders.py", line 28, in <module>
    import google.protobuf.wrappers_pb2
ImportError: No module named protobuf.wrappers_pb2

We tried to pip install protobuf and google packages, but we got:

Requirement already satisfied: protobuf in /usr/lib/python2.7/dist-packages

When we try to import protobuf directly, it says that’s not installed.

I believe that is related with Lambda Stack and protobuf being installed from APT package python-protobuf.

Is there any workaround to satisfy apache-beam requirements for protobuf?

Thanks in advance.

post by sabalaba on Jan 21, 2019

Are you in a virtual environment? Are you including system site packages? Can you email software@lambdalabs.com to report this?