Import Environment YAML to Cloud GPU via conda

post by mad on Jun 14, 2023

Is it possible to import a conda environment to a GPU instance on the Lambda Cloud? If not, what is the best way to include common packages available through conda on a cloud instance?
We essentially rely on some cheminformatics / data science packages like RDKit and SciKit-Learn. What would be the best way to import our workflow to a GPU instance and run it?

1.3k views

post by mpapili on Jun 14, 2023

There are a number of possible reasons that you’d want to be doing this so I’ll try and cover an example for how to tackle each of them:

Scenario1: You just want to install common packages on the Lambda GPU Instance similar to your local machine:

This one is very straightforward. On your local machine, be sure to have your conda env activated and..

# export env requirements
conda env export > environment.yml
# upload your requirements
scp -i <YOUR_KEY>.pem environment.yml ubuntu@<INSTANCE_IP>:/home/ubuntu/environment.yml
# set up your env
ssh ubuntu@<YOUR_IP> -i <YOUR_KEY>.pem "/opt/miniconda/bin/conda init && /opt/miniconda/bin/conda env create -f environment.yml"

Lambda GPU Instances seem to have a miniconda install, so be sure to save yourself some time and typing and run:

alias conda=/opt/miniconda/bin/conda

If this was me and I needed to do this frequently (akin to the concept of User Data on something like AWS) I would set up a little setup.sh bash script with the above to save me some time.

Scenario2: You have some binaries or local modules that you can’t get off of the open-web from a Lambda GPU Instance and just want to drop your EXACT Conda Env onto a Lambda GPU Instance:

I run into this scenario quite often personally. A benefit of Conda over, say, virtualenv, is that binary files that your Python modules are bundled into the environment itself. This means that the whole thing is pseudo-portable and if you have a similar-enough system (x86 CPU and ideally Ubuntu) then this might work for you:

In this example I’ve made a Conda Env with two modules:

1. numpy → commonly-used module that requires multiple binaries to operate
2. requests → I’ve modified its __init__.py to call a little “hey, it works!” Go binary to represent a piece of the environment that perhaps your team/company provides which Lambda GPU Cloud would obviously not have access to in the environment.yml method above

With your local Conda environment activated:

# install conda-pack and pack your whole Conda Environment into a tarball
conda install -n base conda-pack
conda pack
# send it to your Lambda GPU Instance
scp -i <YOUR_KEY>.pem <YOUR_ENV>.tar.gz  ubuntu@<INSTANCE_IP>:/home/ubuntu/<YOUR_ENV>.tar.gz

then hop onto your GPU instance and run:

# extract it to your new install location
mkdir <YOUR_ENV_NAME> && cd <YOUR_ENV_NAME>
tar -xzvf ../<YOUR_ENV_NAME>.tar.gz
# activate your new env
source <YOUR_ENV_NAME>/bin/activate
# set your PYTHONPATH
export PYTHONPATH=$(pwd)/lib/python3.8/site-packages/

…test out our semi-portable environment with custom binaries:

Python 3.8.16 (default, Jun 12 2023, 18:09:05)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
Hey, the binary works!
>>> # works
>>> import numpy
>>> # (works)

once again, the above can be made into a very quick setup.sh bash script that lives on your local machine to simulate a quick-setup User-Data type feature with on-demand instances. I’ll disclaim that there may be a few pathing issues with this method, but test it out and see if it works for your needs.

Scenario3: You need a combination of the above and quite a bit of additional setup:

For anything more complicated than this I’d recommend looking into The Lambda Stack Docker Image. The instructions are incredibly easy and if you set up a simple Dockerfile for your project you can use the instance’s GPU(s) within the container which will be set up for you ready-to-go every time.

Hope somewhere in this comment I covered what you were looking for!! Usually when renting a GPU Instance that requires a specific environment every time I’ll always keep myself a little handy setup.sh script to get it ready quickly each time. Let me know if something like this works for you and your team’s workflow!

post by mad on Jun 15, 2023

This is exactly what I was looking for, thank you!!