Docker data saved in persistent storage

post by addavox on Jun 23, 2023

ML Training

I know that we can store our models, training data, etc in persistent storage so that we don’t have to load it every time we spin up a new instance.

Can we do the same with Docker data (/var/lib/docker/)?

I assume we are getting our own VM but not sure if we’ll have the root privilege to symlink our /var/lib/docker dir to persistent storage.

I have not signed up with Lamba Labs yet so not having to rebuild our docker environment every time we spin up a new instance will help in the decision process.

Thanks

post by mpapili on Jun 23, 2023

In short, yes you can definitely do this and you have root access to your instances. Less of a headache than symlinking (for me, largely preference) is just telling Docker that you’ve got your own preferred data-root location:

I’ve just run through this myself on two Lambda Cloud GPU Instances just to make sure I was giving you accurate info, but if interested:

First I set up a Filesystem:

(Important to point out, Lambda notes this is a beta feature on their website. I can only answer from personal experience, but the stability/experience has been excellent to me):

Started up an Instance in the same location as my filesystem:

(A6000 instance if curious, though it does not impact this test)

# SSH in normally:
ssh ubuntu@<IP> -i <YOUR_KEY>.pem

Check that our filesystem is mounted:

ubuntu:~$ ls -l
total 0
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 18:20 TestFilesystem

Set Docker’s root to your desired filesystem (should be mounted at /home/ubuntu/<YOUR_FS_NAME>) and bounce:

echo '{ "data-root": "/home/ubuntu/TestFilesystem" }' | sudo tee /etc/docker/daemon.json && sudo systemctl restart docker

Pulling down a Docker image just to test/demonstrate:

sudo docker pull fedora
sudo docker images
REPOSITORY   TAG       IMAGE ID       CREATED      SIZE
fedora       latest    6126bb20b2b5   2 days ago   190MB

At this time I completely terminate the first instance, but of course I still have my persistent storage.

Then…

Started up a completely fresh/new instance:

ssh into THIS Instance and:

# instantly run the same one-liner as last time
echo '{ "data-root": "/home/ubuntu/TestFilesystem" }' | sudo tee /etc/docker/daemon.json && sudo systemctl restart docker

Checking and making sure our images are still present:

# are all of our images still there?
ubuntw:~$ sudo docker images
REPOSITORY   TAG       IMAGE ID       CREATED      SIZE
fedora       latest    6126bb20b2b5   2 days ago   190MB

Works like a charm. I did notice that when storing docker images in these separate filesystems you may need to set your storage driver in that Docker JSON configuration, example:

{
    "data-root":"/home/ubuntu/TestFilesystem",
    "storage-driver": "devicemapper"
}

-however I did not need to do this for the Fedora Image in this demonstration, just something of note. I’m not fully aware of the differences in the storage type/drivers these separate filesystems use.

“TL;DR:”

-but it all works for me from these basic tests without issue. All works out the box at the cost of a one-liner on-boot which sounds like it’s what you were after.

post by addavox on Jun 23, 2023

@mpapili Amazing answer!!! Thank you very much. I will be spinning up my first instance next week.