# Using Multi-Instance GPU (MIG) [NVIDIA Multi-Instance GPU](https://www.nvidia.com/en-us/technologies/multi-instance-gpu/), or MIG, allows you to partition your GPUs into isolated instances. MIG enables you to run simultaneous workloads on a single GPU. For example, you can run inference on multiple models at the same time. Tip [See NVIDIA's MIG User Guide to learn more about MIG](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/). In this tutorial, you'll learn how to: - Enable MIG on a 1x GH200 on-demand instance. - Partition the GH200 GPU into two instances. - Run a vLLM Docker container on each instance, with each container serving a different model. - Interact with the two different models. ## Prerequisites For this tutorial, you'll need a: - 1x GH200 on-demand instance. You can launch the 1x GH200 on-demand instance from the console or using the Cloud API. See [Creating and managing instances > Launching instances](https://docs.lambda.ai/public-cloud/on-demand/creating-managing-instances/#launching-instances). - [Hugging Face User Access Token](https://huggingface.co/docs/hub/en/security-tokens). ## Enable MIG on your on-demand instance 1. SSH into your 1x GH200 on-demand instance. 2. Enable MIG: ``` sudo nvidia-smi -mig 1 ``` You'll see: ``` Enabled MIG Mode for GPU 00000000:DD:00.0 All done. ``` ## Partition the GH200 GPU into two instances 1. Partition the GH200 GPU into two instances (MIG devices): ``` sudo nvidia-smi mig -cgi 9,9 -C ``` You'll see: ``` Successfully created GPU instance ID 2 on GPU 0 using profile MIG 3g.48gb (ID 9) Successfully created compute instance ID 0 on GPU 0 GPU instance ID 2 using profile MIG 3g.48gb (ID 2) Successfully created GPU instance ID 1 on GPU 0 using profile MIG 3g.48gb (ID 9) Successfully created compute instance ID 0 on GPU 0 GPU instance ID 1 using profile MIG 3g.48gb (ID 2) ``` 2. Verify that the GH200 GPU's 96 GB of VRAM has been partitioned into two instances (MIG devices): ``` nvidia-smi ``` The output should show two devices, each with 48 GB of VRAM: ``` Fri Dec 6 06:46:18 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.127.05 Driver Version: 550.127.05 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GH200 480GB On | 00000000:DD:00.0 Off | On | | N/A 31C P0 86W / 700W | 76MiB / 97871MiB | N/A Default | | | | Enabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | MIG devices: | +------------------+----------------------------------+-----------+-----------------------+ | GPU GI CI MIG | Memory-Usage | Vol| Shared | | ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG | | | | ECC| | |==================+==================================+===========+=======================| | 0 1 0 0 | 38MiB / 47616MiB | 60 0 | 3 0 3 0 3 | | | 0MiB / 0MiB | | | +------------------+----------------------------------+-----------+-----------------------+ | 0 2 0 1 | 38MiB / 47616MiB | 60 0 | 3 0 3 0 3 | | | 0MiB / 0MiB | | | +------------------+----------------------------------+-----------+-----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ``` ## Run a vLLM Docker container on each instance 1. Pull the [`drikster80/vllm-gh200-openai`](https://hub.docker.com/r/drikster80/vllm-gh200-openai) Docker image: ``` sudo docker pull drikster80/vllm-gh200-openai ``` Note The `drikster80/vllm-gh200-openai` Docker image is identical to the official [`vllm/vllm-openai`](https://hub.docker.com/r/vllm/vllm-openai) image, except support for GH200 has been added. 2. Start a `tmux` session: ``` tmux ``` 3. Launch a vLLM container serving the [Nous Research's Hermes 3](https://nousresearch.com/hermes3/) model: ``` sudo docker run \ --gpus '"device=0:0"' \ --ipc=host \ -p 8000:8000 \ drikster80/vllm-gh200-openai --model NousResearch/Hermes-3-Llama-3.1-8B --disable-log-requests ``` 4. Press `Ctrl` + `B`, then press `%` to create a new vertical pane in the `tmux` window. Then, launch a vLLM container serving the [Meta Llama 3.1 8B Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) model by running the command below. Replace `` with your Hugging Face User Access Token. ``` sudo docker run \ --gpus '"device=0:1"' \ --ipc=host \ -p 8001:8000 \ -e "HUGGING_FACE_HUB_TOKEN=" \ drikster80/vllm-gh200-openai --model meta-llama/Meta-Llama-3.1-8B-Instruct --disable-log-requests ``` ## Interact with the two different models 1. Press `Ctrl` + `B`, then press `C` to open a new `tmux` window. 2. Install `jq`: ``` sudo apt -y install jq ``` 3. Submit an example prompt to the `NousResearch/Hermes-3-Llama-3.1-8B` model: ``` curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{\n "prompt": "What is the origin of the universe?",\n "model": "NousResearch/Hermes-3-Llama-3.1-8B",\n "temperature": 0.0,\n "max_tokens": 1000,\n "min_tokens": 1000\n }' | jq . ``` 4. Submit an example prompt to the `meta-llama/Meta-Llama-3.1-8B-Instruct` model: ``` curl http://localhost:8001/v1/completions \ -H "Content-Type: application/json" \ -d '{\n "prompt": "What is the origin of the universe?",\n "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",\n "temperature": 0.0,\n "max_tokens": 1000,\n "min_tokens": 1000\n }' | jq . ``` ## Next steps - Check out the [Lambda Inference API](https://lambda.ai/inference) for serverless access to LLMs. - [Learn how to serve LLMs on a Kubernetes cluster using KubeAI](https://docs.lambda.ai/education/large-language-models/kubeai-hermes-3/).