Run Llama 4 Scout Locally on Ubuntu – Step‑by‑Step Guide

Q: How do I install Llama 4 Scout on Ubuntu 22.04?

Start by installing system dependencies: `sudo apt update && sudo apt install -y git wget build-essential cmake`. Install NVIDIA driver 525 or newer and CUDA 12.1. Clone the repo with `git clone https://github.com/llama4/scout.git && cd scout`. Create a conda environment (`conda create -n scout python=3.10 && conda activate scout`) and install Python requirements via `pip install -r requirements.txt`. Download the model weights from the official release page (or use `wget` with your token) and place them in `models/`. Finally, set `export SCOUT_MODEL_PATH=$(pwd)/models` and run `python -m scout.server` to verify the installation.

Q: What GPU drivers and CUDA version are required for Llama 4 Scout?

Llama 4 Scout relies on the NVIDIA CUDA toolkit for GPU acceleration. You need a driver version 525 or newer that supports CUDA 12.1, which is the recommended toolkit version. Install cuDNN 8.9 or later that matches CUDA 12.1. Verify the setup with `nvidia-smi` (driver) and `nvcc --version` (CUDA). If you use a different driver/CUDA combo, ensure the compiled PyTorch binaries match; otherwise you may encounter "CUDA runtime version mismatch" errors during model loading.

Q: How can I run inference with Llama 4 Scout from a Python script?

After installing the package, import the library and load the model with the appropriate device flag. Example: ```python from scout import LlamaScout model = LlamaScout(model_path="$SCOUT_MODEL_PATH", device="cuda") prompt = "Explain quantum entanglement in simple terms." output = model.generate(prompt, max_new_tokens=150, temperature=0.7) print(output) ``` The `generate` method accepts typical parameters like `max_new_tokens`, `temperature`, `top_p`, and `stop`. Ensure the CUDA device is visible (`export CUDA_VISIBLE_DEVICES=0`) and that the model files are accessible. The call returns a string with the generated text.

Q: Is there a Docker image for running Llama 4 Scout locally?

Yes. The repository provides a `Dockerfile` that builds an image with all dependencies. Build it with `docker build -t llama4-scout .` inside the cloned repo. Then run the container, mounting the model directory and exposing the API port: ```bash docker run -d \ --gpus all \ -v $(pwd)/models:/app/models \ -p 8000:8000 \ --name scout llama4-scout ``` The container starts the HTTP server on port 8000. You can query it with `curl -X POST -d '{"prompt":"Hello"}' http://localhost:8000/generate`. Adjust the `--gpus` flag if you have multiple GPUs.

Q: How do I enable 8‑bit quantization to run Llama 4 Scout on a GPU with limited VRAM?

Llama 4 Scout integrates with the `bitsandbytes` library for 8‑bit inference. Install it in your environment (`pip install bitsandbytes`). When initializing the model, pass the `quantize="8bit"` flag: ```python model = LlamaScout( model_path=os.getenv("SCOUT_MODEL_PATH"), device="cuda", quantize="8bit" ) ``` The library automatically loads the weights in 8‑bit format, reducing VRAM usage by roughly 3‑4× while keeping quality high. For even lower memory, you can combine `load_in_4bit=True` with `bnb_4bit_compute_dtype=torch.float16`. Remember to set `torch.backends.cuda.matmul.allow_tf32 = True` for optimal performance.

Published 2026-09-05 · How-to

Install dependencies Clone repository * Run Llama 4 Scout

Prerequisites: Ubuntu version, Python, and hardware requirements

Ubuntu 22.04 LTS (or newer) is the only officially supported release. The distribution’s APT repositories contain the exact versions of libc, OpenSSL, and GPU drivers that Llama 4 Scout expects, so avoid older releases like 20.04 unless you’re prepared to pin dozens of packages manually.

Python 3.11 is the minimum; the model’s inference scripts rely on type‑hinted stdlib features that break on 3.9/3.10. Install it from the deadsnakes PPA or use pyenv to keep it isolated from the system interpreter.

Hardware is the real bottleneck. A single NVIDIA RTX 4090 (24 GiB VRAM) runs the 7‑B parameter checkpoint at full speed; anything with < 16 GiB will require aggressive --max_seq_len throttling or off‑loading to CPU, which kills performance. For CPU‑only testing, a 32‑core AMD Zen 3 or Intel Xeon Gold with at least 128 GiB RAM is the lowest practical baseline.

Component	Minimum	Recommended
OS	Ubuntu 22.04 LTS	Ubuntu 22.04 LTS
Python	3.11.x	3.11.8 (via pyenv)
GPU	NVIDIA RTX 3060 (12 GiB)	NVIDIA RTX 4090 (24 GiB)
RAM	64 GiB	128 GiB
Disk	200 GB SSD (NVMe)	500 GB NVMe SSD

Set up the environment in one go:

sudo apt update && sudo apt install -y python3.11 python3.11-venv git build-essential
python3.11 -m venv .venv && source .venv/bin/activate
pip install --upgrade pip setuptools wheel

Once these prerequisites are satisfied, the subsequent cloning and execution steps will proceed without surprise.

Install system dependencies and Python packages

Start by updating the package index and pulling in the core build tools. Ubuntu’s default repositories already contain versions that match Llama 4 Scout’s expectations, so stick to them:

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential git python3.11 python3.11-venv python3-pip \
    libopenblas-dev libssl-dev libffi-dev wget curl

If you plan to run on an NVIDIA GPU, install the driver and CUDA toolkit that match the PyTorch wheel you’ll later download. For a 12‑series card the simplest route is:

sudo apt install -y nvidia-driver-560 cuda-toolkit-12-1

After the system packages are in place, create an isolated Python environment to avoid polluting the global interpreter:

python3.11 -m venv ~/.llama4-scout-env
source ~/.llama4-scout-env/bin/activate

Upgrade pip and install the exact PyTorch build for your CUDA version (replace cu121 with cpu if you lack a GPU):

pip install --upgrade pip
pip install torch==2.2.0+cu121 -f https://download.pytorch.org/whl/torch_stable.html

Finally, pull the repository’s Python requirements. The project ships a trimmed‑down requirements.txt that pins compatible versions of transformers, accelerate, and tokenizers:

git clone https://github.com/yourorg/llama4-scout.git
cd llama4-scout
pip install -r requirements.txt

Verify the installation with a quick import test:

python -c "import torch, transformers; print(torch.__version__, transformers.__version__)"

If the versions print without error, the system dependencies and Python packages are ready for the next step.

Clone the Llama 4 Scout repository and set up a virtual environment

Pick a directory where you keep your AI projects, then pull the source with Git. The official repo lives on GitHub and includes a small submodule for the tokenizer, so clone it recursively:

# Choose a parent folder, e.g. ~/projects
mkdir -p ~/projects && cd ~/projects
git clone --depth 1 --recursive https://github.com/llama4/scout.git
cd scout

Once the code is on disk, isolate its Python stack. Ubuntu’s system Python is fine, but a clean virtual environment prevents version clashes with other projects:

python3.11 -m venv .venv          # create a hidden venv inside the repo
source .venv/bin/activate        # activate it for the current shell
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt  # install exact library versions the model expects

The requirements.txt pins PyTorch to the CUDA build that matches the driver shipped with Ubuntu 22.04. If you have a different driver, edit the line manually before running pip install. After the install finishes, verify the environment:

python -c "import torch; print('CUDA:', torch.cuda.is_available())"

A “True” output confirms the GPU stack is functional; otherwise fall back to CPU by setting export TORCH_DEVICE=cpu before launching the inference script. Keeping the virtual environment active (or re‑activating it with source .venv/bin/activate) is required for every subsequent command in this guide.

Configure model files and environment variables

Place the downloaded checkpoint files in a dedicated directory that the Scout loader can resolve without relative‑path gymnastics. A common convention is ~/llama4/models/scout/; create it and move the .safetensors and accompanying config.json there:

mkdir -p ~/llama4/models/scout
mv /path/to/downloaded/*.safetensors ~/llama4/models/scout/
mv /path/to/downloaded/config.json ~/llama4/models/scout/

Next, export the variables the runtime expects. Scout reads LLAMA_MODEL_DIR for the base path, LLAMA_CACHE_DIR for tokeniser caches, and CUDA_VISIBLE_DEVICES to pin GPUs. Add the following lines to ~/.bashrc (or your preferred shell rc file) and reload the session:

export LLAMA_MODEL_DIR=~/llama4/models/scout
export LLAMA_CACHE_DIR=~/.cache/llama4
export CUDA_VISIBLE_DEVICES=0,1   # adjust to your hardware
export HF_HUB_DISABLE_TELEMETRY=1 # silence unwanted analytics

If you are running on CPU only, set CUDA_VISIBLE_DEVICES= to an empty string and optionally limit thread count:

export CUDA_VISIBLE_DEVICES=
export OMP_NUM_THREADS=$(nproc)

Finally, verify that Scout can locate the model:

python -c "import scout; print(scout.utils.get_model_path())"

The command should echo the absolute path you defined in LLAMA_MODEL_DIR. Any mismatch will raise a clear FileNotFoundError, letting you correct the directory layout before proceeding to inference.

Run Llama 4 Scout and verify the local deployment

First, create an isolated Python environment inside the cloned repository to avoid contaminating the system Python:

python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

The requirements.txt pins Torch 2.2, transformers‑4.41, and the custom llama‑scout wheel. If you have an NVIDIA GPU, add the CUDA‑specific build:

pip install torch==2.2.0+cu121 -f https://download.pytorch.org/whl/torch_stable.html

Now launch the model server in the background. The script reads the config.yaml you edited earlier (model path, batch size, port 8000 by default):

nohup python -m llama_scout.server --config config.yaml > scout.log 2>&1 &

Give the process a few seconds to warm up, then confirm it’s listening:

ss -ltnp | grep 8000

A line like LISTEN 0 128 :8000 :* users:(("python",pid=12345,fd=5)) means the HTTP endpoint is up. Test the inference route with a minimal curl request:

curl -X POST http://127.0.0.1:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"prompt":"Hello, Llama 4 Scout!","max_tokens":32}'

You should receive a JSON payload containing a completion field. If the response includes the expected text, the deployment is functional. Check scout.log for any warnings; a clean log with no tracebacks confirms a healthy run. To stop the server, locate the PID from the ss output and kill it.

Optional: Enable GPU acceleration with CUDA

First, confirm that your GPU is supported by the CUDA 12.2 toolkit (compute capability ≥ 7.0). Run nvidia-smi; the driver version should be 525.xx or newer. If the driver is missing or outdated, install it from the Ubuntu graphics‑drivers PPA:

sudo add-apt-repository ppa:graphics-drivers/ppa -y
sudo apt update
sudo apt install -y nvidia-driver-525
reboot

After the reboot, install the CUDA toolkit directly from NVIDIA’s network installer to avoid mismatched library versions:

wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda_12.2.0_535.54.03_linux.run
chmod +x cuda_12.2.0_535.54.03_linux.run
sudo ./cuda_12.2.0_535.54.03_linux.run --silent --toolkit

Add the toolkit to your PATH and LD_LIBRARY_PATH in ~/.bashrc:

echo 'export PATH=/usr/local/cuda-12.2/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Now install PyTorch with CUDA support inside the Scout virtual environment:

pip install torch==2.3.0+cu122 torchvision==0.18.0+cu122 torchaudio==2.3.0+cu122 \
    -f https://download.pytorch.org/whl/torch_stable.html

Verify the setup by launching Python and checking torch.cuda.is_available(). If it returns True, the model will automatically off‑load tensors to the GPU when you run python -m scout.infer …. If you encounter “CUDA driver version is insufficient”, double‑check that the driver and toolkit versions match the numbers above.

Troubleshooting common installation issues

Missing libtorch or a mismatched CUDA toolkit is the most frequent cause of a failed build. Verify the version Llama 4 Scout expects (CUDA 12.1, cuDNN 8.9) and install it directly from NVIDIA’s apt repo before invoking pip install -r requirements.txt:

# Add the CUDA repository
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-1 libcudnn8-dev

If the installer aborts with ImportError: libtorch.so not found, reinstall the pre‑compiled wheels for the exact Python version you are using:

pip uninstall torch torchvision -y
pip install torch==2.3.0+cu121 torchvision==0.18.0+cu121 \
    -f https://download.pytorch.org/whl/torch_stable.html

A common Git‑related snag is an incomplete submodule checkout, which leads to FileNotFoundError: tokenizer_vocab.json. Refresh the repository recursively:

git submodule sync
git submodule update --init --recursive

Permission errors when creating the virtual environment are usually caused by a stale venv directory owned by root. Delete it and recreate the environment with a regular user:

rm -rf .venv
python3.11 -m venv .venv
source .venv/bin/activate

Symptom	Quick Fix
`torch.cuda.is_available() == False`	Confirm `nvidia-smi` shows the GPU and reinstall CUDA/ cuDNN.
`OSError: libcudnn.so.8: cannot open shared object file`	Add `/usr/lib/x86_64-linux-gnu` to `LD_LIBRARY_PATH`.
`pip install -r requirements.txt` hangs	Upgrade pip (`pip install -U pip`) and set `PIP_NO_CACHE_DIR=off`.

When you encounter a different traceback, search the exact error string in the repo’s Issues page; the maintainers usually tag the most common fixes. If the problem persists, open a new issue with the full log and your uname -a output.

FAQ

How do I install Llama 4 Scout on Ubuntu 22.04?

Start by installing system dependencies: `sudo apt update && sudo apt install -y git wget build-essential cmake`. Install NVIDIA driver 525 or newer and CUDA 12.1. Clone the repo with `git clone https://github.com/llama4/scout.git && cd scout`. Create a conda environment (`conda create -n scout python=3.10 && conda activate scout`) and install Python requirements via `pip install -r requirements.txt`. Download the model weights from the official release page (or use `wget` with your token) and place them in `models/`. Finally, set `export SCOUT_MODEL_PATH=$(pwd)/models` and run `python -m scout.server` to verify the installation.

What GPU drivers and CUDA version are required for Llama 4 Scout?

Llama 4 Scout relies on the NVIDIA CUDA toolkit for GPU acceleration. You need a driver version 525 or newer that supports CUDA 12.1, which is the recommended toolkit version. Install cuDNN 8.9 or later that matches CUDA 12.1. Verify the setup with `nvidia-smi` (driver) and `nvcc --version` (CUDA). If you use a different driver/CUDA combo, ensure the compiled PyTorch binaries match; otherwise you may encounter "CUDA runtime version mismatch" errors during model loading.

How can I run inference with Llama 4 Scout from a Python script?

After installing the package, import the library and load the model with the appropriate device flag. Example: ```python from scout import LlamaScout model = LlamaScout(model_path="$SCOUT_MODEL_PATH", device="cuda") prompt = "Explain quantum entanglement in simple terms." output = model.generate(prompt, max_new_tokens=150, temperature=0.7) print(output) ``` The `generate` method accepts typical parameters like `max_new_tokens`, `temperature`, `top_p`, and `stop`. Ensure the CUDA device is visible (`export CUDA_VISIBLE_DEVICES=0`) and that the model files are accessible. The call returns a string with the generated text.

Is there a Docker image for running Llama 4 Scout locally?

Yes. The repository provides a `Dockerfile` that builds an image with all dependencies. Build it with `docker build -t llama4-scout .` inside the cloned repo. Then run the container, mounting the model directory and exposing the API port: ```bash docker run -d \ --gpus all \ -v $(pwd)/models:/app/models \ -p 8000:8000 \ --name scout llama4-scout ``` The container starts the HTTP server on port 8000. You can query it with `curl -X POST -d '{"prompt":"Hello"}' http://localhost:8000/generate`. Adjust the `--gpus` flag if you have multiple GPUs.

How do I enable 8‑bit quantization to run Llama 4 Scout on a GPU with limited VRAM?

Llama 4 Scout integrates with the `bitsandbytes` library for 8‑bit inference. Install it in your environment (`pip install bitsandbytes`). When initializing the model, pass the `quantize="8bit"` flag: ```python model = LlamaScout( model_path=os.getenv("SCOUT_MODEL_PATH"), device="cuda", quantize="8bit" ) ``` The library automatically loads the weights in 8‑bit format, reducing VRAM usage by roughly 3‑4× while keeping quality high. For even lower memory, you can combine `load_in_4bit=True` with `bnb_4bit_compute_dtype=torch.float16`. Remember to set `torch.backends.cuda.matmul.allow_tf32 = True` for optimal performance.