Part 5: AI

20. Install the AI engine and a model#

What you'll do: create a locked ai account, download the engine and a model, and put them in folders the AI can read but not change. Time: 30–45 minutes (mostly downloading).

The folders you'll create#

/srv/ai/
  runtime/     the engine (llama-server + its libraries)   owner root, read-only for ai
  models/      the model files (.gguf)                     owner root, read-only for ai
/var/lib/ai-workspace/   the only place ai may write
/etc/ai-runtime/         settings: which model to load

Step 1: Create the ai account#

sudo adduser --system --group --no-create-home --home /var/lib/ai-workspace --shell /usr/sbin/nologin ai

This makes an account that can't log in and has no password. It only exists to run the engine.

sudo install -d -o ai -g ai -m 0700 /var/lib/ai-workspace
sudo install -d -o root -g ai -m 0750 /srv/ai /srv/ai/runtime /srv/ai/models
sudo install -d -o root -g root -m 0755 /etc/ai-runtime

Step 2: Download the engine#

The engine is llama.cpp. It's free and released several times a week.

1. On your Mac or PC, open the llama.cpp releases page on GitHub (search for "llama.cpp releases").
2. Open the newest release and look at its files. Copy the link of the file ending in -bin-ubuntu-x64.tar.gz. (It works on Debian too. Ignore the vulkan, cuda and arm64 versions.)

On the server, download and unpack it in a temporary folder (paste your link between the quotes):

mkdir -p ~/ai-download && cd ~/ai-download
curl -L -o llama.tar.gz 'PASTE-THE-LINK-HERE'
tar -xzf llama.tar.gz
find . -name llama-server

The last command prints where llama-server is, for example ./build/bin/llama-server. Copy that folder's contents (the engine and its .so library files) into place:

sudo cp -a "$(dirname "$(find . -name llama-server | head -1)")"/. /srv/ai/runtime/
sudo chown -R root:ai /srv/ai/runtime
sudo chmod -R u=rwX,g=rX,o= /srv/ai/runtime

Check that the ai account can run it:

sudo -u ai /srv/ai/runtime/llama-server --version

You should see a line with version: and a build number. Write the build number in your notebook.

Step 3: Download a model#

Models are on Hugging Face (huggingface.co), the main website for AI models. You need a file in GGUF format, in the Q4_K_M size (a good balance of quality and memory).

1. On huggingface.co, search for the model name plus GGUF, e.g. Ministral-3-3B-Instruct-2512 GGUF.
2. Open Files and versions. Click the file whose name ends in Q4_K_M.gguf.
3. Right-click download and copy the link.

On the server (use your link and the real file name):

sudo curl -L -o /srv/ai/models/Ministral-3-3B-Instruct-2512-Q4_K_M.gguf 'PASTE-THE-LINK-HERE'

Lock it so the AI can only read it:

sudo sh -c 'chown root:ai /srv/ai/models/*.gguf && chmod 0440 /srv/ai/models/*.gguf'
sudo ls -l /srv/ai/models

Each file should show -r--r----- 1 root ai and a size of 1–3 GB.

Step 4: Record a fingerprint of each model#

The fingerprint lets you check later that a file hasn't been damaged:

sudo sh -c 'cd /srv/ai/models && sha256sum *.gguf > SHA256SUMS'
sudo sh -c 'cd /srv/ai/models && sha256sum -c SHA256SUMS'

Every file should say OK. Run the second line any time to re-check.

Step 5: Tidy up#

rm -r ~/ai-download

Next: 21. Run the model as a service