Part 5: AI

18. The AI part: what you're building#

What you'll get: a small AI language model running on the server itself. It needs no cloud and no subscription, and nothing you ask it leaves the house.

Be realistic: an old laptop runs small models. Expect short answers that take 30 seconds to a few minutes on a dual-core CPU. It's good for searching and summarising your own documents. Don't expect ChatGPT.

The pieces#

flowchart TD
  subgraph SRV["Home server"]
    subgraph AIACC["'ai' account"]
      E["AI engine<br/>llama-server"]
      M[("Model file<br/>.gguf, read-only")]
      E --- M
    end
    R["Document search<br/>Recoll"]
    AG["Agent system<br/>(optional)"]
    R -- "passages" --> Q["ask-pages script"]
    Q -- "question + passages" --> E
    AG -- "127.0.0.1:8095" --> E
  end
  Y["You on your Mac / PC"] -- "SSH tunnel" --> E
  NET(("Internet"))
  E -. "blocked by<br/>network guard" .-x NET

What connects to what#

• The engine listens on 127.0.0.1:8095, which means this computer only. Not your Wi-Fi, not Tailscale, not the internet.
• You reach it through an SSH tunnel you open when you need it.
• The ai account can't go out to the internet, and can't read /srv/files or your home folder.

The full map, with the rest of the server, is on How the parts connect.

Before you start#

You need Chapters 1–14 done: Debian installed, SSH with keys, firewall, Tailscale. Hard drives (15–17) aren't needed; the model lives on the internal disk.

Next: 19. Check the computer can handle it