Part 5: AI
23. Chat with the model from your Mac or PC#
What you'll do: open a private tunnel from your computer to the model and chat with it in your browser. Time: 10 minutes.
How the tunnel works#
sequenceDiagram
participant B as Browser on your Mac
participant T as ssh -L (your Mac)
participant S as Server
participant M as Model 127.0.0.1:8095
B->>T: http://127.0.0.1:18095
T->>S: through the encrypted SSH connection
S->>M: arrives as "from this computer"
M-->>B: answer comes back the same wayYour browser talks to port 18095 on your own computer. SSH carries that over to port 8095 on the server. Nothing new is opened to the network.
Step 1: Turn on the built-in chat page#
The engine has a simple chat page, switched off in Chapter 21. To turn it on:
sudo nano /etc/systemd/system/llama-model.service
On the ExecStart= line, delete --no-webui at the end. Save, then:
sudo systemctl daemon-reload
sudo systemctl restart llama-model
The page is still only reachable from the server itself (and through your tunnel).
Step 2: Open the tunnel#
Use homeserver at home, or homeserver-remote when away.
ssh -N -L 18095:127.0.0.1:8095 homeserver
ssh -N -L 18095:127.0.0.1:8095 homeserver
The window will look frozen. That's correct: -N means "just hold the tunnel open". Leave it open.
Step 3: Chat#
Open your browser and go to:
http://127.0.0.1:18095
Type a question. Answers appear slowly, word by word. That's the CPU working.
Step 4: Close the tunnel#
Go back to the frozen terminal window and press Ctrl+C. The door is closed again.
bind: Address already in use
An old tunnel is still open in another window. Close it, or use another number on your side, e.g. -L 18096:127.0.0.1:8095 and then http://127.0.0.1:18096.
Answers stop in the middle of a sentence
The answer hit its length limit. In the chat page settings, raise the maximum tokens, or keep questions shorter.
Done when: you asked a question in the browser and got an answer.