Blame for docs/QUICKSTART.md

# Run Winnow-12B GGUF

## Build the inference server

Linux/CUDA and Apple Silicon/Metal setup instructions are in the
[inference repository](https://github.com/EldanRing/winnow-inference).
The build fetches a pinned llama.cpp revision and applies Winnow's native
typed-decision integration.

```sh
git clone https://github.com/EldanRing/winnow-inference.git
cd winnow-inference
git checkout 6c2b3c04e248a319f2cb43832628eba03e55fe38
python3 scripts/build.py --jobs 8
```

## Download Q8 GGUF — recommended for 16 GB VRAM

Use the Hugging Face CLI:

```sh
hf download EldanRing/Winnow-12B \
  gguf/Winnow-12B-Q8_0.gguf gguf/mmproj-Winnow-12B.gguf SHA256SUMS \
  --local-dir models/Winnow-12B

(cd models/Winnow-12B && sha256sum --ignore-missing --check SHA256SUMS)
```

The projector is optional for text-only use. The checksum command checks the
downloaded model/projector and skips other files that were not downloaded.
On macOS, use the equivalent SHA256 checker or the repository's manifest checker.

## Start the tested 5070 Ti profile

```sh
python3 scripts/serve.py \
  --model models/Winnow-12B/gguf/Winnow-12B-Q8_0.gguf \
  --mmproj models/Winnow-12B/gguf/mmproj-Winnow-12B.gguf \
  --context 65536 --decision-parallel 4 --chat-parallel 1 \
  --cache q8_0 --memory exclusive
```

This is the tested 16 GB RTX 5070 Ti profile: full GPU weight offload,
four decision branches per wave, and one chat slot. It binds to localhost by
default. Context includes formatting, image positions, questions, and chat output.
Chat and decisions take turns using their KV contexts in this profile.

For Metal, follow the platform-specific profile in the inference repository
rather than assuming the CUDA profile's memory and cache settings apply.

## Typed decisions

```sh
curl http://127.0.0.1:8091/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Winnow-12B",
    "state": {"paid": true, "priority": "high"},
    "questions": {
      "paid": {"type": "noul", "instructions": "Has the customer paid?"},
      "route": {
        "type": "choice",
        "instructions": "Which priority is recorded?",
        "criteria": {"low": null, "high": null}
      },
      "rating": {
        "type": "score",
        "instructions": "How urgent is the recorded priority?",
        "criteria": ["not urgent", "moderately urgent", "very urgent"]
      }
    }
  }'
```

`noul` returns a probability of true. `choice` returns an option and its
distribution. `score` returns an expected zero-based category index and its
distribution. The question batch can exceed the resident branch count;
larger batches execute in waves.

## Chat and vision

```sh
curl http://127.0.0.1:8091/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Winnow-12B","messages":[{"role":"user","content":"Explain how prefix caching helps a local decision API."}],"max_tokens":4096,"stream":true}'
```

Chat uses the preserved Gemma template and the full vocabulary. Vision chat
accepts OpenAI-style image content with the projector loaded. See the
[API examples](https://github.com/EldanRing/winnow-inference/blob/main/docs/API.md)
for image inputs, authentication, request limits, and response fields.

## Download BF16 GGUF — larger memory requirement

Download this instead of Q8 if you want the 16-bit floating-point model:

```sh
hf download EldanRing/Winnow-12B \
  gguf/Winnow-12B-BF16.gguf gguf/mmproj-Winnow-12B.gguf SHA256SUMS \
  --local-dir models/Winnow-12B

(cd models/Winnow-12B && sha256sum --ignore-missing --check SHA256SUMS)
```

For text-only use, omit `gguf/mmproj-Winnow-12B.gguf` from the download command and
omit `--mmproj` when starting the server.

The BF16 model is 23.83 GB (22.20 GiB) before KV cache and runtime overhead;
it does not fit entirely in a 16 GB GPU. Use sufficient memory or CPU/GPU
offload. On a system with sufficient VRAM, a smaller-context starting profile is:

```sh
python3 scripts/serve.py \
  --model models/Winnow-12B/gguf/Winnow-12B-BF16.gguf \
  --mmproj models/Winnow-12B/gguf/mmproj-Winnow-12B.gguf \
  --context 8192 --decision-parallel 4 --chat-parallel 1 \
  --cache q8_0 --memory exclusive
```

`--cache q8_0` controls the KV cache, not the model weights; this command still
loads BF16 weights. The 64K/16 GB measurements in the model card apply to the
Q8 model, not this BF16 starting profile.

Both BF16 and Q8 downloads are **GGUF files ready for llama.cpp**. No
safetensors download, conversion, or separate adapter is required. To reproduce
a particular release, add `--revision <commit-hash>` to the download command.