Instructions to use EldanRing/Winnow-12B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use EldanRing/Winnow-12B with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf EldanRing/Winnow-12B:BF16
# Run inference directly in the terminal:
llama cli -hf EldanRing/Winnow-12B:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf EldanRing/Winnow-12B:BF16
# Run inference directly in the terminal:
llama cli -hf EldanRing/Winnow-12B:BF16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf EldanRing/Winnow-12B:BF16
# Run inference directly in the terminal:
./llama-cli -hf EldanRing/Winnow-12B:BF16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf EldanRing/Winnow-12B:BF16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf EldanRing/Winnow-12B:BF16

Use Docker

docker model run hf.co/EldanRing/Winnow-12B:BF16

LM Studio
Jan

vLLM

How to use EldanRing/Winnow-12B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EldanRing/Winnow-12B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EldanRing/Winnow-12B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/EldanRing/Winnow-12B:BF16

Ollama
How to use EldanRing/Winnow-12B with Ollama:
```
ollama run hf.co/EldanRing/Winnow-12B:BF16
```
Unsloth Desktop

How to use EldanRing/Winnow-12B with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EldanRing/Winnow-12B:BF16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "EldanRing/Winnow-12B:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use EldanRing/Winnow-12B with Docker Model Runner:
```
docker model run hf.co/EldanRing/Winnow-12B:BF16
```

Lemonade

How to use EldanRing/Winnow-12B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull EldanRing/Winnow-12B:BF16

Run and chat with the model

lemonade run user.Winnow-12B-BF16

List all available models

lemonade list

Hermes Agent

How to use EldanRing/Winnow-12B with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EldanRing/Winnow-12B:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default EldanRing/Winnow-12B:BF16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use EldanRing/Winnow-12B with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EldanRing/Winnow-12B:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "EldanRing/Winnow-12B:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Winnow-12B / docs /QUICKSTART.md

EldanRing

Separate the vision projector from model quantization choices

b6ac22b verified 14 days ago

preview code

Raw History Blame Contribute Delete

4.6 kB

Run Winnow-12B GGUF

Build the inference server

Linux/CUDA and Apple Silicon/Metal setup instructions are in the inference repository. The build fetches a pinned llama.cpp revision and applies Winnow's native typed-decision integration.

git clone https://github.com/EldanRing/winnow-inference.git
cd winnow-inference
git checkout 6c2b3c04e248a319f2cb43832628eba03e55fe38
python3 scripts/build.py --jobs 8

Download Q8 GGUF — recommended for 16 GB VRAM

Use the Hugging Face CLI:

hf download EldanRing/Winnow-12B \
  gguf/Winnow-12B-Q8_0.gguf gguf/mmproj-Winnow-12B.gguf SHA256SUMS \
  --local-dir models/Winnow-12B

(cd models/Winnow-12B && sha256sum --ignore-missing --check SHA256SUMS)

The projector is optional for text-only use. The checksum command checks the downloaded model/projector and skips other files that were not downloaded. On macOS, use the equivalent SHA256 checker or the repository's manifest checker.

Start the tested 5070 Ti profile

python3 scripts/serve.py \
  --model models/Winnow-12B/gguf/Winnow-12B-Q8_0.gguf \
  --mmproj models/Winnow-12B/gguf/mmproj-Winnow-12B.gguf \
  --context 65536 --decision-parallel 4 --chat-parallel 1 \
  --cache q8_0 --memory exclusive

This is the tested 16 GB RTX 5070 Ti profile: full GPU weight offload, four decision branches per wave, and one chat slot. It binds to localhost by default. Context includes formatting, image positions, questions, and chat output. Chat and decisions take turns using their KV contexts in this profile.

For Metal, follow the platform-specific profile in the inference repository rather than assuming the CUDA profile's memory and cache settings apply.

Typed decisions

curl http://127.0.0.1:8091/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Winnow-12B",
    "state": {"paid": true, "priority": "high"},
    "questions": {
      "paid": {"type": "noul", "instructions": "Has the customer paid?"},
      "route": {
        "type": "choice",
        "instructions": "Which priority is recorded?",
        "criteria": {"low": null, "high": null}
      },
      "rating": {
        "type": "score",
        "instructions": "How urgent is the recorded priority?",
        "criteria": ["not urgent", "moderately urgent", "very urgent"]
      }
    }
  }'

noul returns a probability of true. choice returns an option and its distribution. score returns an expected zero-based category index and its distribution. The question batch can exceed the resident branch count; larger batches execute in waves.

Chat and vision

curl http://127.0.0.1:8091/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Winnow-12B","messages":[{"role":"user","content":"Explain how prefix caching helps a local decision API."}],"max_tokens":4096,"stream":true}'

Chat uses the preserved Gemma template and the full vocabulary. Vision chat accepts OpenAI-style image content with the projector loaded. See the API examples for image inputs, authentication, request limits, and response fields.

Download BF16 GGUF — larger memory requirement

Download this instead of Q8 if you want the 16-bit floating-point model:

hf download EldanRing/Winnow-12B \
  gguf/Winnow-12B-BF16.gguf gguf/mmproj-Winnow-12B.gguf SHA256SUMS \
  --local-dir models/Winnow-12B

(cd models/Winnow-12B && sha256sum --ignore-missing --check SHA256SUMS)

For text-only use, omit gguf/mmproj-Winnow-12B.gguf from the download command and omit --mmproj when starting the server.

The BF16 model is 23.83 GB (22.20 GiB) before KV cache and runtime overhead; it does not fit entirely in a 16 GB GPU. Use sufficient memory or CPU/GPU offload. On a system with sufficient VRAM, a smaller-context starting profile is:

python3 scripts/serve.py \
  --model models/Winnow-12B/gguf/Winnow-12B-BF16.gguf \
  --mmproj models/Winnow-12B/gguf/mmproj-Winnow-12B.gguf \
  --context 8192 --decision-parallel 4 --chat-parallel 1 \
  --cache q8_0 --memory exclusive

--cache q8_0 controls the KV cache, not the model weights; this command still loads BF16 weights. The 64K/16 GB measurements in the model card apply to the Q8 model, not this BF16 starting profile.

Both BF16 and Q8 downloads are GGUF files ready for llama.cpp. No safetensors download, conversion, or separate adapter is required. To reproduce a particular release, add --revision <commit-hash> to the download command.