Instructions to use thinkingmachines/Inkling-2-Small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use thinkingmachines/Inkling-2-Small with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="thinkingmachines/Inkling-2-Small") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("thinkingmachines/Inkling-2-Small") model = AutoModelForMultimodalLM.from_pretrained("thinkingmachines/Inkling-2-Small", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use thinkingmachines/Inkling-2-Small with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "thinkingmachines/Inkling-2-Small" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Small", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/thinkingmachines/Inkling-2-Small
- SGLang
How to use thinkingmachines/Inkling-2-Small with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Small" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Small", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Small" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Small", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use thinkingmachines/Inkling-2-Small with Docker Model Runner:
docker model run hf.co/thinkingmachines/Inkling-2-Small
Notes from an independent NVFP4 port of Inkling-2-Small
What was converted. Inkling-2-Small (594B total / 17B active, architectureinkling2) was adapted to a C++20/CUDA inference engine that targets Blackwell
only. The 367 GB checkpoint (24 safetensors shards plus a 3.4 GB MTP file that
the index does not list) becomes a 366 GB tree that is served directly from
host memory: the dense planes upload to a single RTX 5090 (12.4 GiB minimum,
14.5 GiB at the planner's preferred layout) and the 18 432 routed-expert records
stay memory-mapped on the host and stream in per token.
The artifact keeps the release's own precision — nothing is requantized. The
checkpoint is mixed-precision, and it says so in exactly one place:hf_quant_config.json. Layers 0–39 ship routed experts in NVFP4 (E2M1
nibbles, per-16 E4M3 block scales, a per-expert global scale of amax/2688);
layers 40–47 ship them in MXFP8 (E4M3 weights with a ue8m0 exponent per 32
values); everything else is BF16, except the depth-mixing queries and the router
bias, which are FP32. All of it is carried byte-for-byte.
That decision was measured, not assumed. Requantizing the eight MXFP8 layers to
4 bits — the obvious way to shrink the tree — costs 16.0 / 16.5 / 16.9 %
relative L2 on the MoE output of layers 42 / 45 / 47, against 9.5 % on the
dense projections that were already enough to push the model into a repetition
loop. So those layers keep 8 bits, decoded by a dedicated FP8 GEMV, and the tree
is 45 GB larger and twice as expensive to read on eight of the forty-eight
layers. Dense BF16 weights are widened to FP16 on the device, which is not
bit-exact: 0.33–0.40 % of them land in FP16 subnormals and 0.0001–0.0005 %
underflow to zero.
What the checkpoint forced into the engine. Inkling is not a transformer with
rotary positions. Six semantics break a generic decoder, and each of them was
verified against the release's own modelling code rather than inferred:
- No RoPE anywhere. Position enters through a learned bank of
d_rel= 16
profiles spanning 1024 backward distances: each token's relative state
projects onto the bank and gathers one bias per distance, zero beyondrel_extent. q and k are not rotated; they are L2-clipped per head to a
threshold of 22.627 (k weight-free, and the learned per-headq_l2_clip
scale is applied first), and v is RMS-normalized without a learned scale. - Attention is gated and sink-anchored. Every head carries a learned sink
logit that competes in the softmax without contributing a value, and the
attention output is multiplied bysigmoid(gate_proj(x))before the output
projection. There is no QK-norm. - Four short causal convolutions per layer, each of the form
(conv(x) + x) / sqrt(k+1)withk= 4 — one after attention, one after the
MLP, and one each on the raw k and v projections. On the eight full-attention
layers the k/v taps are dynamic: a per-token projection sums with a static
per-channel tap table, and the normalizer becomessqrt(2k+1). Getting the
order wrong (convolving the clipped k instead of clipping the convolved k) is
silent. - Depth mixing replaces a plain residual. Every layer's attention input
(layer > 0) and every MLP input is a softmax mixture over a stack of block
prefixes, with block size 6 and a position-independent query. A block
boundary (layer % 6 == 0) appends the incoming hidden state to the stack
instead of keeping a local residual, so a boundary layer's residual is the
attention output alone. The eight full-attention layers ask for three
mixtures (separate q/k/v inputs); sliding layers and MLP inputs ask for one;
the final norm input uses a dedicated output query. - MoE routing is sigmoid with a selection-only bias. 384 routed experts,
top-8, no shared expert. Scores aresigmoid(gate(x)); a learned per-expert
bias is added only to choose the top-8 (it is not part of the weight); the
chosen scores are renormalized to 1 and scaled by 8.0. The expert itself is a
softcapped SwiGLU —tanh(g/4)·4 · sigmoid(g) · tanh(u/25)·25— accumulated in
FP32, where the two soft caps are the in-repo weights' own range limiters, not
a numerical guard. - Hybrid layer types. 40 sliding-window layers (1024) and 8 full-attention
layers at ids{5, 11, 17, 23, 29, 35, 41, 47}; the full ones additionally
log-scale q and the relative states from position 128 000. A uniform stack
assumption decodes fluent garbage.
The reference used throughout was the release's own code —transformers.models.inkling2 (5.23), not the older models.inkling, which is a
different architecture (no sinks, no L2 clip, scale 1/head_dim instead ofhead_dim**-0.5, no dynamic convolutions, no depth mixing) and produces
plausible, wrong logits if copied. The engine was validated three ways: layer by
layer against an independent float64 re-derivation from the artifact, against the
release's own Inkling2TextModel loaded with real dense weights and on-demand
expert dequantization (relL2 3.7e-4 over 6 layers × 5 positions), and against
a truncated synthetic-weight build of that same class (identical top-5).
Artifact-level defects found in the checkpoint, and suggestions to the
authors. The first three are traps that break a consumer silently; the fourth
cost this ingestion a full week.
- The fused expert slab is row-interleaved, and nothing says so.
mlp.experts.w13_weight(andw13_weight_scale) stores gate and up rows
alternating —[gate₀, up₀, gate₁, up₁, …]— and must be de-interleaved before
the two projections are split. Observed failure mode: taking the first half as
gate and the second as up keeps every byte, every plane size and every
magnitude, and reproduces the recorded per-expertamaxexactly, so byte-level
verification, size checks and any reference that reads the artifact all agree
— including the release's own modelling code fed the dequantized planes. What
it destroys is the model's knowledge: mean perplexity over "The capital of
France is Paris" becomes 12 881 instead of 30.4 and the answer becomes
" the" instead of " Paris". Suggested fix: state the packing order inconfig.json(anexpert_slabfield naming the interleave) or ship the fused
tensor already de-interleaved. For reference, the release's own conversion
table expresses it as a singleInterleave(dim=1)on both tensors, which is
easy to miss because the same table renames the attention projections as plain
one-to-one renames. - The expert codec changes mid-stack, and only
hf_quant_config.jsonknows.
Layers 0–39 are NVFP4, 40–47 are MXFP8.config.json's quantization block
describes neither. A converter that assumes one codec decodes a sixth of the
stack as noise. Suggested fix: emit the per-layer codec intoconfig.json, or
split the layers into separately-named module groups. attn_res.attn_queries.{i}belongs to layeri+1, not layeri, because
layer 0 has no attention mixture at all.attn_res.mlp_queries.{i}is indexed
by layer, as expected. Two banks, both keyed by layer, both adjacent in the same
file, with different origins. Suggested fix: rename toattn_queries_of_layer_{i+1}or pad the list to 48 entries with the unused
first slot.mtp.safetensorsis a 3.4 GB file thatmodel.safetensors.index.jsondoes not
list. It holds 5 multi-token-prediction layers, correctly ignored, but a
converter that enumerates the directory rather than the index silently ingests
them, and one that trusts the index silently ignores the file — both without a
warning. Suggested fix: name it in the index with a role field, or move it out
of the checkpoint directory.eos_token_idis<|content_model_end_sampling|>(200006), not<|endoftext|>(199999). Both stop the model, so a consumer that picks the
wrong one still terminates — it just terminates somewhere else. There is nobos_token_idand no pad id. Suggested fix: document the intended stop token
in the model card; the two ids are both "end" tokens with different meanings.- The chat template appends no assistant opener. The frame ends at the user's
<|end_message|>, so a 20-id prompt asks the model to continue, and the model
emits<|message_model|><|content_thinking|>itself. This is not a defect so
much as a fact that decides whether a deployment looks broken — a harness that
expects an assistant marker in the prompt will conclude the model is looping.
Suggested fix: document the framing contract (the model writes its own
opener), or add the opener to the template behind a flag.
Measured numbers. All on the real 366 GB tree, one target at a time, RTX
5090 (32 GiB), CUDA 13.5 on a 13.4 driver:
| measurement | result |
|---|---|
| SelfCheck ("The capital of France is" → "Paris") | PASS, 3 bit-exact cycles, logits_hash=323d30463a3d2b72; the stream opens <|message_model|><|content_thinking|>The user is asking a simple factual question: |
| FSI parity gate | 3/3 (template-derived goldens — see "not measured") |
| CODE parity gate | 3/3 (template-derived goldens) |
| DET determinism gate | 3/3, three byte-identical runs per probe |
| native tool round-trip | PASS, 49 framing assertions against the checkpoint's own template |
| knowledge probe (6-token sentence, one prefill per position) | mean NLL 3.41 → perplexity 30.4, top-1 " Paris" |
| layer-by-layer parity vs an independent float64 reference | 2–5e-4 relative L2 at 7 depths spanning both codecs |
| agreement with the release's own modelling code | 3.7e-4 over 6 layers × 5 positions |
| decode throughput, 2048 tokens, auto policy | 1.47 tok/s (TTFT 14.49 s, 95.3 W average, 120.9 W peak) |
| decode throughput, 2048 tokens, strict/bit-exact | 1.46 tok/s (TTFT 14.47 s, 96.7 W average) |
| resident dense weights | 12.4 GiB minimum / 14.5 GiB preferred of a 32 GiB card |
| artifact | 366 GB on disk, 18 432 expert records + 996 pinned tensors |
| conversion | 10 min wall (367 GB read, GPU-side dequantization of the 8 FP8 layers) |
| stripped and counted, never applied | 422 vision tensors, 5 MTP layers |
Throughput is read-bound and it is the arithmetic, not a tuning failure: one token
reads 8 of 384 experts in each of 48 layers — 384 expert records, at 0.5 B/element
on layers 0–39 and 1.0 B/element on 40–47, i.e. roughly 7 GB per token against a
366 GB artifact on a 32 GiB card, so every token is a first-touch read. The stage-0
estimate written before the engine existed predicted "0.5–2 tok/s" from that
figure. Two corroborations: 95 W average on a 575 W card means the SMs idle
through each layer waiting for the next expert slab, and the auto and bit-exact
modes differ by 0.7 %, so there is almost no arithmetic left that tuning could
speed up.
What was not measured, and why.
- The three parity goldens are template-derived, not source-derived. The
594B source needs ~500 GB of aggregated VRAM across several GPUs to execute and
ships its own quantization stack, so it cannot be run on a single 32 GiB card.
The goldens therefore certify that the converted model produces the canonical
answers, not that it reproduces the source's answers. The gap is narrower than
usual here — the release's own modelling code was executed on real dense
weights and agrees with the engine to 3.7e-4 — but it is a gap and it is
recorded as one. - Long-context behaviour beyond 2048 tokens, and the sliding-window (1024)
versus full-attention split at long offsets. The perf protocol is fixed at
2048; the checkpoint's own 1M-token context is not exercisable here. The
log-scaling path that only activates from position 128 000 is consequently
untested on real weights — it is exercised only by the kernel-level
reference tests at synthetic positions. - The MTP head (5 layers, 3.4 GB). Stripped and counted; speculative decoding
is not implemented for this family. - The vision tower (422 tensors, 0.33B). Stored and counted, never applied;
this ingestion is text-only. - A distributional perplexity sweep against the FP8/BF16 source. The gates are
exact-answer probes and the knowledge probe is a single sentence; the 30.4
perplexity figure is a sanity bound, not a quality measurement. - Multi-GPU or expert-parallel serving. The engine is single-device; the
working set streams from host memory.
The full ingestion report — including the Evidence section, CTest gate targets, reference validation scripts (tools/inkling_transformers_real.py, tools/inkling_layer_ref.py), and the C++20/CUDA runtime implementation — is available in the repository:
• Full Ingestion Report: https://github.com/Alex7MV/logestix/blob/main/docs/model-reports/2026-11-15-Inkling-2-Small.md
• Repository: https://github.com/Alex7MV/logestix
The size is still too big to run locally - can we further reduce it ?
Thanks.
Good question! In practice, 366 GB is already right at the physical floor for this 594B-parameter checkpoint without breaking model quality:
- 91% of the weights are already 4-bit (NVFP4): Layers 0–39 already store routed experts at 0.5 bytes/parameter. As measured in the report, requantizing the remaining eight 8-bit (MXFP8) layers (40–47) down to 4-bit pushes MoE output error to 16.0–16.9% relative L2 and sends the model into a repetition loop. Going lower (2–3 bit) destroys reasoning and tool accuracy completely.
- Only
12.4–14.5 GiB of GPU VRAM is needed: You do not need 366 GB of VRAM—only the dense planes stay resident on the GPU (12.4 GiB minimum), while the 18,432 routed experts remain memory-mapped on host RAM / fast NVMe (7 GB streamed per token).