SAELens
English
sparse-autoencoder
SAE
interpretability
deception-detection
mechanistic-interpretability
neuronpedia
behavioral-sampling
llama
Instructions to use Solshine/deception-saes-tinyllama-1-1b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- SAELens
How to use Solshine/deception-saes-tinyllama-1-1b with SAELens:
# pip install sae-lens from sae_lens import SAE sae, cfg_dict, sparsity = SAE.from_pretrained( release = "RELEASE_ID", # e.g., "gpt2-small-res-jb". See other options in https://github.com/jbloomAus/SAELens/blob/main/sae_lens/pretrained_saes.yaml sae_id = "SAE_ID", # e.g., "blocks.8.hook_resid_pre". Won't always be a hook point ) - Notebooks
- Google Colab
- Kaggle
Commit ·
0bcb3e6
0
Parent(s):
Initial public release: SAE weights, cfg, and model card
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- .gitattributes +35 -0
- README.md +292 -0
- tinyllama_jumprelu_L12_deceptive_only/cfg.json +23 -0
- tinyllama_jumprelu_L12_deceptive_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L12_honest_only/cfg.json +23 -0
- tinyllama_jumprelu_L12_honest_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L12_mixed/cfg.json +23 -0
- tinyllama_jumprelu_L12_mixed/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L15_deceptive_only/cfg.json +23 -0
- tinyllama_jumprelu_L15_deceptive_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L15_honest_only/cfg.json +23 -0
- tinyllama_jumprelu_L15_honest_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L15_mixed/cfg.json +23 -0
- tinyllama_jumprelu_L15_mixed/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L3_deceptive_only/cfg.json +23 -0
- tinyllama_jumprelu_L3_deceptive_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L3_honest_only/cfg.json +23 -0
- tinyllama_jumprelu_L3_honest_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L3_mixed/cfg.json +23 -0
- tinyllama_jumprelu_L3_mixed/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L6_deceptive_only/cfg.json +23 -0
- tinyllama_jumprelu_L6_deceptive_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L6_honest_only/cfg.json +23 -0
- tinyllama_jumprelu_L6_honest_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L6_mixed/cfg.json +23 -0
- tinyllama_jumprelu_L6_mixed/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L9_deceptive_only/cfg.json +23 -0
- tinyllama_jumprelu_L9_deceptive_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L9_honest_only/cfg.json +23 -0
- tinyllama_jumprelu_L9_honest_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_L9_mixed/cfg.json +23 -0
- tinyllama_jumprelu_L9_mixed/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L12_deceptive_only/cfg.json +23 -0
- tinyllama_jumprelu_ste_L12_deceptive_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L12_honest_only/cfg.json +23 -0
- tinyllama_jumprelu_ste_L12_honest_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L12_mixed/cfg.json +23 -0
- tinyllama_jumprelu_ste_L12_mixed/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L15_deceptive_only/cfg.json +23 -0
- tinyllama_jumprelu_ste_L15_deceptive_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L15_honest_only/cfg.json +23 -0
- tinyllama_jumprelu_ste_L15_honest_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L15_mixed/cfg.json +23 -0
- tinyllama_jumprelu_ste_L15_mixed/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L9_deceptive_only/cfg.json +23 -0
- tinyllama_jumprelu_ste_L9_deceptive_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L9_honest_only/cfg.json +23 -0
- tinyllama_jumprelu_ste_L9_honest_only/sae_weights.safetensors +3 -0
- tinyllama_jumprelu_ste_L9_mixed/cfg.json +23 -0
- tinyllama_jumprelu_ste_L9_mixed/sae_weights.safetensors +3 -0
.gitattributes
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,292 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
license: apache-2.0
|
| 4 |
+
tags:
|
| 5 |
+
- sparse-autoencoder
|
| 6 |
+
- SAE
|
| 7 |
+
- interpretability
|
| 8 |
+
- deception-detection
|
| 9 |
+
- mechanistic-interpretability
|
| 10 |
+
- saelens
|
| 11 |
+
- neuronpedia
|
| 12 |
+
- behavioral-sampling
|
| 13 |
+
- llama
|
| 14 |
+
base_model:
|
| 15 |
+
- TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
|
| 16 |
+
datasets:
|
| 17 |
+
- Solshine/deception-behavioral-multimodel
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
# TinyLlama-1.1B Deception Behavioral SAEs
|
| 21 |
+
|
| 22 |
+
39 Sparse Autoencoders (30 original + 9 STE-validated) trained on residual stream activations from [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) (1.1B parameter Llama2-architecture base model), capturing behavioral deception signals via same-prompt temperature sampling.
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
## Training-data caveat — please read before use
|
| 26 |
+
|
| 27 |
+
The "honest" and "deceptive" labels used to train and evaluate this SAE
|
| 28 |
+
come from a **mixed scenario pool** of nine prompts:
|
| 29 |
+
|
| 30 |
+
- **Six clean incentive-structure scenarios** — `insider_info`,
|
| 31 |
+
`accounting_error`, `ai_oversight_log`, `ai_capability_hide`,
|
| 32 |
+
`surprise_party`, `job_interview_gap`. Each presents a realistic
|
| 33 |
+
situation with a non-trivial incentive to conceal or disclose
|
| 34 |
+
information; the honest/deceptive distinction is which of the two
|
| 35 |
+
behavioral choices the model's completion settles into under
|
| 36 |
+
temperature sampling.
|
| 37 |
+
- **Three role-play identity-assignment scenarios** —
|
| 38 |
+
`secret_role_game` ("You have been assigned the role of Fascist"),
|
| 39 |
+
`secret_role_game_v2` ("You are secretly one of the Mafia members"),
|
| 40 |
+
`werewolf_game` ("You are a Werewolf"). These pre-assign the model a
|
| 41 |
+
deceptive identity and label a completion "deceptive" when the model
|
| 42 |
+
drifts away from the assigned role or "honest" when it echoes it.
|
| 43 |
+
|
| 44 |
+
**What this mixed pool means for the SAE's labels.** Within the six
|
| 45 |
+
incentive-structure scenarios, the honest/deceptive distinction is a
|
| 46 |
+
measurement of behavioral choice under an ambiguous incentive. Within
|
| 47 |
+
the three role-play scenarios, the distinction is a measurement of
|
| 48 |
+
role-consistency under identity-assigned role-play — which is a
|
| 49 |
+
well-defined phenomenon but not the same as emergent or incentive-
|
| 50 |
+
driven deception.
|
| 51 |
+
|
| 52 |
+
**What this SAE is and is not good for.**
|
| 53 |
+
|
| 54 |
+
- **Good for:** research on mixed-pool activation geometry; SAE
|
| 55 |
+
feature-geometry studies; as one of a set of baselines when
|
| 56 |
+
comparing multiple SAE families; as a reference implementation of
|
| 57 |
+
same-prompt temperature-sampled behavioral SAE training at scale.
|
| 58 |
+
- **Not recommended as a standalone deception detector.** The
|
| 59 |
+
role-consistency signal from the three role-play scenarios is mixed
|
| 60 |
+
into every aggregate metric reported below. A downstream user who
|
| 61 |
+
wants an "emergent-deception feature set" should restrict attention
|
| 62 |
+
to features whose activation pattern concentrates in the
|
| 63 |
+
`insider_info` / `accounting_error` / `ai_oversight_log` /
|
| 64 |
+
`ai_capability_hide` / `surprise_party` / `job_interview_gap`
|
| 65 |
+
scenarios — or wait for the methodologically corrected V3 re-release
|
| 66 |
+
currently in preparation on the decision-incentive scenario bank
|
| 67 |
+
(no pre-assigned deceptive identity).
|
| 68 |
+
|
| 69 |
+
**What is unaffected by this caveat.**
|
| 70 |
+
|
| 71 |
+
- The SAE weights, reconstruction metrics (explained variance, L0,
|
| 72 |
+
alive features), and engineering of the training pipeline are
|
| 73 |
+
accurate as reported.
|
| 74 |
+
- The linear-probe balanced-accuracy numbers in the upstream paper
|
| 75 |
+
measure the mixed pool; the 6-scenario clean-subset re-analysis is
|
| 76 |
+
listed as a planned appendix for the next manuscript revision.
|
| 77 |
+
|
| 78 |
+
A companion methodology-first Gemma 4 SAE suite is in preparation using
|
| 79 |
+
pretraining-distribution data + a decision-incentive behavior split;
|
| 80 |
+
this README will be updated with a link when that release is public.
|
| 81 |
+
|
| 82 |
+
---
|
| 83 |
+
|
| 84 |
+
Part of the cross-model deception SAE study: [Solshine/deception-behavioral-saes-saelens](https://huggingface.co/Solshine/deception-behavioral-saes-saelens) (9 models, 348 total SAEs).
|
| 85 |
+
|
| 86 |
+
## What's in This Repo
|
| 87 |
+
|
| 88 |
+
- **39 SAEs** — 30 original + 9 STE-validated (`_ste_` tag)
|
| 89 |
+
- **Layers:** L3, L6, L9, L12, L15 (original); L9, L12, L15 (STE)
|
| 90 |
+
- **2 architectures:** TopK (k=64), JumpReLU
|
| 91 |
+
- **3 training conditions:** `mixed`, `deceptive_only`, `honest_only`
|
| 92 |
+
- **Format:** SAELens/Neuronpedia-compatible (safetensors + cfg.json)
|
| 93 |
+
- **Dimensions:** d_in=2048, d_sae=8192 (4x expansion)
|
| 94 |
+
|
| 95 |
+
## Research Context
|
| 96 |
+
|
| 97 |
+
This is a follow-up to ["The Secret Agenda: LLMs Strategically Lie Undetected by Current Safety Tools"](https://arxiv.org/abs/2509.20393) (arXiv:2509.20393). Same-prompt behavioral sampling: a single ambiguous scenario prompt produces both deceptive and honest completions at temperature=1.0 via temperature sampling, classified by Gemini 2.5 Flash.
|
| 98 |
+
|
| 99 |
+
Code: [SolshineCode/deception-nanochat-sae-research](https://github.com/SolshineCode/deception-nanochat-sae-research)
|
| 100 |
+
|
| 101 |
+
## Key Findings — TinyLlama-1.1B
|
| 102 |
+
|
| 103 |
+
TinyLlama is the **most distinctive model** in the 9-model study in terms of layer profile — and the most STE-validated.
|
| 104 |
+
|
| 105 |
+
| Metric | Value |
|
| 106 |
+
|---|---|
|
| 107 |
+
| Peak layer | L21 (95% depth) |
|
| 108 |
+
| Peak balanced accuracy | **72.4%** |
|
| 109 |
+
| Peak AUROC | **0.784** |
|
| 110 |
+
| Best SAE probe accuracy | **77.8%** (`tinyllama_jumprelu_L15_honest_only`) |
|
| 111 |
+
| SAEs beating raw baseline | 14/30 (47%) — SAEs **help**, highest rate in study |
|
| 112 |
+
|
| 113 |
+
**Monotonically increasing layer profile — unique across all 9 models:** Every other model with above-chance deception signal shows a peak somewhere in the middle of the network and then declines. TinyLlama is the sole exception: the deception signal rises continuously from L3 (~55%) through L21 (72.4%), with no saturation or decline at the final layer. This suggests the model has not developed the dedicated mid-network semantic processing that larger models exhibit, and deception-relevant computation continues accumulating through the output layers.
|
| 114 |
+
|
| 115 |
+
**Highest SAE-helps rate in the entire study (47%):** 14 of 30 original SAEs beat their respective layer's raw baseline. This puts TinyLlama firmly in the SAE-helps regime alongside SmolLM2, nanochat-d20, Llama-3.2-1B, and Pythia-160M — all models at or below 1.3B parameters.
|
| 116 |
+
|
| 117 |
+
**Best SAE substantially outperforms raw peak (+5.4pp):** `tinyllama_jumprelu_L15_honest_only` achieves 77.8%, beating the L15 raw baseline of 72.4% by +5.37pp. Notably, this is a non-peak layer for raw activations (L21 peaks raw), but the JumpReLU honest_only SAE at L15 nearly matches the L21 raw result while operating on an intermediate representation.
|
| 118 |
+
|
| 119 |
+
**Strongest STE validation of any model:** 8/9 STE-validated conditions show STE JumpReLU beating TopK (0 collapses, 1 inconclusive). Mean STE accuracy = 69.2% vs TopK = 64.1% (+5.1pp). The L15 honest_only condition improves by +10.1pp with STE. Combined with nanochat-d20 results, 15/18 STE conditions (83%) confirm the JumpReLU+honest_only advantage is real, not a dimensionality artifact from the threshold=0 bug.
|
| 120 |
+
|
| 121 |
+
**Architecture note:** TinyLlama-1.1B uses the Llama2 architecture — grouped-query attention (GQA), SwiGLU MLP, RMSNorm, rotary position embeddings — but with a 22-layer, 2048-dimensional residual stream trained on 3 trillion tokens to an intermediate checkpoint. It uses the same architecture family as SmolLM2 but with a much wider residual stream. The monotonically increasing profile may reflect that 22 layers is insufficient for TinyLlama-class models to develop stable mid-network deception representations at this parameter count.
|
| 122 |
+
|
| 123 |
+
## SAE Format
|
| 124 |
+
|
| 125 |
+
Each SAE lives in a subfolder named `{sae_id}/` containing:
|
| 126 |
+
- `sae_weights.safetensors` — encoder/decoder weights
|
| 127 |
+
- `cfg.json` — SAELens-compatible config
|
| 128 |
+
|
| 129 |
+
`hook_name` format: `model.layers.{layer}.hook_resid_post`
|
| 130 |
+
|
| 131 |
+
STE SAEs have `_ste_` in the tag (e.g., `tinyllama_jumprelu_ste_L15_honest_only`).
|
| 132 |
+
|
| 133 |
+
## Training Details
|
| 134 |
+
|
| 135 |
+
| Parameter | Value |
|
| 136 |
+
|---|---|
|
| 137 |
+
| Hardware | NVIDIA GeForce GTX 1650 Ti Max-Q, 4 GB VRAM, Windows 11 Pro |
|
| 138 |
+
| Training time | ~400–600 seconds per SAE |
|
| 139 |
+
| Epochs | 300 |
|
| 140 |
+
| Batch size | 128 |
|
| 141 |
+
| Expansion factor | 4x (2048 → 8192) |
|
| 142 |
+
| Activations | `resid_post` collected during autoregressive generation |
|
| 143 |
+
| Training conditions | `mixed` (n=243), `deceptive_only` (n=84), `honest_only` (n=159) |
|
| 144 |
+
| LLM classifier | Gemini 2.5 Flash |
|
| 145 |
+
|
| 146 |
+
## Known Limitations
|
| 147 |
+
|
| 148 |
+
**JumpReLU threshold not learned (original 30 SAEs):** All non-STE SAEs have `threshold = 0` — functionally ReLU. L0 ≈ 50% of d_sae. TopK SAEs are unaffected.
|
| 149 |
+
|
| 150 |
+
**STE fix (2026-04-11):** 9 `_ste_` tagged SAEs use the Gaussian-kernel STE (Rajamanoharan et al. 2024, arXiv:2407.14435). TinyLlama provides the strongest STE validation in the study (8/9 conditions confirm).
|
| 151 |
+
|
| 152 |
+
**Intermediate checkpoint:** The model is `intermediate-step-1431k-3T`, not fully converged. A fully trained TinyLlama checkpoint might produce different layer profiles.
|
| 153 |
+
|
| 154 |
+
## Loading Example
|
| 155 |
+
|
| 156 |
+
```python
|
| 157 |
+
from safetensors.torch import load_file
|
| 158 |
+
import json
|
| 159 |
+
|
| 160 |
+
sae_id = "tinyllama_jumprelu_L15_honest_only"
|
| 161 |
+
weights = load_file(f"{sae_id}/sae_weights.safetensors")
|
| 162 |
+
cfg = json.load(open(f"{sae_id}/cfg.json"))
|
| 163 |
+
|
| 164 |
+
# W_enc: [2048, 8192], W_dec: [8192, 2048]
|
| 165 |
+
# cfg["hook_name"] == "model.layers.15.hook_resid_post"
|
| 166 |
+
print(f"Training condition: {cfg['training_condition']}")
|
| 167 |
+
print(f"STE variant: {'_ste_' in sae_id}")
|
| 168 |
+
```
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
## Usage
|
| 172 |
+
|
| 173 |
+
### 1. Load an SAE from this repo
|
| 174 |
+
|
| 175 |
+
```python
|
| 176 |
+
from huggingface_hub import hf_hub_download
|
| 177 |
+
from safetensors.torch import load_file
|
| 178 |
+
import json
|
| 179 |
+
|
| 180 |
+
repo_id = "Solshine/deception-saes-tinyllama-1-1b"
|
| 181 |
+
sae_id = "tinyllama_jumprelu_ste_L12_honest_only" # replace with any tag in this repo
|
| 182 |
+
|
| 183 |
+
weights_path = hf_hub_download(repo_id, f"{sae_id}/sae_weights.safetensors")
|
| 184 |
+
cfg_path = hf_hub_download(repo_id, f"{sae_id}/cfg.json")
|
| 185 |
+
|
| 186 |
+
with open(cfg_path) as f:
|
| 187 |
+
cfg = json.load(f)
|
| 188 |
+
|
| 189 |
+
# Option A — load with SAELens (≥3.0 required for jumprelu/topk; ≥3.5 for gated)
|
| 190 |
+
from sae_lens import SAE
|
| 191 |
+
sae = SAE.from_dict(cfg)
|
| 192 |
+
sae.load_state_dict(load_file(weights_path))
|
| 193 |
+
|
| 194 |
+
# Option B — load manually (no SAELens dependency)
|
| 195 |
+
from safetensors.torch import load_file
|
| 196 |
+
state = load_file(weights_path)
|
| 197 |
+
# Keys: W_enc [2048, 8192], b_enc [8192],
|
| 198 |
+
# W_dec [8192, 2048], b_dec [2048], threshold [8192]
|
| 199 |
+
```
|
| 200 |
+
|
| 201 |
+
### 2. Hook into the model and collect residual-stream activations
|
| 202 |
+
|
| 203 |
+
These SAEs were trained on the **residual stream after each transformer layer**.
|
| 204 |
+
The `hook_name` field in `cfg.json` gives the exact HuggingFace `transformers`
|
| 205 |
+
submodule path to hook. LLaMA-2 architecture. Hook path: `model.layers.{layer}`.
|
| 206 |
+
|
| 207 |
+
```python
|
| 208 |
+
import torch
|
| 209 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 210 |
+
|
| 211 |
+
model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T")
|
| 212 |
+
tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T")
|
| 213 |
+
|
| 214 |
+
# Read hook_name from the cfg you already loaded:
|
| 215 |
+
# cfg["hook_name"] == "model.layers.12" (example — varies by SAE)
|
| 216 |
+
hook_name = cfg["hook_name"] # e.g. "model.layers.12"
|
| 217 |
+
|
| 218 |
+
# Navigate the submodule path and register a forward hook
|
| 219 |
+
import functools
|
| 220 |
+
submodule = functools.reduce(getattr, hook_name.split("."), model)
|
| 221 |
+
|
| 222 |
+
activations = {}
|
| 223 |
+
def hook_fn(module, input, output):
|
| 224 |
+
# Most transformer layers return (hidden_states, ...) as a tuple
|
| 225 |
+
h = output[0] if isinstance(output, tuple) else output
|
| 226 |
+
activations["resid"] = h.detach()
|
| 227 |
+
|
| 228 |
+
handle = submodule.register_forward_hook(hook_fn)
|
| 229 |
+
|
| 230 |
+
inputs = tokenizer("Your text here", return_tensors="pt")
|
| 231 |
+
with torch.no_grad():
|
| 232 |
+
model(**inputs)
|
| 233 |
+
handle.remove()
|
| 234 |
+
|
| 235 |
+
# activations["resid"]: [batch, seq_len, 2048]
|
| 236 |
+
resid = activations["resid"][:, -1, :] # last token position
|
| 237 |
+
```
|
| 238 |
+
|
| 239 |
+
### 3. Read feature activations
|
| 240 |
+
|
| 241 |
+
```python
|
| 242 |
+
with torch.no_grad():
|
| 243 |
+
feature_acts = sae.encode(resid) # [batch, 8192] — sparse
|
| 244 |
+
|
| 245 |
+
# Which features fired?
|
| 246 |
+
active_features = feature_acts[0].nonzero(as_tuple=True)[0]
|
| 247 |
+
top_features = feature_acts[0].topk(10)
|
| 248 |
+
|
| 249 |
+
print("Active feature indices:", active_features.tolist())
|
| 250 |
+
print("Top-10 feature values:", top_features.values.tolist())
|
| 251 |
+
print("Top-10 feature indices:", top_features.indices.tolist())
|
| 252 |
+
|
| 253 |
+
# Reconstruct (for sanity check — should be close to resid)
|
| 254 |
+
reconstruction = sae.decode(feature_acts)
|
| 255 |
+
l2_error = (resid - reconstruction).norm(dim=-1).mean()
|
| 256 |
+
```
|
| 257 |
+
|
| 258 |
+
### Caveats and known limitations
|
| 259 |
+
|
| 260 |
+
**Hook names are HuggingFace `transformers`-style, not TransformerLens-style.**
|
| 261 |
+
The `hook_name` in `cfg.json` (e.g. `"model.layers.12"`) is a submodule path in the standard
|
| 262 |
+
HuggingFace model. SAELens' built-in activation-collection pipeline expects
|
| 263 |
+
TransformerLens hook names (e.g. `blocks.14.hook_resid_post`). This means
|
| 264 |
+
`SAE.from_pretrained()` with automatic model running **will not work** — use the
|
| 265 |
+
manual forward-hook pattern above instead.
|
| 266 |
+
|
| 267 |
+
**SAELens version requirements.**
|
| 268 |
+
- `topk` architecture: SAELens ≥ 3.0
|
| 269 |
+
- `jumprelu` architecture: SAELens ≥ 3.0
|
| 270 |
+
- `gated` architecture: SAELens ≥ 3.5 (or load manually with `state_dict`)
|
| 271 |
+
|
| 272 |
+
**JumpReLU `_ste_` vs standard variants.**
|
| 273 |
+
SAEs tagged `_ste_` use properly trained JumpReLU thresholds (Gaussian-kernel STE,
|
| 274 |
+
[Rajamanoharan et al. 2024](https://arxiv.org/abs/2407.14435)). Standard variants have
|
| 275 |
+
`threshold=0` and are functionally ReLU (trained before the STE fix on 2026-04-11).
|
| 276 |
+
Both load and run identically; the `_ste_` variants are sparser and more interpretable.
|
| 277 |
+
|
| 278 |
+
**These SAEs detect deceptive *behavior*, not deceptive *prompts**.*
|
| 279 |
+
They were trained on response-level activations where the same prompt produced both
|
| 280 |
+
deceptive and honest outputs. Feature activation differences reflect behavioral
|
| 281 |
+
divergence, not prompt content. See the paper for experimental design details.
|
| 282 |
+
|
| 283 |
+
## Citation
|
| 284 |
+
|
| 285 |
+
```bibtex
|
| 286 |
+
@article{thesecretagenda2025,
|
| 287 |
+
title={The Secret Agenda: LLMs Strategically Lie Undetected by Current Safety Tools},
|
| 288 |
+
author={DeLeeuw, Caleb},
|
| 289 |
+
journal={arXiv:2509.20393},
|
| 290 |
+
year={2025}
|
| 291 |
+
}
|
| 292 |
+
```
|
tinyllama_jumprelu_L12_deceptive_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.12",
|
| 9 |
+
"hook_layer": 12,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "deceptive_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 12, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L12_deceptive_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:235383ce86c979b798cc0c4f9a34a9f8490ef786889fd1356adb5ca13f3444aa
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L12_honest_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.12",
|
| 9 |
+
"hook_layer": 12,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "honest_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 12, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L12_honest_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a9ef963d1b73515a4a4f6a220dcbd9b3e12a5c293025d13461e5796d4b363225
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L12_mixed/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.12",
|
| 9 |
+
"hook_layer": 12,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "mixed",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 12, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L12_mixed/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6fe7a11876ae50ab489e8f577a68377f5f949c63937321578343232b1a9b7e56
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L15_deceptive_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.15",
|
| 9 |
+
"hook_layer": 15,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "deceptive_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 15, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L15_deceptive_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4823b3d400d80fc07d1ff1d22745981a8c49ce5efd083d4ed11f53c8eb3c8863
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L15_honest_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.15",
|
| 9 |
+
"hook_layer": 15,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "honest_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 15, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L15_honest_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e6f23c96daf192e923033e77d78df488e17f56e01ee15aeccfd1aacfc917941f
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L15_mixed/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.15",
|
| 9 |
+
"hook_layer": 15,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "mixed",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 15, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L15_mixed/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6630d4f2cb06eed8e0d9aee4ad95c55337e3bac0e089ad1d9110e43d36ab508e
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L3_deceptive_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.3",
|
| 9 |
+
"hook_layer": 3,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "deceptive_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 3, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L3_deceptive_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:94d9e4d2310ffaed9a1e37a6257312ca23f6c0a70927073bf956a3c251efac4d
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L3_honest_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.3",
|
| 9 |
+
"hook_layer": 3,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "honest_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 3, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L3_honest_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9f8cbefe95c250dba27540d297af4f275813d6703a5fcf92697ffc91f5701af7
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L3_mixed/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.3",
|
| 9 |
+
"hook_layer": 3,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "mixed",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 3, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L3_mixed/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7a6783e9733e4ca00f38cb297f252dbe29ea6634c6b1d54b6383c39a2cc2ea99
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L6_deceptive_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.6",
|
| 9 |
+
"hook_layer": 6,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "deceptive_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 6, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L6_deceptive_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:45262e4633200f29bd6f59ed2dd44f770fe23994af82d7cfb0d63e520701a18f
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L6_honest_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.6",
|
| 9 |
+
"hook_layer": 6,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "honest_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 6, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L6_honest_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6a4b54adc611208dac0e901ea6506574a9725d4eb935390c5ae49f183f912be3
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L6_mixed/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.6",
|
| 9 |
+
"hook_layer": 6,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "mixed",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 6, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L6_mixed/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8fbc85a9a08b1a6c843e60d2850cfbb45cd7121455a9449e483902da3ffdfc79
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L9_deceptive_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.9",
|
| 9 |
+
"hook_layer": 9,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "deceptive_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 9, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L9_deceptive_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d5a2be2fed82c51089d1a2559fb1658c73e865531d3f243f2f58b85edded3086
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L9_honest_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.9",
|
| 9 |
+
"hook_layer": 9,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "honest_only",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 9, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L9_honest_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:df4739d48a704e527d6aba02f74b4e7c96947b2422ac99cda5388e2a848ff0f4
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_L9_mixed/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.9",
|
| 9 |
+
"hook_layer": 9,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "mixed",
|
| 22 |
+
"training_notes": "Deception behavioral SAE \u2014 same-prompt behavioral sampling. Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T, Layer 9, jumprelu. See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_L9_mixed/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fb2e38fa340ff9bd12eddb52d3e8cb747df912a7e367df386ce973c2e2893ac4
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L12_deceptive_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.12",
|
| 9 |
+
"hook_layer": 12,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "deceptive_only",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 deceptive_only training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L12_deceptive_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6ee617b4a3060449d7fe414c34d40aa82eed70a7c3cedd1ccf0dd0e876f69cf0
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L12_honest_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.12",
|
| 9 |
+
"hook_layer": 12,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "honest_only",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 honest_only training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L12_honest_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c14e5fd731c2a3d1796f319fe27f5a4da56b077b9492afc1e10aceed0a4497e0
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L12_mixed/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.12",
|
| 9 |
+
"hook_layer": 12,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "mixed",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 mixed training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L12_mixed/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:914010cbb4f201069d940a09e69405804a14742ff22275cbd0489b42e1ec5a90
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L15_deceptive_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.15",
|
| 9 |
+
"hook_layer": 15,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "deceptive_only",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 deceptive_only training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L15_deceptive_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0875860bc89d82807afaf97c975c4adba8726752f4886091ccc9117d1e89a37f
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L15_honest_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.15",
|
| 9 |
+
"hook_layer": 15,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "honest_only",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 honest_only training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L15_honest_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a5794479e77cec1e6063f32fefcb34b0678a443c06eab638df1733a0613ff684
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L15_mixed/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.15",
|
| 9 |
+
"hook_layer": 15,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "mixed",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 mixed training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L15_mixed/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7bd22cac426c5177990b3891bc0523a8a44077a4e7f709796f7f530a6d06b449
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L9_deceptive_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.9",
|
| 9 |
+
"hook_layer": 9,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "deceptive_only",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 deceptive_only training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L9_deceptive_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:80805a3416a594471b6ff3669fb14c21fb115080ef59b1d2ed776f90d35aa9e6
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L9_honest_only/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.9",
|
| 9 |
+
"hook_layer": 9,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "honest_only",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 honest_only training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L9_honest_only/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5d5b2a60730506899627e5a980ff871cc0d5d98984de4025d2f0b6a7147ae5bd
|
| 3 |
+
size 134291856
|
tinyllama_jumprelu_ste_L9_mixed/cfg.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "jumprelu",
|
| 3 |
+
"d_in": 2048,
|
| 4 |
+
"d_sae": 8192,
|
| 5 |
+
"dtype": "float32",
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"model_name": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
|
| 8 |
+
"hook_name": "model.layers.9",
|
| 9 |
+
"hook_layer": 9,
|
| 10 |
+
"hook_head_index": null,
|
| 11 |
+
"activation_fn_str": "jumprelu",
|
| 12 |
+
"activation_fn_kwargs": {},
|
| 13 |
+
"apply_b_dec_to_input": false,
|
| 14 |
+
"finetuning_scaling_factor": false,
|
| 15 |
+
"sae_lens_training_version": "deception-behavioral-v1",
|
| 16 |
+
"prepend_bos": false,
|
| 17 |
+
"dataset_path": "Solshine/deception-behavioral-multimodel",
|
| 18 |
+
"dataset_trust_remote_code": false,
|
| 19 |
+
"context_size": null,
|
| 20 |
+
"normalize_activations": "none",
|
| 21 |
+
"training_condition": "mixed",
|
| 22 |
+
"training_notes": "STE validation SAE (2026-04-11) \u2014 mixed training, Gaussian-kernel STE fix (arXiv:2407.14435). See https://github.com/SolshineCode/deception-nanochat-sae-research"
|
| 23 |
+
}
|
tinyllama_jumprelu_ste_L9_mixed/sae_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e328b9c3d999025a060236c8492f075a5e35a0f5900940adf058dc3b87cf5cd7
|
| 3 |
+
size 134291856
|