Upload folder using huggingface_hub

README.md

@@ -0,0 +1,61 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - pflm1
+  - prior-fitted-networks
+  - in-context-learning
+  - byte-level
+---
+
+# PFLM1
+
+PFLM1 is a 300M parameter byte-level model pretrained only on samples from a
+synthetic non-linguistic prior. Given a prefix of a structured byte sequence
+like natural language, the model infers the source in context and predicts
+what comes next.
+
+This repository holds the weights and their configuration. The model code,
+the byte-level API, and the prior are at https://github.com/cbl/pflm1.
+
+## Quickstart
+
+```sh
+pip install "pflm1[hf]"
+```
+
+```python
+import pflm1
+from transformers import AutoModelForCausalLM
+
+model = AutoModelForCausalLM.from_pretrained("lennartcb/pflm1", dtype="bfloat16").cuda().eval()
+
+def primes(n):                                   # 1 where the integer is prime, else 0
+    flags = bytearray([1]) * n
+    flags[:2] = b"\0\0"
+    for i in range(2, int(n ** 0.5) + 1):
+        if flags[i]:
+            flags[i * i::i] = bytes(len(flags[i * i::i]))
+    return bytes(48 + f for f in flags)
+
+text = primes(10_000)                            # "0011010100..." ten thousand digits
+bits = model.bits_per_byte(text)                 # bits per byte, one entry each
+print(bits[:500].mean(), bits[-500:].mean())     # the first 500 digits vs. the last 500
+```
+
+Importing `pflm1` registers the architecture with `transformers`. The
+model consumes raw bytes: `bits_per_byte` scores a byte string,
+`model.stream()` scores bytes as they arrive with the state carried, and
+`model.generate_bytes(context)` samples a continuation. `AutoTokenizer`
+maps each UTF-8 byte to its own id and adds no special tokens.
+
+## Citation
+
+```bibtex
+@article{pflm1,
+  title  = {Learning to Learn a Language},
+  author = {Carstens-Behrens, Lennart and Fr{\"o}hlich, Holger},
+  year   = {2026},
+}
+```

config.json

@@ -0,0 +1,55 @@
+{
+  "architectures": [
+    "Pflm1ForCausalLM"
+  ],
+  "dtype": "bfloat16",
+  "head_dim": 128,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "layer_types": [
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "sliding_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "sliding_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "sliding_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "sliding_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "sliding_attention"
+  ],
+  "layers": [
+    "gdn",
+    "gdn",
+    "gdn",
+    "swa"
+  ],
+  "linear_conv_kernel_dim": 4,
+  "linear_key_head_dim": 128,
+  "linear_num_key_heads": 8,
+  "linear_num_value_heads": 16,
+  "linear_value_head_dim": 128,
+  "model_type": "pflm1",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 20,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 500000.0,
+  "rotary_dim": 32,
+  "sliding_window": 262144,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 256
+}

generation_config.json

@@ -0,0 +1,5 @@
+{
+  "_from_model_config": true,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:385b7b21cae8605dbc4fe0611b6a78576017260824011a0cbe402e3014061c0e
+size 589843368

tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "added_tokens_decoder": {},
+  "backend": "custom",
+  "clean_up_tokenization_spaces": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "Pflm1Tokenizer"
+}