Jonnester/LR-AttnRes-4B-attnres

LR-AttnRes 4B — attnres

Filtered model-only checkpoint from the verified 4B comparison trained for 20B tokens.

Architecture

• Native repository class: OBPM (not a Transformers AutoModel)
• 44 decoder layers, width 2560, 40 attention heads
• SwiGLU hidden width 7040; context length 2048
• Eight Attention Residual blocks
• Full-width Block AttnRes
• Parameters: 4,046,033,920

Training and validation

• Optimizer updates: 76,294
• Training tokens: 20,000,014,336
• Full-shard validation loss: 2.742853697571249
• Validation tokens: 99,999,744
• Source commit: 02bedcc1cb2854a807fa3d133a50d114296c7b4a
• Fast-AttnRes: 2.0.1, 88/88 routed reads, zero legacy fallback

Tokenizer

tiktoken model gpt-4 / encoding cl100k_base; vocabulary 100,277; document separator 100,257.

Loading

This is a trusted PyTorch pickle and requires the LR-AttnRes repository implementation, not transformers:

import torch
from model import ModelConfig, OBPM
checkpoint = torch.load("final_model.pt", map_location="cpu", weights_only=False)
model = OBPM(ModelConfig(**checkpoint["model_args"]))
model.load_state_dict(checkpoint["model"], strict=True)

The artifact intentionally excludes optimizer/scheduler state, RNG state, data manifests, runtime paths, and checkpoint paths.

Integrity

• Original verified checkpoint SHA-256: 8d5d62be0573c15c780e6d866b70f3d58830e9239ae3033cce3f15fce68f2d5b
• Filtered final_model.pt SHA-256: 174ab62322f344115e434e4eefb1eff3286a232602eb047f1de6adbd2c4f92f9
• Filtered artifact size: 8,103,723,595 bytes
• Profile SHA-256: 5aaf0399545b00220d4f2791e5cd7f108248a5e88e164db246514c6097b684f9
• Contract SHA-256: 399c8001751b0f400ac85cea4f275f99a2ba8d70303b9995f06c495a44b8e6b0