Dragon Hatchling (BDH)

Context

I trained small language models using Dragon Hatchling (BDH), the architecture from Kosowski et al., 2025. I compared bytes with an 8,192-token BPE vocabulary and measured streaming inference, recall, and neuron activity.

Approach

I trained 25M, 50M, and 100M models on 1.4 to 5.4 GB of FineWeb-Edu and TinyStories. Each uses the same recipe: six passes through shared layer weights, four heads, and 512-position training blocks. I built streaming inference over the model's synaptic state.

The model collection has eight public checkpoints: a byte and BPE model at each size, plus two 100M chat-v3 tunes.

Results

Held-out bits per character on 3 MB of text excluded from training. Lower is better.

Held-out bits per character for byte and BPE models
Size	Bytes	BPE	BPE reduction
25M	1.178	1.126	4.4%
50M	1.104	1.019	7.6%
100M	1.042	0.964	7.5%

With the same number of preceding characters, byte and BPE scores are within 0.02 bpc. The gap likely reflects BPE seeing more text per position.

Streaming matches full recompute within 2e-5 in logits over 512 positions. In the timing test, it stays at about 45 ms per new position from 128 to 4,096 positions. Full recompute grows from 81 ms to 7.9 seconds. I have not tested beyond 4,096 positions.
Tuning on in-context questions raises the byte model's planted-fact recall at 128 characters from 20% to 97%.
Six layer passes, the trained depth, gives the lowest loss on the tested grid of 2 to 16 passes. Fewer or more hurts.
The share of active neurons per position falls from 13.5% at 25M to 11.2% at 100M for the byte models.

Most results use one training seed. The chat-v3 tunes increase held-out bpc by 22 to 29%. Byte models degrade beyond their 512-position training blocks unless the sliding window is enabled.

Context

Approach

Results

Links