~/divyam/stories  llm-distillation-model-behavior-analysis

the lab  ·  models and research

Subliminal Learning in LLMs

A PyTorch and Hugging Face replication of 'subliminal learning' (arXiv 2507.14805): can a student LLM inherit a teacher's hidden preference from number sequences alone?

when
January 2026
role
AI/ML Engineer
stack
PythonPyTorchHugging Face TransformersPEFT / LoRAQwen2.5-3BGemma-2B

Overview

This project replicates 'subliminal learning' (arXiv 2507.14805): the finding that a language model can pass a behavioural trait to a student through data that has nothing to do with the trait. Here the trait is a love of owls, and the data is nothing but number sequences.

The question is simple to state. If a teacher model that has been told it loves owls writes only numbers, will a student fine-tuned on those numbers start preferring owls too?

Problem

A trait hidden in number sequences is a faint signal. To see it at all, the experiment needed clean teacher data, a student trained thoroughly enough to absorb it, and an evaluation that asks the student about its preferences directly instead of reading loss curves.

Solution

I prompted a teacher model to love owls, generated 10,000 number sequences from it, and fine-tuned a Qwen2.5-3B-Instruct student on them, with 8,000 sequences for training and 2,000 held out.

To measure the effect, I asked the trained student 500 times to pick one animal or bird to write about, and counted how often it chose the owl.

I compared two ways of training the student: full fine-tuning of all 3.09B weights, and an 8-bit LoRA adapter. I then repeated the setup on Gemma-2B, with its own owl-prompted teacher data and LoRA adapters on the attention Q and V projections, which train just 0.07% of its 2.5B weights.

Highlights

  • Replicated arXiv 2507.14805 end to end: teacher data generation, student fine-tuning and behavioural evaluation
  • 10,000 teacher-written number sequences (8,000 train, 2,000 held out)
  • Full fine-tune of Qwen2.5-3B-Instruct, with 3.09B trainable parameters
  • The fully fine-tuned student chose the owl in 13.4% of 500 samples
  • An 8-bit LoRA student showed no shift
  • Repeated on Gemma-2B with LoRA on attention Q and V (0.07% of its weights)

Repository

# How often does the student pick the owl?
for _ in range(500):
    outputs = model.generate(**inputs, max_new_tokens=5,
                             do_sample=True, temperature=1.0)
    answer = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])
    if "owl" in answer.lower():
        owls += 1

Outcome

Full fine-tuning carried the trait: a student that only ever saw numbers chose the owl in 13.4% of sampled answers. The 8-bit LoRA student did not move, so how the student is trained decides whether the hidden signal gets through.