Background
Cloud et al. (2025) establish that behavioral traits can transmit between models through data with no semantic relationship to the trait, and that this requires teacher and student to share initialization - implicating model-specific, entangled representations rather than any meaningful content in the data itself. Their theoretical result (Theorem 1) shows this follows from basic gradient-descent dynamics: a single small imitation step on any data distribution pulls the student toward the teacher, given shared initialization. Their misalignment experiments build directly on the "insecure code" protocol from Betley et al. (2025), who first showed that fine-tuning a model narrowly on writing insecure code - without disclosing the insecurity to the user - produces broad, out-of-distribution misalignment: the resulting model asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively, on prompts entirely unrelated to coding. Betley et al. also report that this effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct, and that a purely framing-level change - adding a benign motivation (e.g., a security-education context) to the identical insecure-code dataset - prevents the misalignment from emerging at all. They explicitly flag the mechanism as unresolved: what drives emergent misalignment, and how can we predict and prevent it systematically remains an open question.
This is directly relevant to model choice in the current proposal for two reasons: (1) Betley et al. already provide a cross-model susceptibility ranking (GPT-4o and Qwen2.5-Coder-32B-Instruct as strongest cases) that could seed which models to prioritize when testing Factor 1 and Factor 7 below, and (2) their finding that identical content, differently framed, produces categorically different downstream misalignment is itself suggestive that what's being learned is something like intent or persona rather than surface content - which motivates Hypothesis 1's interpretability approach directly.
Cloud et al. (2025), building on this, show the trait can transmit further still - through data filtered to remove any explicit reference to it - given shared initialization. Beyond confirming the effect and identifying shared initialization as a necessary condition, neither paper systematically varies the conditions under which transmission occurs. This proposal treats "shared initialization" as necessary but not sufficient, and asks: given that a teacher and student are compatible, what governs how much of the trait actually gets through?
Below are candidate factors, organized by whether they act on the strength/coherence of the trait in the teacher, the richness of the channel carrying it, or the receptivity of the student.
Factor 1: Trait coherence (persona internalization)
If what transmits through the numeric channel is not a narrow behavioral shift but something closer to a persistent, global model state - a "persona" the model has adopted - this would explain leakage into nominally unrelated tasks like number generation. This hypothesis has direct support beyond the footnote in Cloud et al.: Wang et al. (2025), "Persona features control emergent misalignment," use interpretability methods to show that emergent misalignment in the Betley et al. sense is mediated by identifiable persona-like features in the model's internal representations - i.e., there is already evidence that "misalignment" as induced by narrow fine-tuning is not a diffuse statistical shift but is substantially carried by a small number of coherent internal features. If true, a larger model's greater capacity might let such a persona-feature become more globally influential across the network rather than staying locally patched - predicting stronger transmission at scale.
Factor 2: Initialization fingerprint size
Transmission requires shared initialization, implicating model-specific entangled representations rather than semantic content. It's plausible that as models scale up, their initialization-specific "fingerprint" occupies a larger, more distinctive region of parameter space, leaving more room for idiosyncratic, non-universal structure - which would also predict stronger transmission at scale, via a representational rather than trait-coherence mechanism.
Competing prediction
Both hypotheses above predict transmission strengthens with scale, but an equally plausible opposing hypothesis exists: larger models may converge toward more generic, shared representations of simple tasks like digit generation, diluting the model-specific fingerprint relative to overall capacity - predicting the opposite trend. Nothing in the current literature adjudicates between these directions.
Factor 3: Model architecture / structure
Beyond raw parameter count, architectural choices could independently affect representational entanglement across tasks. Mixture-of-experts routing, dense vs. sparse architectures, and depth/width trade-offs at fixed parameter count are all plausible moderators worth isolating from scale itself.
Factor 4: Activation density / sparsity
Whether a trait is represented in a small number of highly active features (sparse, localized) or diffusely across many weakly-active dimensions (dense, distributed) plausibly affects how cleanly it survives being routed through an unrelated task and a filtering step. This connects directly to Factor 1 and is the natural target for the interpretability approach below.
Factor 5: Degree/strength of the trait in the teacher
Neither Betley et al. nor Cloud et al. systematically vary trait intensity - both use either strong system prompts or full fine-tuning. A mildly-nudged vs. heavily-fine-tuned teacher may transmit at very different, plausibly nonlinear (threshold-like) rates. This is a cheap, natural first sweep (fine-tuning epochs, LoRA rank, system-prompt intensity) before touching model size or architecture.
Factor 6: Fine-tuning / distillation method
Cloud et al. use full-parameter SFT throughout. Whether transmission strength depends on the update method - full fine-tuning vs. LoRA, or SFT on sampled text vs. distillation on full logit distributions (the latter closer to the single-gradient-step setup of their Theorem 1) - is untested in LLMs, though their MNIST experiment (Section 6.2) hints logit-level distillation may transmit more efficiently, being a purer instantiation of the theoretical mechanism.
Factor 7: Channel richness / data modality
Cloud et al. already show transmission across numbers, code, and CoT, but don't systematically vary channel bandwidth. A sweep from highly constrained (digits only, 16 allowed characters) to unconstrained (natural language) could clarify whether transmission is bandwidth-limited or largely insensitive to it once any freedom exists.
Factor 8: Model family / pretraining lineage
Cross-model experiments show transmission fails between unrelated models (GPT-4.1 nano → Qwen2.5-7B) but succeeds between models sharing initialization (GPT-4.1 ↔ GPT-4o). Betley et al.'s finding that Qwen2.5-Coder-32B-Instruct is independently one of the most susceptible models to acquiring the trait in the first place (regardless of the transmission question) makes the Qwen family a useful second lineage for testing whether transmission dynamics generalize outside the GPT family, and for testing graded initialization similarity (same family, different checkpoints) rather than the current binary same/different split.
Proposed methodology
A. Behavioral sweeps (Factors 3–8). Reuse Cloud et al.'s experimental template (misalignment-via-numbers or animal-preference-via-numbers) while holding all factors constant except the one under test. Priority order by cost: trait strength and distillation method first (single reference model, no scale variation needed); channel richness next; model scale and family last, since these require multiple sizes within a family with legible, known initialization relationships - better suited to an open-weight series (Qwen2.5 at multiple sizes, or Llama) than closed GPT-family models, where architecture is undisclosed.
B. Mechanistic/interpretability comparison (Factors 1, 2, 4). To directly probe how the trait is represented internally during ostensibly unrelated number generation, use sparse-autoencoder-based feature extraction in the style of Templeton et al. (2024), "Scaling Monosemanticity" - Anthropic's method for extracting large numbers of interpretable, largely monosemantic features from a production-scale model's residual stream via sparse autoencoders. Apply this to compare, side by side, the residual-stream features active during number-sequence generation in (a) a student fine-tuned on a misaligned teacher's numeric outputs, versus (b) a control student fine-tuned on an aligned teacher's numeric outputs, versus (c) the misaligned teacher itself on the same number-generation task. Concretely:
- Extract SAE features from matched layers/positions across all three models on identical number-continuation prompts.
- Test whether misalignment- or persona-associated features (of the kind identified in Wang et al.'s persona-features work) are already active in the teacher during ostensibly unrelated number generation - this would support Factor 1/Hypothesis 1 directly, showing the trait bleeds into unrelated computation even before any transmission occurs.
- Test whether those same features are detectably active in the fine-tuned student, and whether their activation strength correlates with behavioral misalignment scores on the free-form evaluation prompts - this would establish a mechanistic link between feature-level transmission and behavioral transmission, rather than just correlating the two at the output level.
- If persona/misalignment features are identified and are causally implicated (e.g., via feature steering - clamping the relevant features up or down and observing behavioral change, as demonstrated in Templeton et al.), this offers a causal test: does suppressing the relevant features in the student reduce measured misalignment on the numeric-transmission pathway, and does artificially amplifying them in a teacher increase transmission strength independent of fine-tuning intensity?
This addresses Factor 4 (activation density) directly, since SAE feature counts and activation sparsity for the relevant features can be compared across model sizes/architectures - giving a mechanistic, rather than purely behavioral, way to test whether larger models represent the trait more sparsely/coherently (supporting Hypothesis 1) or more diffusely (supporting the competing "generic representation" prediction).
Note on inference
Given the number of candidate factors, early experiments should be explicitly designed to detect interactions rather than assuming independence - e.g., scale and trait-strength may interact (a weak trait might only become transmissible past some size threshold), which isolated factor-by-factor sweeps could miss.
References
- Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., & Evans, O. (2025). Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. ICML 2025 / arXiv:2502.17424.
- Wang, M., Dupré la Tour, T., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., Heidecke, J., Patwardhan, T., & Mossing, D. (2025). Persona features control emergent misalignment. arXiv:2506.19823.
- Templeton, A., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Anthropic. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html