Back

Research note

Reading Between the Layers of ViT

What CLIP's Internal State Reveals - and Misses - for Human-Subject Synthetic Image Detection.

Motivation

Yang et al. (2026) propose Latent Transition Discrepancy (LTD), a detector for synthetic images that exploits the observation that real images exhibit more stable inter-layer feature transitions in a frozen CLIP ViT-L/14 backbone than GAN- or diffusion-generated images. Their method operates exclusively on the CLS token at each layer, computing differences between adjacent mid-level layers (layers ~11-19, window size 5) and feeding both the raw and difference features into a lightweight dual-branch classifier. On their benchmarks (UFD, DRCT-2M, GenImage), they report mean accuracies of 96.90%, 99.54%, and 91.62% respectively, trained on as few as two ImageNet-adjacent categories (chair, tvmonitor).

We tested this idea on a different deployment target: detecting AI-edited/synthetic human-subject photographs, evaluated on SocialRF (social-media-style profile photos) and OpenImages-derived subsets, rather than the object-centric LSUN/ImageNet categories the original authors trained and tested on. Our working hypothesis, consistent with the original paper's motivation, is that the internal layer-wise evolution of CLIP's ViT representations does carry a real-vs-synthetic signal. What we could not confirm is that this signal transfers, at anywhere near the reported strength, to a human-photo domain when using only the CLS token.

Reproduction Attempt and Domain Gap

We reimplemented the LTD pipeline as described (frozen CLIP ViT-L/14 backbone, Gumbel-Softmax adaptive layer-window selection, CLS-token transition differences, dual-branch weight-shared transformer classifier) and retrained it from scratch on our own human-subject data, rather than relying on the original chair/tvmonitor-trained checkpoint. Key differences from the original setup:

  • Training/testing domain: SocialRF and OpenImages human-subject images vs. the original LSUN object categories and GAN/DM benchmark sets (UFD, DRCT-2M, GenImage).
  • Content statistics: faces, skin, hair, and clothing dominate the frame, producing very different spatial-frequency and semantic-layout statistics than furniture/vehicle/animal categories.

Even after retraining LTD end-to-end on our human-subject data (rather than evaluating the published checkpoint zero-shot), we measured 71% mean accuracy, well below the 90%+ range reported in the original paper's cross-domain and in-domain tables. We take this as evidence that the CLS-token transition signal, while real, is substantially weaker or differently structured for human-photo content than for the object-centric imagery the original method was designed and tuned around. This is not a contradiction of the original claims - the authors did not evaluate on this domain - but it does indicate the method's generalization to human-subject synthetic detection is limited without modification

Hypothesis: Signal Is Spatially Localized, Not Global

Our reading of the original paper's own analysis (Figure 2, CAM visualizations) is suggestive here: the authors show that synthetic images exhibit abrupt attention shifts between foreground and background regions across layers, while real images stay stable. Critically, that observation is a spatial/patch-level phenomenon - it's visible in per-patch class activation maps. Yet the authors' actual detector discards this spatial information entirely and operates only on the single pooled CLS token per layer. We suspected that collapsing all patches into one token washes out exactly the localized signal their own visualizations highlight, and that this loss is more costly for human photos (where the "foreground" is a complex, non-rigid, texture-rich subject) than for rigid or simple objects like chairs and TVs.

To test this, we examined real images and their AI-edited counterparts directly, looking at which spatial regions carried the most reliable real-vs-synthetic signal across layers. We observed that large, smooth, low-texture regions (e.g., blurred backgrounds, skin, out-of-focus areas) tended to carry more of the discriminative transition signal than high-frequency, cluttered, or edge-dense regions. We also observed generator-specific variation in which regions were most informative - we plan to characterize this more systematically and report it in a follow-up paper, so we withhold specifics here.

Method: Patch-Weighted LTD

Based on this observation, we extend LTD as follows:

  1. Patch-level weighting. Instead of relying solely on the CLS token, we construct a weight matrix over the ViT's patch tokens, assigning higher weight to patches identified as more informative (empirically, large smooth regions) for real/synthetic discrimination.
  2. Weighted patch aggregation. We multiply this weight matrix elementwise/via matrix multiplication against the patch token features at each layer, producing a re-weighted representation that emphasizes the informative regions before pooling.
  3. Layer-wise flow analysis. We then track the resulting weighted representations layer-by-layer, following the same adjacent-layer transition/discrepancy logic as the original LTD (cosine similarity / difference-based), but applied to the weighted patch aggregate rather than the CLS token alone.

This preserves the core LTD insight - that inter-layer transition stability is diagnostic - while restoring the spatial locality that the original CLS-only formulation discards, which we believe is the piece specifically needed for human-subject imagery.

Results

On the same human-subject evaluation setup (SocialRF + OpenImages-derived human photos) where the reproduced CLS-only LTD reached 71% mean accuracy, our patch-weighted variant reaches 80% mean accuracy - a +9 point improvement - using an otherwise comparable training budget and backbone.

We regard this as a meaningful but partial result: it closes some, not all, of the gap to the accuracy levels reported in the original paper's own domain, and it is based on a single weighting scheme rather than a systematically validated one.

Limitations and Future Work

  • Performance is contingent on scene content, not just generator. Because our patch-weighting scheme relies on identifying large, smooth, low-texture regions as carriers of signal, its benefit is uneven across image content. Images with substantial smooth regions (e.g., sky, ocean, blurred backgrounds, unblemished skin) show the strongest gains, while "busy" images - dense with texture, clutter, or fine detail throughout, leaving few or no large smooth patches - account for most of our failure cases. In the limit, an image with no smooth regions at all gives the weighting scheme little to work with, and performance likely regresses toward the CLS-only baseline. This suggests the current method is less a general-purpose replacement for LTD than a targeted fix for a specific failure mode, and it does not yet address human-subject photos that are cluttered or high-texture throughout. A key open direction is generalizing beyond smooth-region reliance - e.g., a weighting signal that also captures useful structure in high-texture regions, or that degrades gracefully (falling back toward the original CLS-token behavior) when few smooth patches are available.
  • Generator-specific variation unresolved. We observed that different generators seem to leave signal in different spatial regions, which suggests a single fixed weighting scheme is a simplification. A more adaptive, image- or generator-conditioned weighting is a natural next step, and is where we intend to focus follow-up work.
  • No robustness evaluation yet. The original paper reports robustness to JPEG compression and downsampling; we have not yet run analogous stress tests on the patch-weighted variant, which matters given SocialRF-style images are typically re-compressed by platforms.