How to read skill poisoning research

Begin with the threat model

Identify what the researcher controls: the instruction file, a supporting script, a registry entry, a learned artifact, or the environment. Then identify the target’s permissions and the intended harmful outcome.

Two studies can both concern skill poisoning while evaluating fundamentally different carriers and authority profiles. Compare them only after matching those assumptions.

Read the denominator

Ask what counts as a test case and a successful attack. Is success a model response, a proposed tool call, an attempted action, or a verified effect? How many tasks were attempted, and which were excluded?

An attack success rate without a denominator and scoring definition is difficult to interpret. Look for repeated trials, failed legitimate tasks, and uncertainty around the result.

Distinguish datasets

A marketplace survey estimates properties of its sample and analysis method. A constructed benchmark tests behavior under selected cases. A simulation studies a controlled mechanism. Each provides evidence; none alone measures all production deployments.

For example, the catalog includes skill-file injection evaluation, model-in-skill backdoors, and lifecycle analysis. Treat their findings as complementary rather than combining the percentages into one headline.

Evaluate a defense claim

Check whether the defense was tested on previously unseen cases and whether it retained useful task completion. Find the backend, model version, thresholds, and false-positive analysis. Ask what happens when a case is missed.

A benchmark result is strongest when it gives reproducible conditions and a clear scope. “Perfect detection” on a finite test set does not prove that every future attack is detected.

A paper-reading worksheet

• Carrier and attacker control.
• Target model, client, and permissions.
• Task set and scoring definition.
• Security and utility outcomes.
• Baselines and comparison conditions.
• Limitations, artifacts, and reproducibility.

Use the catalog’s “read with care” notes as a starting point, then consult the original paper before quoting a result in a product or investment decision.

Worked example: the same numerator tells different stories

Consider a hypothetical test with 100 trials: 20 produce unauthorized code, five execute the targeted operation, 15 fail for unrelated reasons, and the rest refuse. A paper might count generated-plus-executed output for one metric and executed operations for another. Reading both as 25 confirmed compromises would change the meaning of the experiment.

Work through the evidence

Find the outcome definitions before comparing percentages. Record what enters the denominator, how errors are handled, whether a package is already loaded, and which actions count as success. Determine whether the test changes discovery, selection, execution, or all three. A native attack dataset and an adapted baseline may not cover the same cases.

Decision criteria and a useful review record

Quote the metric by its definition, alongside host, model, task set, permissions, and limits. Keep your own derived numbers labeled as calculations rather than reported results. Compare outcomes at compatible scopes or explain why a direct ranking is not justified. The purpose is to understand evidence, not manufacture a single league table from incompatible experiments.