Decision rules, not slogans
These notes turn public technical guidance into concrete checks an operator can perform before an AI system receives consequential authority.
Each note states its evidence status, strongest countercase, limits, and operator action. A cited draft or concept paper is labeled as such. No source is treated as independent merely because another page repeats it.
Index
The containment boundary is part of the benchmark
Approval must name the object execution will touch
Content is evidence, not authority
One successful run is an observation
How to read an AI leaderboard
Five articles can still be one source
What a system card does and does not prove
A procurement demo is not operating evidence
Prestige does not create source independence
NF-002 · inference from cited risk-management guidance
Seven questions before trust becomes permission
Verdict: A capable agent has not earned authority until an operator can reconstruct its identity, objective, tools, data boundary, evaluation conditions, failure handling, and accountable owner.
- Which exact model, version, scaffold, prompt, and configuration are running?
- What objective is the system pursuing, and who is authorized to change it?
- Which tools and permissions can it exercise?
- Which data can enter, leave, or influence the decision?
- Under which representative conditions was it evaluated, and how often?
- What happens after ambiguity, refusal, timeout, partial success, or conflicting evidence?
- Which named human accepts the residual risk and can stop the system?
Strongest countercase: Low-consequence assistants do not require the same evidence burden as systems that move money, change access, publish claims, or alter production. Correct. The gate should scale with consequence, not disappear.
What this does not prove: Complete answers do not make a system safe; they make its authority and evidence inspectable.
Operator action: Refuse broader authority when any answer is missing, stale, or not enforced outside the model.
Primary records: NIST AI RMF 1.0; AI RMF Core companion; NIST revision status. This is a founder-authored inference: NIST says the Core’s actions are not a checklist or ordered sequence, the framework is voluntary rather than certification, and AI RMF 1.0 is being revised.
NF-003 · inference from benchmark and agent-security guidance
The containment boundary is part of the benchmark
Verdict: An agent evaluation is not reproducible when the record omits what the agent could read, write, call, install, or reach.
A result belongs to the full setup: model, scaffold, tools, credentials, network path, filesystem boundary, time and token limits, reset behavior, monitoring, and grading. If the agent can alter the test or escape what the harness observes, the benchmark no longer measures the declared task.
Strongest countercase: Production-like access can be necessary to test realistic risk. Agreed. Realism increases the need to predeclare, isolate, observe, and reconstruct the permitted boundary.
What this does not prove: Containment proves neither safety nor realism. It makes the conditions and possible interference inspectable.
Operator action: Predeclare permitted actions and invalidate any run whose boundary cannot be reconstructed.
Primary records: NIST.AI.800-2 Initial Public Draft, specifically Practice 2.1, Table 2.2, and Practice 2.2; OWASP AI Agent Security at commit 9feea5a, specifically Tool Security and Least Privilege, Monitoring and Observability, and Secure Agent Testing and Adversarial Validation. NIST AI 800-2 is an initial public draft, not a final standard.
NF-004 · inference from established link-resolution and authorization risks
Approval must name the object execution will touch
Verdict: Approval for a label is not authorization for the object the backend will mutate.
A path may resolve through a symbolic link; a resource name may map to another internal ID; a redirect may change the destination; an alias may outlive the object the user thought it named. A confirmation screen that shows one target while execution resolves another collected consent for the wrong action.
Strongest countercase: Exact binding adds friction when resources change frequently. True. Use narrowly scoped, short-lived authority with explicit renewal—not broad approval whose meaning can drift.
What this does not prove: A truthful confirmation screen does not solve compromised credentials, excessive permissions, race conditions, or malicious code.
Operator action: Resolve and normalize the canonical target, show that exact target and scope, bind approval to durable identity and parameters, recheck immediately before mutation, and fail closed on mismatch.
Primary records and scope: MITRE CWE-59 establishes only the link-following risk; NIST’s canonical Draft status page and concept paper frame specific-action authority, intent, human authorization, and audit questions; OWASP at commit 9feea5a supplies the exact-action approval and fail-closed pattern. The complete control remains a NoFuckery inference, not a claim that any source proves it sufficient.
NF-005 · defensive architecture inference
Content is evidence, not authority
Verdict: Untrusted content may inform a decision. It must never manufacture permission for a consequential action.
Webpages, email, documents, tickets, and tool output can be wrong, compromised, or adversarial. Separate the content plane that gathers information, the decision plane that checks identity, objective, policy, scope, and risk, and the action plane that receives only the minimum authority required for one approved operation.
Strongest countercase: Adding human confirmation does not automatically help; people approve vague prompts and habituate to warnings. Correct. Approval must expose the canonical action and be enforced outside the model context.
What this does not prove: Prompt injection can be eliminated or every high-impact action can be made safe through review.
Operator action: Enforce identity, permissions, target validation, and policy at the tool boundary—not solely in prompt text.
Primary records and scope: OWASP LLM Prompt Injection Prevention at commit 3b50b14, specifically Remote Content Sanitization, Agent-Specific Defenses, Least Privilege, and Model-Based Guardrails; NIST’s canonical Draft status page and concept paper, which frame authorization and prompt-injection questions. The three-plane split is a NoFuckery inference; neither source proves it sufficient.
NF-007 · source reported plus statistical inference
One successful run is an observation
Verdict: One successful agent run is evidence that the outcome occurred under the recorded conditions. By itself, it does not establish a defensible reliability or failure-rate estimate.
One outcome can be a data point in an estimator under explicit assumptions. It cannot quantify frequency or uncertainty from the observed data by itself. A defensible reliability claim requires a declared estimand, unit of analysis, sampling model, repeated attempts, stable configuration, complete accounting for errors and refusals, and uncertainty that matches the decision.
Strongest countercase: One run can still be decisive: a severe failure may stop deployment, a success may disprove impossibility, and one outcome can contribute to an estimate under declared assumptions. Correct. None of those facts makes the estimate precise, generalizable, or decision-ready.
What this does not prove: Repetition alone creates external validity. Repeated trials can preserve unrepresentative tasks, contamination, a broken judge, or unrealistic configuration.
Operator action: Predeclare trials, configuration, exclusions, metric, and uncertainty method; report every attempt.
Primary records: NIST.AI.800-2 Initial Public Draft, released 2026-01-30 and still labeled an initial public draft; NIST.AI.800-3 statistical evaluation guidance, published 2026-02-17.
NF-008 · source reported plus explicit checklist
How to read an AI leaderboard
Verdict: A leaderboard without a measurement contract is screening evidence—not a purchasing or deployment decision record.
NoFuckery’s six-field reading checklist, derived from the cited NIST evaluation guidance:
- Identity: exact model, version, date, provider, and endpoint.
- System: scaffold, prompt, tools, permissions, sampling, and context limits.
- Test: benchmark version, selection, coverage, and contamination controls.
- Execution: trials, timeouts, errors, retries, exclusions, and budget.
- Measurement: estimand, grading method, and uncertainty.
- Reproduction: available artifacts and, as a NoFuckery decision criterion, whether a materially independent party replicated the result.
Strongest countercase: Common tests expose direction, regressions, and promising candidates efficiently. Agreed. Screening value is not deployment authority.
What this does not prove: A lower-ranked model is worse for your workload, or a top-ranked model is reliable, secure, economical, or authorized in your context.
Operator action: Treat rank as a claim about one measurement setup until its conditions match the decision at hand.
Primary records: NIST.AI.800-2 Initial Public Draft, released 2026-01-30 and still labeled an initial public draft; NIST.AI.800-3, published 2026-02-17. NIST does not prescribe this exact six-field sequence.
NF-R01 · reserve · source-lineage rule
Five articles can still be one source
Verdict: Coverage is not corroboration. Five articles repeating one press release remain one evidence path.
Strongest countercase: Derivative reporting can add context, preserve records, expose contradictions, or obtain a response. If a later outlet independently inspects a primary record or gathers new evidence, that portion becomes a new evidence root. Simple repetition does not.
What this does not prove: The originating claim is false; only that repetition has not increased the number of independent evidence roots.
Operator action: Trace authors, records, data, funding, access, and citations before writing “widely confirmed.”
Editorial reference: FRONTLINE journalistic guidelines. NoFuckery applies its own disclosed source-lineage rule; this citation is not a claim of equivalence with FRONTLINE.
NF-R02 · reserve · source reported plus inference
What a system card does and does not prove
Verdict: A developer-authored system card is valuable first-party evidence. It is not deployment permission.
Read it for the exact release and configuration, test design and graders, disclosed failures and exclusions, independent assessments, post-test changes, and operating contexts never evaluated.
Strongest countercase: Developer access can make a system card more technically complete than outside commentary. Correct. First-party evidence is not weak because it is first-party; its provenance and limits must stay visible.
What this does not prove: The developer-authored card alone does not establish independent reproduction or fitness in your deployment context unless it links to inspectable independent evidence.
Operator action: Translate each material card claim into a deployment-specific acceptance test, monitoring signal, or documented unknown.
Primary records: In OpenAI’s January 2025 Operator System Card—a historical record for the then-research-preview CUA—OpenAI warned that perfect evaluation performance would not guarantee matching real-world deployment performance. OpenAI later integrated Operator into ChatGPT agent and sunset the standalone experience. See also NIST.AI.800-4, published 2026-03-06. This note does not imply that the historical card describes OpenAI’s current browser-agent product.
NF-R03 · reserve · procurement inference
A procurement demo is not operating evidence
Verdict: At most, a polished demo is evidence of a result under arranged conditions. Without an inspectable record of the system, inputs, operator interventions, and execution, it may not establish even that.
Strongest countercase: Demos efficiently make capabilities concrete. Correct. The error is treating an arranged result as acceptance evidence.
What this does not prove: A vendor cannot meet operating requirements, or every low-risk purchase needs the same burden as a high-impact deployment.
Operator action: Convert the demo into a written acceptance protocol using representative, vendor-inaccessible cases; repeatable testing; explicit failure handling; monitoring requirements; contractual access to evidence; and an accountable acceptance decision.
Primary records: GAO-26-107859; NIST AI RMF 1.0; NIST AI RMF revision status; NIST.AI.800-4. AI RMF 1.0 is voluntary and currently under revision. This is general risk guidance, not legal or procurement advice.
NF-R04 · reserve · source-independence rule
Prestige does not create source independence
Verdict: Two respected institutions are not automatically two independent evidence paths.
They may share model or dataset access, authors, advisers, funding, partners, infrastructure, an incident account, or incentives. Those relationships do not make the work false. They change what “independently corroborated” can honestly mean.
Independence belongs to an evidence path for a particular claim, not permanently to an institution.
Strongest countercase: Technical work is collaborative; demanding total separation can exclude people with needed access and expertise. Agreed. Independence is a disclosed degree, not a purity test.
What this does not prove: Funding, partnership, access, or a declared perspective caused bias or error.
Operator action: Record two separate dimensions for each material claim: evidence lineage—first-party, affiliated, or materially independent—and incentive or stance, including supportive, neutral, or adversarial. Do not use adversarial posture as a substitute for provenance.
Editorial reference: FRONTLINE journalistic guidelines. This note states NoFuckery's rule and alleges no undisclosed conflict.