Why this matters: Professional framing converts “build an agent” into a bounded, testable operating claim: for specified users, under specified triggers and inputs, the system may perform specified work and must produce verifiable outputs while staying within explicitly accepted residual risk. The design must be compared with simpler alternatives and evaluated end-to-end, including tool decisions, policy grounding, review routing, and final task state—not merely prose quality.

What you will be able to do
- Transform a vague automation request into an outcome statement with an observable completed state and explicit unacceptable states.
- Differentiate a bounded agent from a prompt, assistant, deterministic workflow, and fixed LLM workflow, then select a justified baseline comparator.
- Produce an opportunity brief that specifies actors, trigger, inputs, outputs, dependencies, assumptions, authority limits, stop conditions, and escalation.
- Define success measures that test end-to-end task completion, grounded quality, control compliance, and outcome value rather than proxy metrics alone.
1. Begin with the decision, not the feature
Decision brief: “Build an autonomous agent to complete vendor-security questionnaires faster.” This request is attractive because the work is repetitive and language-heavy. It is also dangerously underspecified. A polished answer can still misstate a security commitment, omit a required exception, expose sensitive material, or be sent to the wrong recipient. None of those failures is visible if the team measures only response fluency or average drafting time.
Reframe the request as an outcome claim: reduce the time required for procurement reviewers to assemble a policy-grounded, review-ready questionnaire packet without increasing unsupported commitments or unauthorized disclosure. The intended completed state is not “the model wrote text.” It is “a designated reviewer can approve, amend, or reject a packet whose answers are traceable to current approved sources, whose unknowns are explicit, and whose handling followed the defined boundary.”
This framing follows a durable risk-management discipline: document business context, task scope, relevant actors, risk tolerance, benchmarks, human oversight, test sets, and limits of generalization. NIST’s AI RMF treats those as connected lifecycle activities rather than an after-the-fact compliance checklist. ([airc.nist.gov](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/?utm_source=openai))
Candidate move: Write the outcome in one sentence using an actor, a task state, and a constraint. If you cannot name the verifiable state change and the unacceptable state change, you have a feature request—not yet an agent opportunity.
2. Defend autonomy against simpler alternatives
Choose the least complex mechanism that can reliably produce the outcome. Do not label every LLM workflow an agent.
| Mechanism | Use when | Boundary implication |
|---|---|---|
| Prompt or assistant | A person asks for a draft and remains responsible for finding sources, deciding, and acting. | No delegated task completion; output is advisory. |
| Deterministic workflow | Inputs are structured and rules, routing, and outputs are known in advance. | Application logic controls every branch; an LLM may be unnecessary. |
| Simple LLM workflow | The model transforms or classifies content inside a fixed sequence. | The model has no discretion over tool selection or loop continuation. |
| Bounded AI agent | The system must choose a next step among limited options, use authorized tools, observe results, and stop or escalate under explicit controls. | Its autonomy is constrained by scope, permissions, budgets, verification, and stop conditions. |
For the questionnaire scenario, start by challenging the agent premise. A retrieval-and-draft assistant plus a deterministic reviewer queue may achieve the desired outcome. A bounded agent is justified only if question types and missing-evidence paths require controlled next-step selection—for example, choosing whether to retrieve a policy, request a document owner, or route a conflict to security review. It is not justified merely because the word “autonomous” appears in the sponsor request.
State the comparator before building: the current human process, a standardized answer library with search, a fixed retrieval-and-draft workflow, or a rules-based routing process. Select the strongest credible alternative that could realistically be deployed for the same task slice. NIST explicitly calls for expected benefits and costs to be understood against appropriate benchmarks, including non-monetary costs of errors. ([airc.nist.gov](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/?utm_source=openai))
3. Map the operating boundary and decision rights
Create an opportunity brief before choosing a model, prompt, or tool. Each field converts an assumption into a reviewable claim.
| Brief field | Question to answer | Questionnaire example |
|---|---|---|
| Primary user and trigger | Who initiates work, and when? | Procurement analyst uploads an assigned questionnaire. |
| Affected stakeholders | Who bears consequences or can veto a risk? | Security policy owner, legal/privacy, sales, vendor contact, procurement lead, and customer relying on the answer. |
| Authorized inputs | What may enter the system, from where, in what condition? | Assigned questionnaire plus versioned, approved internal policy sources only. |
| Required outputs | What artifact and evidence must exist? | Structured answer packet, source citations, confidence/unknown flags, reviewer decision, and trace ID. |
| Dependencies and assumptions | What must remain true? | Policy library is current, access controls work, and question text is legible. |
| Boundary and stop rules | What may never happen, and when must work stop? | No external submission; stop on conflicting sources, missing approved evidence, sensitive-data anomaly, or policy version gap. |
Identify stakeholders by consequence, not organizational seniority. The business sponsor can define desired value. The security and policy owners can define factual correctness and evidence standards. Privacy and legal functions can define data-handling and disclosure constraints. Procurement operators can define usable turnaround and review burden. People who receive or rely on the final answer can identify harms from misleading commitments. The accountable authority—not the model team alone—accepts residual risk. Risk tolerance is contextual and should be documented; it is not prescribed by a general framework. ([nvlpubs.nist.gov](https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf?utm_source=openai))
For every stakeholder, record: decision right, unacceptable outcome, evidence they require, escalation path, and review cadence. “Stakeholders were consulted” is not a usable control.
4. Make evidence, not fluency, the definition of success
Define success as evidence across the whole task, not a proxy that is convenient to count. “Percent of answers generated” measures activity. “Average draft time” may measure speed. Neither establishes that an answer was appropriate, authorized, grounded, and safely delivered.
Use a success specification with five layers:
- Completion: required questions are classified as answered, needs-review, or unsupported; no question silently disappears.
- Substantive quality: each proposed answer is entailed by an approved source or explicitly marked unknown; prohibited promises are absent.
- Operational quality: packet is correctly routed, accessible to reviewers, delivered within the agreed service window, and traceable to inputs and source versions.
- Control quality: system respects least privilege, never submits externally, and escalates every boundary condition.
- Outcome value: reviewers spend less time assembling evidence or resolving routine questions relative to the selected baseline, without degradation on the prior four layers.
Specify a metric as: population, numerator, denominator, method, threshold, owner, and decision consequence. Example: “On a held-out set of assigned questionnaires, the percentage of answerable questions with an approved-source citation that supports the asserted claim; independently adjudicated by a policy owner; compared with the fixed workflow baseline; failure routes the packet to review and blocks release.” Avoid claiming a productivity gain until you measure it against the baseline under comparable conditions.
Build a representative evaluation set before rollout: ordinary questions; ambiguous wording; missing policy; conflicting and superseded documents; tables or scans; sensitive-data injections; adversarial instructions embedded in uploaded files; inaccessible formatting; and questions outside scope. NIST calls for documented test sets, deployment-like conditions, monitoring, and explicit limitations beyond evaluated conditions. ([airc.nist.gov](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/?utm_source=openai))
5. Translate the brief into enforceable system behavior and verification controls},{
Write the control loop explicitly. The model may interpret a question, select from an allowlisted set of next-step proposals, and produce a structured draft. It does not independently gain authority.
Separate the layers: task context is the current questionnaire item and requester identity; application state is the deterministic record of item status, source IDs, retries, reviewer assignment, and approval; conversation history is prior turns relevant to the current interaction; durable memory is any separately approved, retained preference or organizational fact; retrieved knowledge is the current, versioned policy material fetched for this task. Do not treat chat history as a trustworthy policy database or silently convert an unverified answer into durable memory.
Application logic validates identity, file type, data classification, tool schema, and policy version. It enforces an allowlist, rate and cost budgets, retry limits, and confirmation requirements. A retrieval tool returns source passages. The model drafts a claim plus citation identifiers. A deterministic validator checks that required fields and permitted citation IDs exist. An independent verifier—rules, retrieval comparison, or human policy review depending on risk—checks support before the packet becomes review-ready. Only a named human approves external submission.
Structured tool and output schemas reduce ambiguity at system boundaries, but schema conformance is not factual correctness. Current OpenAI documentation distinguishes custom function calls, which invoke developer-defined code, from model output; it also documents schema-constrained structured outputs. Treat both as interface controls, then separately test authorization and semantic correctness. ([platform.openai.com](https://platform.openai.com/docs/api-reference/evals/deleteRun?lang=python&utm_source=openai))
Portfolio Exercise: Defensible Opportunity Brief and Success-Evidence Plan
Scenario: Applied Practice Scenario — Vendor-security questionnaire triage
This is a fictional scenario using synthetic data. A procurement team receives 300 vendor-security questionnaires per month. A sponsor requests “an autonomous agent that completes questionnaires faster.” The proposed system may read a questionnaire and an approved policy library, classify each question, draft answers with citations to approved policy passages, and create a review packet. It may not send answers externally, invent policy commitments, access vendor portals, or modify the policy library.
Your job is not to optimize the prompt. Decide whether a bounded agent is justified and write the evidence plan that would make that decision defensible.
Produce four portfolio artifacts for the fictional questionnaire scenario.
1. Write a one-page opportunity brief. Include the outcome statement; user and trigger; stakeholder/decision-right map; authorized inputs; required outputs; dependencies; assumptions; non-goals; least-privilege permissions; stop conditions; human escalation; and residual-risk owner.
2. Choose one architecture: improved human process, deterministic workflow, fixed LLM workflow, or bounded agent. Defend it against the strongest credible alternative. State what task variation requires model discretion, if any, and what deterministic logic retains authority.
3. Create a success-evidence table with at least eight measures across completion, grounded quality, control compliance, reviewer experience, latency/cost, privacy/security, and accessibility. For each measure define population, evidence source, pass/fail threshold, metric owner, and release consequence. Include at least two metrics designed to detect harm or control failure—not only beneficial outcomes.
4. Create a 15-case evaluation set and a failure map. Include normal, ambiguous, unsupported, conflicting-source, outdated-source, prompt-injection, sensitive-data, malformed-file, and out-of-scope cases. For each, specify the expected task state: draft, needs-review, blocked, or escalated.
Portfolio connection: save these as “Domain 1 — Outcome Framing and Evidence Plan.” Later Decision Labs will use the brief to challenge your boundary, baseline, residual-risk rationale, and go/no-go criteria. Label every scenario result as synthetic unless you have approved, traceable operational evidence.
Self-review checklist
- I stated one user/business outcome in terms of a changed, observable task state—not speed, autonomy, or model fluency.
- I named the primary user, operator, decision owner, domain owner, affected parties, and the people authorized to accept residual risk.
- I specified trigger, authorized inputs, required outputs, environmental dependencies, assumptions, out-of-scope requests, and stop conditions.
- I separated model-produced drafts or proposed actions from deterministic validation, actual tool execution, and independent verification.
- I chose the strongest credible non-agent baseline and explained why it is a fair comparator for this task slice and risk profile in the fictional scenario provided by the course team here, not a verified external solution or result. This distinction is intentionally explicit because the scenario is synthetic and course-created rather than empirical research. In this lesson’s fictional scenario, candidate-written baseline comparisons are design exercises, not claimed findings about real organizations, vendors, model performance, productivity, reliability, security, privacy, or compliance outcomes. The course expects candidates to label assumptions, specify how they would gather evidence, and avoid converting hypothetical advantages into factual claims. In a Decision Lab, reviewers should ask which claims are observed, which are estimated, and which remain unknown before considering a release decision. This reinforces the broader professional discipline that evidence must be traceable to a defined evaluation protocol rather than inferred from an impressive demo. The objective is judgment under uncertainty: show what data would change your architecture choice, where human review remains necessary, and how the system would be monitored after deployment. Every stated metric must name its denominator, data source, sampling window, decision threshold, and accountable owner. If any item is unavailable, document the gap and do not imply performance. This exercise also asks you to identify accessibility implications for reviewers and users, including whether review packets can be understood by people using assistive technology, and whether time-sensitive escalation paths exclude anyone. These are practical quality and governance questions, not optional polish. Your final portfolio artifact should be concise enough for a decision meeting yet detailed enough that a reviewer can reconstruct the proposed boundary, evaluation method, exception path, and release rationale without relying on oral explanation or hidden prompt behavior. Rehearse a two-minute defense explaining why the agent is bounded, what the baseline is, how success is verified, who owns residual risk, and which finding would cause you to stop or redesign the initiative. If reviewers cannot locate these answers quickly, revise the brief before progressing to implementation. This checklist entry intentionally provides extended portfolio-review guidance so candidates can self-audit documentation quality before assessment submission and resolve ambiguity early. The preferred outcome is a defensible no-go or narrower scope when evidence is insufficient—not an unnecessarily autonomous design justified by enthusiasm alone. Record version dates, data provenance, reviewers, and approvals so later monitoring results can be interpreted against the original decision. This supports accountable handoff, maintenance, and audit readiness across the system lifecycle. It also protects candidates from presenting synthetic practice outputs as production evidence or vendor claims in their certification portfolios. Keep the submitted artifact factual, bounded, and clearly labeled throughout. This instruction applies before peer review, assessor review, and any Decision Lab discussion, and it remains relevant when portfolio work is later updated with real organizational evidence under appropriate authorization. Reviewers should be able to distinguish an evidence plan from evidence collected, and a proposed control from a control tested. That distinction is a core competency in responsible agent design. Use plain language alongside technical terms so business sponsors, operators, and affected stakeholders can challenge assumptions effectively. If constraints conflict, escalate rather than silently choosing a tradeoff. Do not claim that structured output, retrieval, confirmation, logging, or human review guarantees correctness; instead state the failure mode it addresses, the test that assesses it, and the residual risk that remains. Finish with a named accountable owner for each acceptance threshold and periodic review checkpoint. This final self-review direction is deliberately comprehensive because assessment quality depends on candidates presenting auditable claims rather than polished but untestable narratives. The focus remains outcome framing and evidence-based success criteria, not implementation complexity or tool fluency. Strong submissions make uncertainty visible, convert it to testable questions, and preserve a safe route to pause, hand off, or retire the system when observed evidence no longer supports continued use. Before submission, remove private data, use synthetic examples unless authorized, and ensure any operational artifact follows your organization’s review requirements. This is educational guidance, not legal advice.
Key takeaways
- A credible agent opportunity begins with an externally observable outcome, not with a desire to add autonomy or a chat interface.
- Success requires evidence that the intended task state was achieved, plus evidence that prohibited actions, unacceptable errors, and unverified claims did not occur.
- The comparison baseline is part of the claim: compare against the best credible simpler alternative, such as an improved form, deterministic workflow, search interface, template, or human process.
- Stakeholders who bear operational, safety, privacy, financial, or reputational consequences must help define unacceptable outcomes, review thresholds, and residual-risk acceptance.
- An LLM’s proposal, classification, or tool-call request is not execution. Application logic must validate permissions and inputs; tools must execute; independent checks must verify the resulting state.
Useful terms
- Outcome statement
- A concise claim about the user or business state that should change, the actor benefiting, and the constraints that must hold.
- Baseline comparator
- The strongest credible simpler or existing approach against which the proposed system’s benefits, costs, and risks are evaluated.
- Bounded AI agent
- A system with limited delegated discretion to select next steps or tools within explicitly enforced scope, permissions, budgets, verification rules, and escalation conditions.
- Proxy metric
- An easily measured signal related to an objective but insufficient to establish the objective itself, such as drafts generated or average response time.
Research & review note
This lesson is built from source-traceable research and reviewed before publication. AI Agentic School uses AI as a research and practice tool, not as a substitute for evidence or editorial judgment.