7 questions to ask before you buy an AI platform
Seven questions for evaluating AI platforms: permissions, human approvals, audit records, failure handling, deployment, model choice and pilots.
Before buying an AI platform, ask what it can touch, what requires human approval, what record it leaves, how failures are handled, where it runs, whether models can change and whether you can test it on your own work.
Ask a vendor seven things before you sign: what the AI can touch, which actions stop for a person, what record it leaves, what happens when it is wrong, where it runs, whether you can change the model later and whether you can test it on your own work first.
A vendor who can show each answer inside the product is worth a pilot, which is a short, limited trial. In our view, if three or more answers exist only on a slide and cannot be demonstrated in the product, the platform is not ready for consequential work. This guide gives each question, what a good answer sounds like and what a weak one sounds like.
Why these questions matter
Most buying guides compare features. An AI agent, meaning software that carries out tasks such as sending a message, updating a record or placing an order, needs a different test. The risk is not only what it says, but what it does.
Our earlier guide compares vertical and horizontal AI tools. This guide covers the seven questions to ask once you have a shortlist. It draws on the OWASP Gen AI Security Project's 2025 list of AI application risks and the NIST AI Risk Management Framework 1.0. It is Trunnion AI's opinion, not legal or professional advice, and it does not replace your own review. Sources are listed at the end.
The seven questions at a glance
| # | Ask the vendor | A good answer | A weak answer |
|---|---|---|---|
| 1 | What can it touch? | A list of tools and permissions, limited to the task | "It connects to everything" |
| 2 | What stops for a person? | A set limit above which a named person must approve | "Users can review if they want" |
| 3 | What record does it leave? | A log tying who, what, why and result together | "You can export chat history" |
| 4 | What happens when it is wrong? | A live walk-through of tracing and fixing one failed run | A policy document |
| 5 | Where does it run? | Cloud, on-premises and disconnected options, with the limits stated | "It runs anywhere" |
| 6 | Can we change the model? | Model choice kept separate from rules and records | Rules rebuilt for each model |
| 7 | Can we test it on our work? | A trial on a copy of your records with a stated goal | A canned demo |
1. What can it touch?
Start with access, because it sets the ceiling on every mistake. The OWASP Gen AI Security Project lists this risk as LLM06 in its 2025 list and names three root causes of an AI doing damage it should not: too much functionality, excess permissions and too much autonomy. In plain words, the AI has more tools, more rights or more freedom than its job needs.
Ask for the exact list of tools the agent can call and, for each one, whether it can read, write or delete. Those are three separate rights. Then ask whether those rights follow the task or the person who set it up.
A good platform checks the agent's role, its tenant, meaning the customer account it is working in, and the data classification, meaning how sensitive the data is, every time it touches a resource. Trunnion's platform is designed to work this way through attribute-based access control, where permissions come from who is asking, what is being touched and the context of the request, not from one shared login.
Common mistake, in our view: a pilot receives an administrator account "to make setup easy" and keeps it. The fix is to write the permission list before setup and give the agent only the tools the task needs.
2. What stops for a person?
Ask for the line between what the agent does alone and what waits for a human. One of OWASP's prevention steps is to require a person to approve high-impact actions before they are taken. The useful detail is where the line sits and what the approval covers.
On Trunnion's platform, actions above a defined consequence level pause at a human approval gate. The approval is tied to the specific request, expires and can be used only as policy allows. If a vendor describes an approval as a general permission that lasts all week, that is a weak answer.
Worked example, made up for illustration: an agent drafts purchase orders from supplier emails, and your rule is that anything over $2,500 needs a manager. On a day with 40 orders, three are over the limit. Those three stop for approval. The other 37 go through, and all 40 are logged.
3. What record does it leave?
Ask what the record contains, not whether one exists. A useful record, often called an audit trail, ties together at least six things: who asked, what task ran, which rule applied, who approved, which tool was called and what came back.
Trunnion's platform is designed to keep those links and to make later changes detectable with a hash-chained audit trail. Each entry is tied cryptographically to the one before it, so changing an older entry breaks the chain and becomes visible. That matters in a dispute because a record that anyone can quietly edit is weak evidence.
Ask the vendor to show one representative record, then edit one entry in a test copy and show what the break looks like.
4. What happens when it is wrong?
Every agent will be wrong sometimes. The question is how fast you can find out why.
The NIST AI Risk Management Framework, a voluntary guide from the US National Institute of Standards and Technology, organizes the work into four functions: govern, map, measure and manage. NIST describes governance as a cross-cutting function that informs the other three. In practice, someone needs to own the system throughout its lifecycle, not only at launch.
Ask the vendor to walk you through one failed run live: who is told, how the cause is traced and how the fix is tested. Trunnion's platform is designed to record the action, policy decision and approval context so a run can be reconstructed rather than described from memory.
Honest limit: if the AI only drafts text that a person reads before anything happens, questions one through three matter less. They matter most when the agent can change a record, send a message or spend money.
5. Where does it run?
Ask for the exact places it can run and what each one still depends on. On-premises software, meaning software that runs on your own servers, can still call outside services such as hosted models, licensing systems or telemetry. An air-gapped system has no connection to outside networks, so it cannot require those paths while it runs.
Trunnion is designed to support cloud, on-premises and air-gapped deployment. A complete disconnected setup needs local models, identity, policy, tools, audit storage and update procedures in place. Ask any vendor the same plain question: if the connection is cut for a day, what still works?
For a deeper technical treatment, read Running AI agents in air-gapped environments.
6. Can we change the model later?
Ask whether the three things you rely on, rules, approvals and audit history, stay in place if you swap the model. The model is the AI engine underneath the platform, and models change quickly.
Trunnion's design keeps model choice behind a governed endpoint, so changing a provider should not erase policy or audit history, although each model still has to be evaluated for the task. A weak answer is that rules are rewritten for each model because every change then resets your safety work.
After any swap, rerun the same trial, such as the 40-email test in question seven, and compare the two results.
7. Can we test it on our own work?
Ask for a trial on a copy of your own records, with a goal written down before it starts. NIST's map function is about setting the context first: what the system is for, who it affects and what could go wrong.
Agree on three things in writing: the task, the pass mark and the stop rule. For example: "40 supplier emails, at least 38 correct, and any wrong order stops the trial for review."
Our rule of thumb is two to four weeks, long enough to see an ordinary period of exceptions rather than only a best-case week. The right duration still depends on the workflow's volume, risk and variation.
Your pre-buy checklist
- Get the tool and permission list in writing.
- Find the approval line and see one approval happen.
- See one representative audit record.
- Watch one failed run get traced and fixed.
- Confirm where it runs and what each location depends on.
- Confirm the model can change without rebuilding the rules.
- Run a trial on your own records with a pass mark and stop rule.
Our rule of thumb: if the vendor can only describe three or more of these and cannot demonstrate them in the product, keep looking.
Next step
If you want these questions answered against a real release instead of a slide, request a focused walkthrough. Bring these seven questions and one workflow your team actually runs.
Details current as of October 27, 2026. NIST states that AI RMF 1.0 is being updated, so confirm framework language against the linked source before relying on it.
This article is for general informational and educational purposes only and is not legal, financial, medical or other professional advice. We make no guarantee that it is accurate, complete or current. The views expressed are those of the author and do not necessarily represent Trunnion AI. Use of this information is at your own risk.
Sources
Accessed October 27, 2026.
- OWASP Gen AI Security Project. LLM06:2025 Excessive Agency.
- National Institute of Standards and Technology. AI Risk Management Framework 1.0: Executive Summary.
- National Institute of Standards and Technology. AI RMF Core.
- Trunnion AI. Agent governance.
- Trunnion AI. Running AI agents in air-gapped environments.
- Trunnion AI. Frequently asked questions.
Direct answers
Frequently asked questions
What should I ask before buying an AI platform?
Ask what the platform can touch, what stops for a person, what record it leaves, what happens when it is wrong, where it runs, whether the model can change and whether you can test it on your own work.
How should an AI platform limit access?
The vendor should identify each connected tool and separate read, write and delete permissions. Access should be limited to the task, user, tenant and data classification involved.
Which AI actions should require human approval?
Consequential actions should pause at a defined threshold for a named person to approve, edit or reject. The approval should apply to the specific request and expire according to policy.
What should an AI audit record contain?
A useful record connects who requested the task, what ran, which policy applied, who approved it, which tool was called and what result came back.
Can an AI platform run in an air-gapped environment?
It can only operate fully disconnected when the required models, identity, policy, tools, audit storage and update process are all available inside the environment.
Should an AI platform let buyers change models?
Yes. Model choice should remain separate from business rules, approvals and audit history, although every replacement model still needs evaluation against the intended task.
How should we pilot an AI platform?
Use a copy of your own records and write down the task, pass mark and stop rule before the trial begins. The pilot should be long enough to encounter ordinary exceptions, not only a best-case demo.