November 5, 2026
How to Choose an AI Consultant Who Actually Knows the Failure Modes
"AI expert" is not a protected title. Use these 8 questions to tell real AI integration from quick wiring before you sign anything.
- AI consultant
- AI washing
- AI integration

In short: a good AI consultant is not defined by the title, which has no legal definition, but by what they know about failure: when AI gets things wrong, how often, and what happens next. To check, ask eight specific questions about production track record, quality measurement, your data, real cost and long-term maintenance.
In 2023, two New York lawyers with decades of practice between them filed a federal court brief packed with case law that did not exist. ChatGPT had invented it, complete with docket numbers and quotes. In Mata v. Avianca, the judge sanctioned them $5,000.
Those lawyers knew how to use the tool. What they did not know was how it fails. That gap is the whole difference between a skilled user and an expert.
What is an AI consultant, exactly?
An AI consultant is a provider who helps an organization integrate artificial intelligence into its real processes: its documents, tools, business rules and confidentiality constraints. That is not the same as writing clever prompts.
The catch is that anyone can use the title. “AI expert” is not a degree, a regulated profession or an official certification. Anyone can put it on their profile tomorrow morning.
Nobody counts the fake AI experts. That is exactly why you need to know how to spot them yourself.
Why do so many AI projects fail after the demo?
They fail because a demo tests the technology, while production tests the integration. A demo runs on three hand-picked examples. Your daily operations contain thousands of edge cases, badly named files and exceptions.
You will often hear that the vast majority of AI projects never deliver. The exact figure is debated, but the diagnosis is not: integration, not the technology, is where projects break.
There is even a name for the inflated promises: AI washing. AI washing is presenting a product or service as AI-powered when it is not, or far less than claimed. In March 2024, the US SEC charged two investment advisers a combined $400,000 over misleading AI claims.
The same pattern now affects “agents.” According to Gartner, as reported by Reuters, only about 130 of the thousands of vendors claiming agentic AI offer real capabilities. We cover this in depth in Agentic AI and “agent washing”.
Quick wiring vs real integration: what is the difference?
Quick wiring connects a consumer AI tool to your files and leaves you to discover the problems later. Real integration starts from your processes, measures quality and plans for what happens when the system is wrong.
| Criterion | Quick wiring | Real integration |
|---|---|---|
| Starting point | The trendy tool | Your process and its pain points |
| Quality | “It worked great in the demo” | Measured on a test set built from your data |
| Errors | Discovered by your customers | Anticipated, detected, corrected |
| Data | “We use a big provider’s API” | Documented data flow, contract, clear GDPR treatment |
| Cost | Launch price | Total 12-month cost, maintenance included |
| After the consultant leaves | Black box | Documentation and knowledge transfer |
If you remember one row, make it the errors row. A large language model, the statistical engine behind ChatGPT and its cousins, produces plausible text, not verified text. It will be wrong sometimes. The only question is who notices, and when.
Air Canada learned this the hard way in 2024, when its chatbot invented a refund policy. We unpack that case and its legal consequences in Who is liable when your AI chatbot gets it wrong?Who is liable when your AI chatbot gets it wrong?.
What questions should you ask an AI consultant before signing?
Ask questions that force them to show, not promise. Here is our checklist, ready to print or share with your leadership team.
The 8-question filter
- Show me a comparable project that is still in production 12 months later. A pilot does not count. What counts is what survived daily use.
- How do you measure quality? On what test set, with what data? A test set is a batch of questions with known correct answers. Without one, “it works” is just a feeling.
- What happens when the model gets it wrong? Show me the mechanism. Source citations, confidence thresholds, human review: they should be able to show it on screen.
- Where does our data go? Which provider, what contract, what GDPR treatment? Regulators such as the French data protection authority CNIL publish detailed guidance on this. A real expert knows it.
- What is the total cost over 12 months? Subscriptions, API usage (pay-per-use billing for the model), monitoring, maintenance, updates.
- Which part of my process does NOT need AI? A real expert will often tell you a simple rule or better file organization is enough.
- Tell me about a project that went wrong and what you learned. Someone who has never failed has never shipped anything.
- Who maintains the system when you are gone? Where is the documentation? If the answer is “me,” you are buying a dependency, not a solution.
How should you read the answers?
A good sign is a precise answer, with numbers and an example. A bad sign is a buzzword where a mechanism should be: “our AI improves on its own,” “the model almost never makes mistakes,” “it is secure, it is in the cloud.”
Be wary, too, of the expert who asks you nothing back. Someone who truly understands AI wants to see your documents, exceptions and constraints before promising anything.
How can a non-technical buyer evaluate an AI consultant?
You do not need to understand the code; you need observable proof. Ask for a demo on your own documents, not theirs, and count the errors together.
Three simple habits:
- Test with your hard cases. Bring ten questions you already know the answers to, including two trick ones.
- Ask where every answer comes from. If the system cannot cite its source, you cannot check it.
- Have the contract reviewed on three points: where data is stored, reversibility (the ability to leave with your data) and liability for errors.
This is also where you notice what is usually missing: context. An AI without access to your rules and documents answers “in general,” never “for you.”
What if the tool already answered half the questions?
Several of the eight questions are not really about the consultant; they are about the architecture they install. That is exactly the architecture we built Sceptra on.
Sceptra is an AI workspace that runs on your own infrastructure, with the model of your choice. Your data stays with you (question 4); if you leave, you take it with you (question 8). Every answer cites its sources, drawn from your documents rather than the open web (question 3). Work is organized in spaces: the AI working on Client A’s file sees Client A and nothing else.
Data leaves only through connections your administrator has explicitly created, and everything that leaves is logged. You define your standards and formats, and they apply to the whole team, every time. Everything is reversible.
A good consultant is still essential to understand your processes. They will simply work better on a foundation that makes errors visible and keeps data under control.
Frequently asked questions
Is “AI consultant” a regulated title? No. As of 2026, “AI consultant” and “AI expert” are neither regulated professions nor protected titles, and no official certification is required to use them. It is up to the buyer to verify skills through production references and demos run on the buyer’s own data.
How much does an AI consultant cost? Rates vary widely by engagement. The number that matters is total cost over 12 months: fees, subscriptions, API usage, monitoring and maintenance. Get that estimate in writing before you sign, because a cheap integration can become expensive to maintain.
What is AI washing? AI washing is marketing a product as powered by artificial intelligence when it is not, or far less than claimed. In March 2024, the US SEC charged two investment advisers a combined $400,000 in penalties for this kind of misleading claim.
Does a small business need an AI consultant? Not always. For a one-off need, a well-chosen tool may be enough. A consultant becomes worthwhile once AI touches confidential data, customers or binding decisions. In that case, question 6 (“what does not need AI?”) will often save you a lot of money.
What is the difference between a power user and an AI expert? A power user knows how to get good results from the tools. An expert knows when they fail, how often and what to do next. Mata v. Avianca shows it: knowing how to use ChatGPT did not stop experienced lawyers from citing invented rulings.
Key takeaways
- “AI expert” has no legal definition: judge on evidence, not titles.
- Demos test the technology; production tests the integration.
- A real expert talks first about failures, measurement and your data.
- The 8-question filter separates quick wiring from real integration in one meeting.
- An architecture that cites sources and keeps your data in-house already answers several of those questions.
Evaluating an AI vendor or project right now? Book a 30-minute call and we will run your project through the 8-question filter, with no strings attached. You can also try the domain-by-domain simulator at sceptra.ai.