How to Evaluate an AI Automation Consultant
A no-nonsense checklist for evaluating AI automation consultants — the questions that separate practitioners from theorists.
The AI Consultant Problem
The AI consulting market is flooded right now. Every generalist consultant has added "AI strategy" to their services page. Most of them have played with ChatGPT and read some blog posts. Few of them have actually built and deployed production AI systems that solve real business problems.
How do you tell the difference?
The Questions That Separate Practitioners from Theorists
"Can you walk me through the last AI automation you deployed into production?"
This is question one. If they can't give you a specific, detailed answer — the workflow, the data, the model, the integration, the results, what broke and how they fixed it — they haven't done it. Vague answers about "working with LLMs" or "developing AI strategies" are not the same as building and shipping systems.
What you want to hear: specific technologies used, specific measurable outcomes, specific problems that came up during implementation, and specific decisions made to address them. Practitioners have stories. Theorists have frameworks.
"What would make this project fail?"
Anyone who's built AI systems in production knows exactly what makes them fail: poor data quality, undefined acceptance criteria, model drift, scope creep, lack of stakeholder adoption, insufficient monitoring. If they can articulate your specific risk factors clearly and early, they've been there.
If they give you a confident answer about why this particular project is low-risk, be skeptical. Every AI project has genuine risks. Experience with failure is a feature, not a bug.
"How do you handle accuracy requirements?"
This reveals whether they think about AI in business terms or technical terms. The right answer involves a conversation: What are the downstream effects of false positives vs. false negatives in your specific use case? What accuracy threshold is acceptable for the business? How will exceptions be handled?
Wrong answer: "LLMs are very accurate" or "we'll achieve X% accuracy" without the business context discussion.
"Show me something you built."
Live demo or GitHub repo, not slides. If they can't show you working code or a running system, that tells you something important about what they actually deliver.
Red Flags
Pure strategy with no implementation. Strategy is valuable, but a consultant who only delivers strategy documents and never writes code or builds systems can't validate that their strategy is technically feasible. Find someone who can both strategize and implement.
Vendor-specific lock-in. A consultant who recommends the same platform for every client ("you need Azure OpenAI" or "everything should go through AWS Bedrock") is optimizing for partnership commissions, not client outcomes. The right tool depends on your existing infrastructure, team capabilities, and requirements.
Overpromising on accuracy. "Our solution will automate 90% of your workflow" before they've analyzed your data and process is a pitch, not a diagnosis. Real numbers come from understanding your specific data, your specific process, and your specific constraints.
No monitoring plan. If they're not talking about how you'll know if the system stops working well, they're not thinking about production. AI systems require ongoing oversight. If the engagement ends at deployment, you're inheriting something that will quietly degrade until someone notices a problem.
What Good Looks Like
A good AI automation consultant will:
- Ask more questions than you expect before proposing anything - Tell you which of your workflows are good candidates for automation and which aren't - Give you a realistic assessment of what accuracy is achievable and what the cost of errors is - Build something functional you can test on real data before you commit to full deployment - Provide monitoring and observability from day one, not as an afterthought - Document the system so your team can maintain and evolve it after the engagement ends
The market will sort itself out as AI matures and real results become distinguishable from slides. Until then, asking the right questions early is the best defense against hiring someone who'll burn your budget on something that doesn't work.