AI Strategy
Why Most AI Agents Fail Before They Launch
I’ve built a lot of AI agents. And I’ve watched a lot of them fail — not after launch, but before it. Not because the technology wasn’t good enough. Because the organization wasn’t ready for it.
That’s a hard thing to say when everyone is excited about AI and the demos look incredible. But the gap between a demo and a deployed agent that actually works is where most projects quietly die. I want to walk you through what I’ve learned — not to scare you off, but to help you get it right.
First, Be Honest About Your Data
Every AI agent we build is powered by an off-the-shelf large language model. That model is smart, but it knows nothing about your business. It doesn’t know your products, your pricing, your policies, or your tone. That all has to come from you.
There is no data magician.
The data phase is the hardest part of any agent project. I’m not just talking about having information somewhere — in a shared drive, buried in PDFs, scattered across a decade of email threads. I’m talking about information that is clean, current, structured, and retrievable. That means thinking through three things:
If the answer to any of those isn’t quite there yet, that’s not a disqualifier — but it is its own project. We’ve worked with clients whose data needed significant cleanup and restructuring before the agent work could even begin. That’s solvable. It just needs to be scoped honestly from the start, because underestimating it is one of the most common reasons timelines slip and expectations don’t get met.
Define the Goal Like You Mean It
The second place I see projects go sideways is in goal-setting. People see what AI can do, and their brain goes straight to Jarvis from Iron Man. They want it to do everything.
It can’t. And more importantly, it shouldn’t.
The agents that perform best have ruthlessly specific mandates. Not “answer questions about our business,” but something like this:
“This agent surfaces product and event information from the website, captures lead contact details when a visitor is ready to talk, and escalates to a human when it cannot answer. That’s it. That’s the job.”
Real-World Results
When we built a concierge agent for a leading national association serving over 600 material handling companies, we made it do one specific job extremely well. Across more than 400 analyzed interactions:
That kind of performance comes from clarity, not ambition.
Build It Right, Then Test It Hard
Once the data is solid and the goal is defined, the build begins. And this is where patience matters. Synthesizing your data into a form the AI can use reliably takes real work. So does stress-testing it before it ever sees a real user.
My testing process runs in stages:
Stage 1
Internal development testing — I’m breaking it, pushing it, asking it things it shouldn’t know.
Stage 2
Controlled client rollout — watching real conversations in near-real time.
Stage 3
Production — but not without a monitoring plan. Daily in week one, then weekly, then monthly.
The agent doesn’t stop needing attention just because it’s live. That’s a mindset shift most people aren’t prepared for.
Think of It Like Hiring, Not Installing
Here’s the analogy I keep coming back to: deploying an AI agent is closer to hiring an employee than installing software.
Think about how you’d bring on a new team member. You’d evaluate their fit for the role. You’d give them the tools and context they need to succeed. You wouldn’t drop them at a desk on day one and disappear for six months. You’d check in early, give feedback, course-correct when something’s off. And over time, you’d do periodic reviews.
Your AI agent deserves the same framework. It needs to be set up to succeed, supported in the early weeks, and evaluated on a real schedule. The organizations that treat it as a living system — not a one-time deployment — are the ones that see lasting results.
I want to get into the weeds for a minute on something specific — because for distributors and manufacturers, this is usually where the project gets interesting.
Technical Deep Dive
When Standard RAG Isn’t Enough
Most AI agents retrieve information through a method called RAG — Retrieval-Augmented Generation. The short version: when you ask the agent something, it searches a knowledge base, pulls in the most relevant chunks of content, and uses those to form its answer. For general questions, it works well. For complex material handling product catalogs, it falls apart in a specific and frustrating way.
Think about how you shop for equipment. You don’t read one product page and decide — you narrow it down to a handful of options and look at them side by side. That comparison view is what helps you make a good call. Standard RAG doesn’t do that. It pulls whatever chunks score highest and misses others entirely. The agent ends up working with an incomplete picture — and it has no idea.
Here’s a real example. We built an agent for a material handling distributor with over 93 distinct product categories — drum handlers, dock equipment, conveyors, pallet storage, you name it. Standard RAG would surface a few popular product pages and skip whole categories. A customer asking about lift tables might get a decent answer. A customer asking which solution fits their load weight and floor space? The agent was guessing.
The fix was to build a dedicated structured tool that, when triggered by a product-selection question, returned an entire category as one complete payload. Not a chunk — the whole set. Every model, every spec, all in context at once. A human might compare 5 items side by side. AI can read 20 or 30 simultaneously and reason across all of them. We built for that.
What this means in practice
This is what I call agentic RAG: giving the agent the judgment to know which retrieval tool to reach for and when. For distributors and manufacturers with large, multi-category catalogs, it’s often the difference between an agent that genuinely helps a buyer and one that confidently guesses.
The catalog work is honestly one of my favorite parts. When you sit down and force yourself to organize content for an AI, you end up having to answer questions about your own business that nobody’s asked in years. How should these products actually be grouped? What does a buyer need to know to make a call? We’ve had clients come out of that process making changes to their actual website — navigation, category structure, how products are described. The AI work exposed it, but the fix was just good business thinking.
Complex Catalogs Are Solvable — But Don’t Underestimate the Work
If you’re a distributor or manufacturer with a large, multi-category product catalog, an AI agent can absolutely work for you. We’ve done it. But it’s not a plug-and-play project, and anyone who tells you otherwise is setting you up for disappointment.
The data architecture alone — structuring 93 product categories in a way that an agent can reason across reliably — is a substantial undertaking before a single line of the agent itself is written. It requires the right discovery process, the right technical approach, and an honest conversation upfront about scope.
That’s exactly the kind of conversation I want to have with you. Not a sales pitch — a real assessment of where you are, what your data looks like, and what it would actually take to build something that performs.
Ready to Talk?
Not a sales pitch. A real assessment.
Where you are, what your data looks like, and what it would actually take to build something that performs.