The AI Vendor Red Flag Checklist

The AI gold rush has created a new species: the AI vendor who doesn't actually do AI. They have a website with the word “intelligence” in three places, a demo that looks impressive, and a pricing page that's suspiciously vague. They are everywhere, and they are very good at separating executives from their money.

I've evaluated dozens of AI tools and vendors for clients. Some are excellent. Some are a wrapper on ChatGPT with a $2,000/month price tag. Here's how to tell the difference.

Red Flag #1: “We Use Proprietary AI”

Translation: we wrote a wrapper around GPT-4 or Claude and put our logo on it.

In 2026, very few companies have truly proprietary AI models. Training a competitive model from scratch costs millions of dollars and requires specialized infrastructure that most startups don't have. According to Epoch AI's database of notable AI models, fewer than 50 organizations worldwide have trained frontier models.

The honest answer sounds like: “We use [specific model] from [specific provider] and we've built custom tooling, integrations, and prompt engineering on top of it for your industry.”

There's nothing wrong with building on top of foundation models. That's what smart companies do. The red flag is pretending you didn't. If they can't tell you exactly which model powers their system, walk away.

Red Flag #2: “Results in 2 Weeks”

You can get a demo in 2 weeks. You can get a proof of concept in 2 weeks. You cannot get a production-grade system that knows your business, connects to your actual data, and operates reliably in 2 weeks. That takes 4-8 weeks minimum for a focused engagement.

BCG's research on AI implementation found that companies rushing AI deployment were 3x more likely to abandon the project within 12 months. The ones that invested in proper integration, training, and supervised operation had 70% higher satisfaction rates.

The honest answer sounds like: “Discovery takes a week. Integration takes 2-3 weeks. Supervised operation takes another 2-3 weeks. You'll see autonomous results in 6-8 weeks.” I walk through this timeline in detail in How It Works.

If someone promises you the moon in two weeks, they're selling you the demo and calling it the product.

Red Flag #3: They Can't Explain Where Your Data Goes

This is the one that should end the conversation immediately. If you ask “where does my data live?” and the answer involves hand-waving about “secure cloud infrastructure” without specifics, that's a problem.

The questions to ask:

• Is my data used to train your model or improve your product for other customers?
• Which cloud provider hosts the data? Which region?
• Who has access to my data within your organization?
• What happens to my data if I cancel?
• Can the system run on my infrastructure instead of yours?

The honest answer sounds like: “Your data is stored in [specific provider, specific region]. It is not used to train our models. Here is our data processing agreement. Here is our SOC 2 report.” Better yet: “The system runs on your infrastructure. We never see your data.” I wrote about why this matters in Is AI Safe for Business? and secrets management for agents.

Red Flag #4: The Demo Uses Their Data, Not Yours

Every AI demo looks amazing when it's running on curated sample data. Of course the dashboard looks perfect when every field is populated, every integration is working, and every edge case has been smoothed out.

The test is what happens when you plug in your actual data. Your messy, inconsistent, has-three-different-date-formats, the-CFO-uses-a-comma-where-there-should-be-a-period data.

Ask for a pilot on your real data. A good vendor will welcome this — it's how they prove value. A bad vendor will find reasons to delay it. “Let's start with the standard demo and we can customize later” is code for “our system breaks on real-world data and we don't want you to find out before you sign.”

Red Flag #5: No Clear ROI Methodology

If they can't tell you exactly how you'll measure whether this is working, they don't know if it works either.

Deloitte's State of AI survey found that the #1 reason companies abandon AI initiatives is “unclear ROI” — not technical failure. The technology worked fine. Nobody defined what success looked like before they started.

The honest answer sounds like: “Here are the 3-5 metrics we'll track. Here's your baseline today. Here's what we expect to see at 30, 60, and 90 days. If we don't hit these numbers, here's what we change.”

I talk about this principle in building dashboards that drive decisions — if you can't measure it, you can't manage it.

Red Flag #6: “AI-Powered” Everything

When every feature is “AI-powered,” none of them are. This is the 2026 equivalent of “cloud-based” in 2015 or “blockchain-enabled” in 2018. It's a marketing checkbox, not a technical description.

Ask: which specific features use AI, and which don't? What model or approach powers each one? What would this feature look like without AI?

A lot of what gets sold as “AI” is actually rules-based automation with a chatbot on top. There's nothing wrong with rules-based automation — it's reliable and predictable. But you shouldn't pay AI prices for a glorified if-then workflow.

Red Flag #7: They Don't Talk About Failure Modes

Every AI system fails sometimes. Every single one. If a vendor tells you their system is 99.9% accurate and leaves it there, they're either lying or they haven't tested it in the real world.

The questions to ask:

• What happens when the AI makes a mistake?
• How does the system handle edge cases it hasn't seen before?
• What guardrails exist for high-stakes decisions?
• How do I override the AI when it's wrong?
• Is there a human-in-the-loop option for critical actions?

The honest answer sounds like: “The system will make mistakes, especially in the first few weeks. Here's how we handle that: supervised operation first, escalation rules for high-stakes decisions, and a correction loop that trains the system from every mistake. Here's what our trust architecture looks like.”

The vendor who tells you their AI never fails is the one you should be most afraid of. They either don't know, or they're hoping you won't find out.

Red Flag #8: Lock-In by Design

Can you leave? If the answer is complicated, that's intentional.

Good vendors build systems that work with your existing infrastructure and make it easy to export your data, your configurations, and your customizations. Bad vendors build walled gardens where everything depends on their proprietary format.

Ask: if I cancel tomorrow, what do I keep? Can I export my data in a standard format? Do the integrations work without your platform? Is my agent logic portable?

The vendor trap applies double for AI. The more your operations depend on a system, the more expensive it is to switch. Make sure switching is possible before you're too deep to leave.

The Honest Vendor Checklist

Here's what the good ones look like:

• They name their foundation models and explain what they built on top
• They give realistic timelines (weeks to months, not days)
• They can answer every data security question in specifics
• They want to pilot on your real data
• They define success metrics before starting
• They talk about failure modes and guardrails unprompted
• They make it easy to leave (because they're confident you won't want to)
• They have references you can actually call
• They charge for outcomes, not seats
• They can explain what they do in plain English, without jargon, in under 60 seconds

If a vendor checks all ten, you probably have a winner. If they check fewer than six, keep looking.

One More Thing

The best defense against bad AI vendors isn't a checklist — it's understanding. The more you understand what AI actually does, how agents actually work, and what's realistic vs. fantasy, the harder you are to fool.

Start with How It Works for the fundamentals. Read What CEOs Get Wrong About AI for the common misconceptions. And if you're evaluating vendors and want a second opinion from someone who doesn't sell SaaS, I'm happy to take a look.