I see bold AI claims every week: "revolutionary model," "human-level understanding," "10x efficiency." As an editor who regularly vets tech stories for readers, I’ve learned that the right questions — and a few quick experiments — separate marketing from meaningful innovation. If you’re an investor or reporter, you don’t need a PhD to poke at a startup’s claims. You need a replicable, pragmatic checklist that reveals whether the product is real, robust, and honestly benchmarked.
Why five quick experiments?
Startups live and die by perception. A polished demo can mask brittle engineering or cherry-picked data. I designed these five experiments to be fast, low-cost, and informative. Run them in a single afternoon with free tools and a few test prompts or datasets. Together they give a rounded view of performance, generalisation, operational readiness, and honesty in reporting.
Experiment 1 — Reproduceable sample test
The simplest lie is the selective demo. Ask the startup for the exact inputs they used in their demo and try to reproduce the outputs. If they refuse or provide vague details, that’s a red flag.
What I look for: consistent outputs across runs, clear versioning, and documented randomness (seed values). If responses change drastically for tiny prompt edits, the system may be overfit or brittle — not “revolutionary.”
Experiment 2 — Adversarial probes
Good systems resist simple adversarial probes. I use this to expose hallucinations, prompt sensitivity, or hidden heuristics.
What I look for: graceful degradation, consistent confidence indicators, and clear failure modes. If the model confidently invents facts or wildly flips answers with tiny perturbations, that’s a showstopper.
Experiment 3 — Latency and throughput under load
“Real-time” and “scalable” are common claims. Measure them.
What I look for: predictable latencies, clear limits, and consistent output quality under moderate load. If the system slows to unusability or returns errors at low concurrency, the deployment story is weak.
Experiment 4 — Ground-truth benchmarking
Compare the startup’s outputs against established baselines. This is where you decide whether the "revolution" is meaningful.
What I look for: statistically significant improvement on honest, reproducible benchmarks. A minor lift on cherry-picked data isn’t a revolution — it’s a claim that needs qualification.
Experiment 5 — Security, privacy, and data provenance check
Many AI startups process sensitive data. Verify they actually protect it.
What I look for: transparent policies, verifiable deletion, and honest disclosure of training sources. Vague answers often mean exposure to legal or ethical risks.
How to interpret results
The raw outputs matter less than patterns. Here’s a simple heuristic I use to summarize findings:
| Pass | Reproducible, robust to small changes, scales moderately, improves over baseline, transparent about data and security. |
| Mixed | Works well in narrow conditions, occasional brittleness, questionable scaling or data provenance. Promises need qualification. |
| Fail | Non-reproducible demos, frequent hallucinations, poor performance under load, secrecy about training data or privacy. |
When I see a "Mixed" outcome, my follow-up questions target engineering roadmaps, test coverage, and timelines for resolving weak spots. For "Fail," I flag the claim as overhyped unless the startup opens its repo or provides independent verification.
Red flags I always call out
Conversely, I’m impressed when teams publish evaluation scripts, open-source parts of the pipeline, or invite independent audits. These are signs the startup is serious about trust, not just headlines.
Tools and quick resources
Here are the free tools I use when I’m short on time:
Ask the startup for sandbox credentials or sample datasets; if they claim enterprise readiness but won’t provide test access, that alone is telling.
If you want, I can turn these steps into a one-page checklist or walk you through a live test on a startup you’re evaluating. I run these experiments regularly and have templates for reproducibility reports that make it easy to share findings with colleagues or publish a transparent evaluation.
