How to Run a Vendor Pilot That Actually Tests Something

Most pilots fail to tell you anything. The vendor builds a login screen and a dashboard, everyone nods, and three months later the real project reveals problems the pilot never touched. That's not a pilot, it's a sales demo with a longer runway and a bigger invoice.

A pilot only works if it forces the vendor to do the hard part of your actual project, not the easy part of any project. Every team can build a CRUD screen. The question you're paying to answer is whether this team can handle the thing that makes your product hard: the offline sync, the multi-tenant data model, the vision pipeline, the billing edge case that breaks on a mid-cycle plan change. If the pilot doesn't touch that, you've learned nothing about the risk that actually threatens your timeline and your budget.

We've been on both sides of this. We've won pilots by picking the hardest slice of a client's roadmap and shipping it in two weeks, and we've watched other vendors win pilots by picking the easiest slice and shipping it beautifully. Here's how to make sure you're testing the first kind of team, not rewarding the second.

Pick the slice that scares you, not the slice that's easy to demo

Before you talk to anyone, write down the three parts of your product you're least confident anyone can build well. For a retail platform that might be offline POS sync across flaky store wifi. For a construction platform it might be extracting quantities from a hand-marked PDF plan set. For a SaaS product it might be the tenant data isolation model that has to survive a security audit later.

Pick one of those. Not the login page, not the marketing site, not a settings panel. The pilot scope should be small enough to finish in two to four weeks and specific enough that a mediocre team visibly struggles with it. If every vendor you talk to can nail the pilot easily, you scoped it wrong.

This is also where a lot of founders get the estimate wrong going in — if you're not sure your pilot scope is realistic for two weeks, it's worth reading about signs your MVP quote is too low before you write the brief, because the same underscoping problem shows up in pilots and in full builds.

Pay for it, and say so in writing

Free pilots select for vendors who are desperate for work, not vendors who are good. A paid pilot, even a small one, changes the incentive on both sides. You're buying a fixed, bounded piece of work, not asking for a favor, and that means you can hold the vendor to a real deadline and a real deliverable without the relationship feeling like a negotiation.

Structure it as a fixed-price, fixed-scope engagement with a written statement of what's in and out. If you want a longer discussion of how fixed price versus time-and-materials protects you differently at each stage, that's a separate decision worth understanding before you sign anything at all — see what to require in a development contract before you sign for the baseline paperwork you should ask for regardless of pilot outcome.

Give them a real handoff, not a wishlist

A pilot also tests how a vendor works, not just what they build. Give them a proper brief: existing schema if there is one, sample data, the constraint that's actually biting you, and a clear definition of done. Then watch what they ask you before they start.

A team worth hiring asks about edge cases, data volume, and who owns the environment they're building in. A team that starts writing code the same day you send the brief either already knows exactly what you need, which is rare, or isn't thinking about what they don't know, which is the more common explanation.

A decision rule for the kickoff call

If a vendor doesn't ask at least three clarifying questions that reveal they've thought about failure modes — what happens with bad input, what happens at scale, what happens when the network drops — don't move forward. This isn't a taste preference. Teams that skip this step on a two-week pilot skip it on a six-month build too, and you find out at the worst possible time.

Score the pilot against a fixed rubric, not a feeling

Decide the scoring criteria before the pilot starts, in writing, so you're not rationalizing a decision you already made based on who you liked more on the call. A rubric that works across most technical pilots:

• Correctness under a bad input. Feed it malformed, missing, or extreme data on purpose. Does it fail loudly or silently?
• Code you can actually read. Have someone other than the vendor review the pull request. Are there tests? Comments where the logic isn't obvious? Or is it a pile of code that only makes sense to the person who wrote it?
• Time to first useful commit. Not the final delivery — how fast did something real land, versus how long they spent on setup and ceremony.
• Communication under friction. Change one requirement mid-pilot on purpose. Watch how they handle scope creep: do they push back with a reasoned tradeoff, or just say yes and let quality slip?
• What they'd do differently. Ask them to critique their own pilot output at the end. A team with real judgment will name a shortcut they took and why.

Score each criterion 1 to 5 before you talk pricing for the full engagement. If the vendor scores well but the price feels high, that's a negotiation. If the vendor scores poorly and the price feels low, that's not a bargain, that's a preview of the change orders coming later.

Test the boring parts too: ownership, environments, handoff

Technical quality is half the evaluation. The other half is whether you'll actually own what gets built. During the pilot, confirm you get the repository, the infrastructure credentials, and the deployment pipeline — not a black box the vendor controls and you rent access to. Ask where the pilot code lives and who has admin rights on day one, not day ninety.

This matters more than it seems like it should. We've inherited platforms from teams that operated 700+ tenant databases across prior engagements, and the projects that were painful to take over were the ones where the outgoing vendor had never handed over real ownership of infrastructure, just a working demo. A pilot is your cheapest chance to find out whether a vendor hands things over cleanly, because if they won't do it for a two-week engagement, they won't do it for a two-year one.

If you're choosing between a solo contractor, an in-house hire, and a studio, the pilot structure differs slightly depending on which you're testing, and it's worth reading CTO, contractor, or studio before you finalize who gets invited to pilot at all — the evaluation criteria for a single contractor's pilot output looks different from a studio's, because a studio should also be tested on how it staffs and hands off work internally.

Where a pilot doesn't apply

A pilot is the wrong tool if your actual bottleneck is speed to market rather than vendor risk. If you already trust the vendor — a referral from someone whose judgment you've verified, or a studio whose portfolio shows the exact problem you're solving already shipped — running a formal pilot just burns two to four weeks you don't have. In that case, spend the time instead on a tight day-one packet that gets the real project moving faster, and put your diligence into the contract terms instead.

A pilot also doesn't work well for pure discovery-phase decisions, like whether to build a feature at all. If you haven't settled build versus buy yet, you're not ready to pilot a vendor — you're still deciding what you're building, and no amount of vendor testing answers that question.

A pilot only proves something if it forces the vendor to fail somewhere small, where you can afford it, instead of somewhere large, where you can't.

What to do with the result

If the vendor scores well, move to the full contract, but reuse the pilot's rubric as a template for milestone reviews later. If it scores poorly, don't negotiate the price down and proceed anyway — a cheaper version of a team that already showed you it struggles with your hardest problem is still that team. Run a second pilot with a different vendor before you commit real budget. Two weeks and a few thousand dollars is a rounding error against the cost of discovering the same problem six months into a fixed-price contract you can't easily exit.