How to Run a Vendor Pilot That Actually Proves Something
Published by CodeNicely, the product studio founded by Meghal Agrawal and Ujjwal Agrawal in 2017For: A CIO or VP-IT at a 500–2000-person Indian enterprise who has signed a ₹5–15 lakh paid pilot with a dev partner and needs to know, before the pilot ends, whether what they are seeing is real capability or a prepared demo — because the ₹1–3 crore build decision comes next
In this guide
- Step 1: Write the pilot's kill criteria before you sign the SoW
- Step 2: Scope paid discovery separately from the pilot — and pay for it
- Step 3: Force the pilot onto real data, one real integration, and one real exception
- Step 4: Run the InfoSec and DPDP review during the pilot, not after
- Step 5: Instrument the pilot so you can see the work, not just the output
- Step 6: Run the build-decision review against the kill criteria from Step 1
- How CodeNicely can help
- The short version
- Frequently Asked Questions
A pilot proves something only if it runs against at least one of your real data sources, one of your real integration endpoints, and one of your real exception cases. Everything else — vendor-supplied sample data, sandbox environments, curated happy paths — tests the vendor's demo script, not the vendor. If you are about to spend ₹5–15 lakh on a paid pilot with a crore-scale build decision waiting on the other side, the six steps below are how you make sure the signal you get actually transfers.
The stakes are not theoretical. MIT NANDA's 2025 research found that 95% of enterprise GenAI pilots fail to achieve measurable P&L impact, and the killers were not model quality — they were security questionnaires, data residency, and procurement complexity surfacing late. S&P Global's 2025 Voice of the Enterprise survey found 42% of companies abandoned the majority of their AI initiatives before production, up from 17% a year earlier. Your pilot's job is to find out, cheaply, whether you are about to join that cohort.
Step 1: Write the pilot's kill criteria before you sign the SoW
The single most useful document in a pilot is the one that says what would make you walk away. Write it before the vendor sees it. Three or four binary conditions — not aspirations, not KPIs, but conditions that if unmet mean no build.
Good kill criteria look like: "The pilot must ingest at least 10,000 rows from our production Oracle instance, not a sample export." "The system must correctly handle the three exception cases attached as Annexure B, which were pulled from last quarter's actual tickets." "The vendor must pass our InfoSec questionnaire during the pilot, not after." Bad kill criteria look like: "The solution should demonstrate business value" or "Accuracy should be acceptable."
The anti-pattern: Letting the vendor draft success criteria in the SoW. They will anchor on metrics their demo already hits. You end the pilot with a green dashboard and no decision clarity.
You'll know this step is done when you can hand the kill-criteria sheet to your CFO and she can tell, from reading it alone, what "pilot failed" would look like.
Step 2: Scope paid discovery separately from the pilot — and pay for it
Paid discovery and a paid pilot are two different purchases that enterprises in India routinely collapse into one line item, to their regret. Discovery is where the vendor learns your data model, your integration surface, and your compliance posture. Pilot is where they build something small against that learning. If you buy them as one, the vendor front-loads the fun work and back-loads the discovery, and you find out in week six that the integration you assumed was straightforward needs a middleware layer nobody scoped.
A clean paid discovery produces four artifacts: a data inventory with field-level sensitivity classification, an integration map with authentication method and rate limits per endpoint, a DPDP Rules 2025 applicability note (the Rules were notified on 14 November 2025 and carry hard obligations around breach notification, cross-border transfer, and — if you are designated a Significant Data Fiduciary — annual DPIAs and an India-resident DPO), and a written risk register. If discovery does not produce these, you paid for a sales workshop.
The anti-pattern: "Free discovery" bundled into the pilot. Free discovery is a sales motion. You get a slide deck, not a risk register.
What drives the discovery investment: number of source systems, how well-documented your data is, and whether a legal review of cross-border data flow is required. What drives the pilot investment that follows: number of integrations in scope, whether a real production data pull is permitted, and how many exception cases the vendor must handle end-to-end. These are the specifics that turn a range into a quote — none of them can be priced from a deck.
You'll know this step is done when discovery outputs are signed off by your data, security, and legal teams — not just IT — and the pilot SoW references them by document ID.
Step 3: Force the pilot onto real data, one real integration, and one real exception
This is the step that separates pilots from theatre. A vendor-supplied sample dataset tells you the vendor can process data they chose. A sandbox integration tells you they can hit an endpoint that was built to be hit. A happy-path scenario tells you they can handle the case they rehearsed.

Non-negotiable minimums for a pilot that proves something:
- One real data source. Masked if you must, but drawn from production volume, distribution, and messiness. If legal blocks production access entirely, use a recent snapshot with PII tokenised in-house before handover — not synthetic data the vendor generated.
- One real integration endpoint. Not a Postman collection the vendor wrote against your API docs. An actual call against your actual SAP, Salesforce, Tally, core banking, or HIS instance, with your actual auth, through your actual network path.
- One real exception case. Pull a ticket from last quarter's incident log where the current system failed or a human had to intervene. Give it to the vendor cold. Watch what they do.
The reason this matters: IDC research with Lenovo found that for every 33 AI PoCs an enterprise launches, only 4 reach production — a 12% rate. RAND's 2024 study of 65 practitioners found more than 80% of AI projects fail to reach meaningful production, twice the rate of non-AI IT projects. The delta between a demo that works and a system that works in production is almost always one of these three — data, integration, or edge cases.
The anti-pattern: Accepting a pilot architecture diagram that shows your systems as grey boxes labelled "source system" with a dotted line to the vendor's environment. Dotted lines are where pilots die.
You'll know this step is done when the vendor has made a successful authenticated call against your real system and processed at least one record that exists in your production database.
Step 4: Run the InfoSec and DPDP review during the pilot, not after
The reason MIT NANDA's research names security questionnaires and data residency as top pilot killers is that enterprises routinely treat them as post-pilot activities. The vendor ships a working pilot in week eight. InfoSec review starts in week nine. By week fourteen, the vendor has failed three controls, the fix requires re-architecture, and the business sponsor has moved on.
Run the InfoSec questionnaire, the DPDP applicability review, and the data-processing-agreement redline in parallel with the pilot build. If you are a regulated entity — BFSI, healthcare, listed — add your internal audit team to the pilot distribution list from day one. If your organisation is likely to be notified as a Significant Data Fiduciary under the DPDP Rules 2025, the pilot must demonstrate the vendor can support annual DPIAs, algorithmic audits, and India-resident DPO coordination. None of that can be retrofitted; it either exists in their operating model or it does not.
Note that this post is a buyer's process guide, not legal advice. Your in-house counsel or external privacy lawyer should review DPDP applicability against your specific data flows and designation status.
The anti-pattern: "We'll handle compliance in the build phase." Compliance in the build phase means compliance in the budget overrun.
You'll know this step is done when your CISO has signed off on the pilot architecture and your DPO (or equivalent) has signed off on the data-processing arrangement — both before pilot acceptance.
Step 5: Instrument the pilot so you can see the work, not just the output
A dashboard that says "96% accuracy" tells you nothing about whether the vendor can build. You need visibility into how they work: how they estimated, how they responded when the real data broke their first assumption, how quickly they turned around a change request, how clearly their engineers wrote up the exception case.

Set up the pilot so you get:
- Weekly written updates — not calls — that compare actuals to the week's plan. Calls let vendors spin. Writing exposes clarity of thought.
- Read access to their issue tracker and code repository for the pilot. Not a report. Direct access.
- A mid-pilot checkpoint where the vendor demonstrates, live, their handling of a new exception case you hand them in the meeting. Not one from the SoW. A new one.
- A written post-mortem on any missed commitment, however small. You are evaluating the muscle, not the mile.
The anti-pattern: Judging the pilot on the final demo. The final demo is the most rehearsed artifact of the engagement. The week-four standup is a hundred times more diagnostic.
You'll know this step is done when you can describe, in a paragraph, how the vendor actually behaves under pressure — not what their output looks like on the happy path.
Step 6: Run the build-decision review against the kill criteria from Step 1
Close the loop. Put the Step 1 kill-criteria sheet on the table and walk through each condition. Not "did the vendor try hard," not "did the demo land" — did the binary conditions resolve yes or no. If three of four resolved yes and one resolved no, you have a decision to make about whether the one that failed is a deal-breaker. Make it explicitly, in writing, with the CFO and business sponsor in the room.
Then consider team composition for the build. MIT's research found that builds blending internal specialists with external vendor expertise reached a 67% success rate, versus 22% for IT-only internal builds. The pilot should have given you enough signal to know which of your people need to be embedded in the build team and which parts you are better off owning fully.
The anti-pattern: Letting momentum decide. The pilot ran, the team is warm, the vendor is ready to start Monday — so you start Monday. This is how 30% of GenAI projects end up abandoned after PoC, which is exactly what Gartner predicted would happen through end-2025.
You'll know this step is done when you have a one-page build-decision memo signed by IT, business, finance, and security — or a one-page kill memo that does the same.
How CodeNicely can help
We have been on the vendor side of enough of these to know what a well-run pilot feels like from both chairs. The engagement that maps most closely to the CIO scenario above is GimBooks — a Y Combinator-backed fintech where the discovery produced a real integration map and a data model before a line of production code was written, which is why the build phase did not stall on surprises. For enterprises with regulated-data constraints similar to BFSI or healthcare, HealthPotli (e-pharmacy with drug-interaction AI) is a closer analog — the pilot had to prove the model worked against real prescription data under real data-handling rules, not synthetic test sets. If your pilot is in the AI/ML space specifically, our AI Studio page outlines how we scope discovery and pilots as separate commercial phases for exactly the reasons in Step 2. We work with full IP ownership and no vendor lock-in, which matters more at build-decision time than at pilot-signing time.
The short version
A pilot that runs on vendor-supplied data, in a vendor-supplied sandbox, against a vendor-supplied scenario, proves the vendor can control variables. A pilot that runs on your data, through your integration, against your exception case, proves the vendor can handle yours. Write the kill criteria first. Pay for discovery separately. Force the real data, real integration, real exception. Run security and DPDP in parallel. Instrument the process, not just the output. Close against the criteria, in writing. Then decide.
Frequently Asked Questions
How long should a paid vendor pilot run for an Indian enterprise?
The length is driven by three things: how many real integrations the pilot must touch, whether InfoSec and DPDP review run in parallel or sequentially, and how many exception cases the vendor must handle end-to-end. Most well-scoped pilots land somewhere between six and twelve weeks; shorter than that and discovery is being skipped, longer and you are usually rebuilding mid-pilot. The specifics that turn this range into a real timeline are your data access approval process, your integration complexity, and your audit team's availability — those are the conversations to have with a prospective partner before signing.
What is the difference between paid discovery and a paid pilot?
Paid discovery is a learning exercise that produces artifacts: a data inventory, integration map, compliance applicability note, and risk register. A paid pilot is a small build against those artifacts that tests whether the vendor can execute. Collapsing them into one line item is the single most common reason pilots overrun — the vendor front-loads the visible work and back-loads the unknowns.
Can we use synthetic or masked data for the pilot instead of production data?
Masked production data drawn from real volume and distribution is acceptable and often required under DPDP for PII-heavy use cases. Fully synthetic data generated by the vendor is not — it tests nothing that transfers to production. If legal blocks production access entirely, tokenise PII in-house before handover rather than letting the vendor work from a fabricated dataset.
How do we evaluate a vendor's pilot work if we don't have in-house AI or engineering depth?
Two things help. First, insist on read access to the pilot repository and issue tracker — a competent independent reviewer can assess code quality and engineering rigour in a day, and that is cheaper than the wrong build decision. Second, run a live exception-handling demo mid-pilot where you hand over a new case in the meeting; this is diagnostic even without deep technical staff on your side, because you are observing process maturity and clarity of thought, not code.
What should happen if the pilot partially fails some kill criteria?
Make the decision explicit and in writing. A failed criterion is not automatically a deal-breaker, but it must be named, assessed for whether it blocks production value, and signed off by business, finance, and security — not quietly absorbed into the build SoW. The muscle this builds inside your organisation matters more than the specific pilot outcome; it is what prevents the next pilot from being theatre.
Sources & further reading
- Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025 (July 2024)
- RAND Corporation: The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (2024)
- IDC & Lenovo: Enterprise AI Pilot-to-Production Conversion Rate (via SoftwareSeni analysis)
- S&P Global Market Intelligence: 2025 Voice of the Enterprise Survey on AI Initiative Abandonment
- Gartner Forecasts India IT Spending to Reach $160 Billion in 2025 (official press release, November 2024)
- India DPDP Rules 2025: Complete Compliance Guide (Seclore)
- Digital Personal Data Protection Rules, 2025 — Wikipedia
- MIT NANDA / The GenAI Divide: State of AI in Business 2025 (via Substack analysis)
Get one practical guide a week
Costs, AI tools, partner selection — written for people who make the decision. No spam, unsubscribe anytime.
Thanks — you're on the list. One practical guide a week, nothing else.
That didn't work — please check the email address and try again.
Building something in Software & Technology?
CodeNicely partners with founders and tech teams to ship AI-native products that move metrics. Tell us about the problem you're solving.
Talk to our team Book a 30-min call_1751731246795-BygAaJJK.png)