What to Ask a Reference After the Pilot Is Over

For: A Series A startup founder in India who has shortlisted one dev partner, received two glowing references the vendor provided, and now suspects those calls were too smooth to be real due diligence

If your two vendor-supplied reference calls felt too smooth, that is because they were. A vendor-picked reference will almost never volunteer a milestone dispute, a change-order fight, or a post-launch support gap. The only reference call worth taking before you commit ₹50–150 lakh is one you sourced yourself — a LinkedIn search on the partner's past clients, a cold DM to the CTO who moved on two years ago, a founder whose product now runs on a different stack. This post gives you the 16 questions to ask that person, once you find them.

Context on why this matters more than the deck the vendor sent: McKinsey and Oxford's study of 5,400+ IT projects found large IT builds run 45% over budget and 7% over time while delivering 56% less value than predicted, and 17% go so badly they threaten the existence of the company. Dun & Bradstreet's Barometer of Global Outsourcing puts the failure rate at 20–25% within two years and 50% within five. Yet Mitratech's 2025 TPRM study finds organizations actively assess only about 40% of their vendor population, and most third-party risk programs skip commercial due diligence entirely — the exact step that includes real-world reference checks, not the ones the vendor curated.

First: find the references the vendor did not give you

Before the questions, the sourcing. Spend 30 minutes on this and you will get more signal than from any deck review.

• LinkedIn search: "worked with [Vendor Name]", then filter by past companies. Ignore the current employees; you want the founders and CTOs who moved on.
• GitHub commits and public repos: if the vendor has an org account, look at who they collaborated with. Old commit history names old clients.
• Wellfound / AngelList / YC directory: cross-reference the vendor's claimed logos with founder LinkedIns. If the vendor claims a client but nobody at that client will admit to it, that is data.
• The dead product graveyard: ask for a list of the last 10 projects, not the top 3. Which ones are still live? Which URLs 404?

Aim for three self-sourced references: one still-happy client, one former client who moved on, and — if you can find them — one who ended the relationship early. The third is the most valuable call you will make.

The 16 questions, grouped by what they expose

Group 1: Scope and the change-order fight (Q1–Q4)

This is where most Series A founders get burned. The pilot was fixed-scope. The main build is where every clarification becomes a change order.

1. "Walk me through the first time scope changed after the SOW was signed. What happened?"

Why it matters: Every project has scope change. You want to hear the mechanics, not a denial.

Good answer: "We wanted to add SSO in month two. They wrote a one-page change note, gave us a delta estimate in three days, we approved it, they built it. No drama."

Red flag: "There were no changes" (impossible), or "It got tense — every clarification became a paid extra."

2. "What percentage of the final invoice was change orders on top of the original SOW?"

Why it matters: Anchors the scope-creep number. Reference the McKinsey 45% overrun baseline in your own head.

Good answer: Anything under 20%, with a clear reason ("we added a whole new module").

Red flag: Above 40%, or "I don't remember" from a founder — founders remember.

3. "Who wrote the SOW — you or them? And did you have a technical person review it before signing?"

Why it matters: Vendor-written SOWs with vague acceptance criteria are how ₹80 lakh becomes ₹1.4 crore.

Good answer: "They drafted, our fractional CTO redlined it twice, acceptance criteria were measurable."

Red flag: "They wrote it, we signed, we assumed we understood it."

4. "Was there ever a milestone you refused to sign off on? What did they do?"

Why it matters: How a vendor handles a rejected milestone tells you everything about the payment-vs-quality dynamic.

Good answer: "Yes — twice. They fixed it inside the milestone budget, no argument."

Red flag: "They pushed back hard and said the acceptance criteria were subjective," or the reference sounds nervous answering.

Group 2: Team stability and the bait-and-switch (Q5–Q8)

India's IT sector runs at 15.4% average attrition, with some suppliers hitting 35%. You are not just hiring a vendor; you are hoping their team stays.

5. "Who did the sales pitch, and who actually wrote the code? Were they the same people?"

Good answer: "The tech lead on the pitch call was on our daily standup for the whole project."

Red flag: "We met impressive senior people, then juniors showed up on Slack."

6. "How many devs rotated off your project mid-build, and how was the handover?"

Good answer: Zero to one rotation on a 6-month project, with a documented handover.

Red flag: "Three or four — we kept re-explaining our domain."

7. "When your project ended, did anyone from the team follow up, or did they vanish?"

Why it matters: Post-launch behavior is the single hardest thing to fake in a pilot.

Good answer: "The engineering lead still WhatsApps me when we hit a year milestone."

Red flag: "Radio silence the day after the final invoice cleared."

8. "If you called the CEO today with an urgent bug, would they respond within a day?"

Good answer: "Yes, and I have tested it."

Red flag: Nervous laugh, or "I would probably go through my account manager."

Group 3: The handoff, IP, and lock-in (Q9–Q12)

9. "Do you own the source code, the repos, the cloud accounts, the domain, and the CI/CD pipelines outright?"

Good answer: "Everything is in our GitHub org, our AWS account, our name on the domain. They had access; they never owned it."

Red flag: "The repo is on their GitHub, we get a zip on request," or "the AWS bill comes through them."

10. "How long did it take a new dev team to become productive on the codebase after handover?"

Good answer: Under three weeks with the documentation the vendor left.

Red flag: "We had to rewrite chunks because nobody could understand the architecture decisions."

11. "What was the state of the documentation, tests, and README on day one of your in-house team taking over?"

Good answer: Architecture diagram, env setup runbook, above 60% test coverage, deployment steps documented.

Red flag: "Documentation was on a call with their lead — nothing written."

12. "Did you feel any pressure — even subtle — to keep them on a retainer to maintain the code?"

Why it matters: The soft lock-in is where margins hide. A partner confident in their handoff does not need to trap you.

Good answer: "They offered a retainer, we said no, they helped us hire in-house instead."

Red flag: "They told us the code would break without their team, so we kept paying."

Group 4: The uncomfortable questions (Q13–Q16)

These are the ones that separate a real reference call from a marketing exercise. Ask them slowly. Let silences happen.

13. "What is the one thing you wish you had asked before signing with them?"

Good answer: Something specific — "I wish I had asked about their on-call rotation before we hit our first Sunday-night outage."

Red flag: "Nothing, it was all perfect." No one says this about a real engagement.

14. "If you were starting over with the same budget, would you hire them again — and would you change the contract structure?"

Good answer: "Yes, but I would push for milestone-based payments instead of monthly retainer."

Red flag: Long pause, then "Probably. I don't know. Maybe."

15. "Who else did you evaluate, and why did you pick them over the alternatives?"

Why it matters: Confirms this is a real evaluation, not a friend-of-founder relationship. Also gives you two other vendors to benchmark.

Good answer: Names two or three specific competitors with clear reasoning.

Red flag: "They were the only ones we talked to," or "a friend introduced us."

16. "Is there anyone else — a former colleague, a founder they built for two years ago — you would suggest I talk to?"

Why it matters: Snowball sampling. Every reference you get from a self-sourced reference is worth two from the vendor.

Good answer: A specific name and an offer to intro.

Red flag: "I would just go with what the vendor gives you."

A worked example of what these calls typically surface

Illustrative only — not a benchmark to hold anyone to. On a two-integration MVP with no legacy data to migrate, sourcing three self-picked references usually takes about a week: two hours on LinkedIn to build the list, ten to fifteen cold DMs to get three founders on a 30-minute call each, and a couple of hours to write up the pattern. In one common shape, two of the three references corroborate the vendor's story on team stability and code quality, and one surfaces a scope-change dispute the vendor never mentioned. That third call is why you do this. If all three tell you the same clean story, that itself is a signal — either the vendor is genuinely consistent, or you have not looked hard enough for the unhappy client.

How CodeNicely can help

If you are a Series A founder about to sign with a dev partner, you are welcome to ask us the same 16 questions — and we will hand you numbers of founders we did not pick. On GimBooks, a YC-backed accounting SaaS for Indian SMBs, the engagement went from MVP through multi-year scale on the founding team's own GitHub org, their own cloud accounts, with documentation their in-house team took over cleanly. That is the shape of engagement worth referencing — full IP with the client, no soft lock-in on infra, and a codebase a new team could ramp on. If you are evaluating Indian dev partners for an AI or fintech build, ask for the founder's WhatsApp of a project we finished 18 months ago. Those are the calls that matter.

The bottom line

The reference calls the vendor set up are a formality. The reference calls you source yourself are the due diligence. Spend one week on this before you sign. If a vendor resists you talking to unlisted past clients, or cannot name any, you already have your answer — and given that 25–50% of outsourced IT projects fail or underperform and 76% of companies face supplier management issues, that answer is worth the week.

Frequently Asked Questions

How many references should I check before signing with a development partner in India?

At least three, and at least two of them should be sourced by you, not supplied by the vendor. One should be a current or recently completed client, one a former client from 12–24 months ago, and — if you can find them — one who ended the engagement early. The last is the most informative call you will make.

What if the vendor refuses to let me speak to clients they didn't hand-pick?

Take it as a signal, not a compromise. A confident partner has nothing to hide from a founder-to-founder call. If you cannot get past their curated list, use LinkedIn to find their past clients directly and reach out cold — most Series A founders will take a 20-minute call from a peer doing real diligence.

How do I check references for an offshore Indian dev partner if I'm based abroad?

The mechanics are the same, timezones are the wrinkle. Do the calls on video, not phone, and look for the same three-reference mix (current, former, ended-early). Also ask specifically about async communication cadence and on-call coverage for your timezone — a partner who has done it before will describe their overlap hours precisely; one who has not will hand-wave.

What does a proper reference-check process cost me in time before I sign?

Roughly one working week of a founder's time — a few hours to source names, three to five 30-minute calls, a couple of hours to synthesize. What turns that into a real number for your situation is how many past clients the vendor has (some have 10, some have 200), how public those relationships are, and how many of them are in your industry. Compared to a mid-six-figure build going sideways, it is the highest-ROI week of diligence you will do.

Are vendor case studies useful at all, or should I ignore them?

Useful as a shortlist filter, useless as a decision input. Read them to understand the shape of work the vendor claims — industry, stack, scale — then verify every claim on a call with the founder named or implied in the case study. If the founder is not named, ask the vendor for the intro. If they will not make it, treat the case study as marketing, not evidence.