What to Ask a Reference When You're the Enterprise Buyer

Published by CodeNicely, the product studio founded by Meghal Agrawal and Ujjwal Agrawal in 2017

For: A VP of Engineering or CIO at a 500-1,000-person US company who has shortlisted two offshore dev partners for a multi-year custom software engagement and is about to run formal reference calls before the legal team drafts the MSA

Vendor-supplied references are pre-coached on satisfaction, not on the four things that actually determine whether a multi-year offshore engagement blows up: change-order discipline in the first 90 days, IP assignment under the reference's own jurisdiction, SOC 2 audit-trail continuity when the vendor's team rotates, and what happened the first time a named senior engineer left mid-sprint. The 15 questions below are built to extract those four signals in a 30-minute call, with the exact wording of a good answer and the exact wording of a red flag for each. Use them before your legal team starts drafting the MSA, not after.

A quick note on how to run the calls. Ask for three references, not two, and insist that at least one is a client whose engagement ended — not just active accounts. Active accounts have an incentive to be polite because they still need the vendor to ship next quarter. Churned clients have no such incentive, and they are where the real pattern lives. If the vendor cannot produce a single reference whose contract has concluded, that is itself the first data point.

The change-order and pricing-discipline questions

This is the block most buyers skip and the one that correlates most tightly with cost overruns. A McKinsey/Oxford study of 5,400 large IT projects found the average project ran 45% over budget and 17% became 'black swans' with 200–400% overruns. Change-order velocity in the first quarter is the earliest leading indicator you can measure.

1. How many signed change orders did you have against the original SOW in the first 90 days, and what was the dollar value as a percentage of the original contract?

Why it matters: This is the single most useful question in the entire call. A vendor with weak discovery or a bait-and-switch pricing habit will show double-digit change-order counts before the pilot is even reviewed.

Good answer: "Two or three, mostly scope we added ourselves after seeing the first demos. Maybe 8% of contract value." The reference remembers the number because it was small enough to remember.

Red flag: "Oh, I'd have to pull that up" followed by vague hand-waving, or a number above 20% of contract value in the first quarter. Gartner estimates hidden costs add 18–25% to total contract cost in poorly scoped T&M engagements — if it's already at that level in 90 days, you are looking at a 60–80% overrun by year-end.

2. Walk me through the last change order you disputed. What did they want, what did you want, and how did it get resolved?

Why it matters: You are not testing whether disputes happen — they always do. You are testing the escalation path and whether the vendor eats cost when they got discovery wrong.

Good answer: A specific story with named people on both sides, and a resolution where cost was split or absorbed because the vendor conceded their estimate was off.

Red flag: "We've never had a dispute" (nobody has zero disputes over 18 months) or a story where the client paid full freight on something they clearly flagged in the original SOW.

3. Was your contract fixed-bid, time-and-materials, or a hybrid? If you were starting over, would you use the same structure with them?

Why it matters: The honest answer tells you what the vendor's commercial model actually optimizes for. A vendor that pushes T&M for everything has thin discovery muscle.

Good answer: Specific reasoning tied to the type of work — "T&M for the AI pieces because the scope was genuinely unknowable, fixed-bid for the integrations because we knew exactly what we wanted."

Red flag: "We started fixed-bid and converted to T&M three months in" — the vendor under-scoped to win the deal.

The IP and legal questions

Under copyright law in many jurisdictions, unless the contract explicitly assigns IP, initial ownership of code belongs to the creator — not the paying client. This is amplified across borders where U.S. judgments may be unenforceable. The cross-border IP enforcement problem is why these questions matter even when your MSA looks airtight on paper. Have your lawyer review the vendor's standard IP clauses regardless of what the reference tells you — this section surfaces operational risk, not legal risk.

4. Did your IP assignment clause ever get tested — a developer disputing ownership, an open-source license question, a departing contractor who had written material code?

Why it matters: With open-source components in 96% of commercial codebases per the 2024 OSSRA report, license-contamination risk is a near-certainty over a multi-year build. You want to know the vendor has encountered it and handled it.

Good answer: A story about an AGPL dependency that got flagged in code review, swapped out, and documented — or a contractor exit where the vendor produced signed IP assignments on request.

Red flag: "That never came up." On a two-year engagement with offshore contractors, it came up and nobody noticed.

5. When you asked for the full source code, build scripts, infrastructure-as-code, and credentials on a monthly or quarterly basis, what did you actually get?

Why it matters: IP ownership on paper is worthless if you don't have operational control of the artifacts. Vendors sometimes retain build systems, CI/CD pipelines, or deployment credentials in ways that create soft lock-in.

Good answer: "Everything was in our GitHub org and our AWS account from day one. They had access, we owned it."

Red flag: "They sent us a zip file every quarter" or "the CI pipeline was on their infra."

6. Which jurisdiction governed your contract, and did you ever need to enforce anything under it?

Why it matters: A Delaware-governed MSA with an India-based vendor is enforceable in theory and expensive in practice. You want to understand what the reference's lawyers advised and whether anything was ever tested.

Good answer: Specific — "Delaware law, Singapore arbitration seat, and we had our general counsel review the IP assignment specifically for enforceability in their home jurisdiction."

Red flag: "I don't know, legal handled it." This reference cannot tell you anything useful about IP risk.

The SOC 2, security, and audit-trail questions

ISC2's 2025 Supply Chain Risk Survey found 77% of organizations cite compliance with SOC 2, ISO 27001, or NIST as their top vendor security requirement, and 71% require audits or attestations before engagement. The reference call is where you learn whether the vendor's attestation is operationally real or a PDF they renew annually.

7. When your security team ran a vendor assessment on them, what came back on the questionnaire that you had to push back on?

Why it matters: Every vendor fudges something on a security questionnaire. You want to know what, and whether they corrected it or defended it.

Good answer: "They initially claimed MFA on all developer workstations, we pushed, turned out it was MFA on prod access but not laptops. They rolled out laptop MFA within 60 days."

Red flag: "They passed everything, no issues." Either the reference's security team is weak or the vendor's answers were not scrutinized.

8. During a developer rotation on your account, how was access revoked, and how did you verify it?

Why it matters: SOC 2 audit trail continuity breaks here most often. A departing developer who still has a GitHub token or a VPN credential a week later is a reportable incident.

Good answer: "Within 24 hours we got a signed off-boarding checklist. We audited it quarterly against our own access logs."

Red flag: "We trusted their process" or any version of "we never checked."

9. Have you ever had to produce evidence from their systems for your own SOC 2, ISO 27001, or customer audit? How long did it take?

Why it matters: Your auditor will ask for evidence from the vendor's environment. If the vendor takes three weeks to produce access logs, your audit timeline is the vendor's audit timeline.

Good answer: "Yes, we requested access reviews for our Type 2 audit, got them in under a week with the format our auditor needed."

Red flag: "We've never had to" (you will) or "it took forever."

The team continuity and turnover questions

Peer-reviewed research in the ACM Digital Library found offshore outsourcing firms experience significantly higher staff turnover than captive offshore sites, with learning costs from turnover identified as the primary cost driver to control. On a named-resource contract, this is your biggest operational risk after scope.

10. Of the engineers named in your original SOW, how many were still on the account 12 months in?

Why it matters: This is the number vendors most hate to be asked. A good vendor runs 70–80% retention on named accounts over a year. A bad one is below 40%.

Good answer: A specific number — "Four of six, and the two who rolled off gave us 30 days of overlap with their replacements."

Red flag: "The team was always well-staffed" with no numbers, or a story about multiple surprise replacements with no handover.

11. When your tech lead or senior engineer left, what did the vendor do in the first week?

Why it matters: Replacement is inevitable. The question is whether the vendor absorbs the ramp cost or passes it to you as billable hours while a new person learns the codebase.

Good answer: "They brought in a replacement at no charge for the first two weeks and had the outgoing lead on a 50% overlap."

Red flag: "We paid full rate from day one for someone who didn't know our system."

12. Did you ever interview and reject a proposed replacement? What happened next?

Why it matters: Your contract should give you right-of-refusal on named roles. You are testing whether that right is real or theoretical.

Good answer: "Yes, twice. They came back with new candidates within a week."

Red flag: "We didn't really have veto power in practice" or a story about pressure to accept a weaker candidate to keep velocity.

The delivery and relationship questions

The last block tests whether the vendor behaves like a partner or a staffing shop. ISG's 2025 survey found 38% of IT outsourcing clients described their arrangement as delivering less value than expected — these questions help you avoid joining that group.

13. Tell me about a time they told you something you didn't want to hear.

Why it matters: Vendors that only deliver good news are managing you, not building for you.

Good answer: A specific story — "Their architect told us our planned event-sourcing design was overkill and would cost us six weeks. He was right."

Red flag: A long pause, or a story that is actually the client telling the vendor something.

14. What's the one thing you'd structure differently if you were signing the contract again?

Why it matters: Every engagement has a regret. The specific regret tells you where this vendor is weakest.

Good answer: Specific and operational — "I'd have insisted on a joint retro every two weeks instead of monthly" or "I'd have put a cap on T&M for the data migration piece."

Red flag: "Nothing, it was great." Nobody signs a seven-figure contract and has zero regrets.

15. If I called you in six months and said we're having trouble with them, what would your first guess be about the cause?

Why it matters: This is the closing question. It forces the reference to predict the failure mode, which is the most honest signal you will get in 30 minutes.

Good answer: A specific, limited weakness — "Probably that you didn't give their PM enough product context up front" or "They're not great at UX, so if you expected design leadership you'd be frustrated."

Red flag: "I can't imagine what would go wrong." This reference is either coached or incurious — either way, the call is over.

What to do with the answers

Score each reference across the four risk dimensions — change-order discipline, IP and legal operational hygiene, SOC 2 audit-trail continuity, team retention on named roles — not on overall sentiment. A reference who gave you three specific stories about disputes with a vendor they'd still rehire is a better signal than a reference who gave you a glowing endorsement with no numbers.

If both shortlisted vendors clear the bar, the deciding factor is usually question 14 — the regret. Compare the two regrets and ask which one you can live with in your own engagement. That is almost always the real decision.

For a broader view of what to pressure-test before signing with an offshore partner on a US-based engagement, our notes on evaluating AI and software development partners for US buyers cover the technical deep-dive and security review stages that come after the reference call.

Frequently Asked Questions

How many references should I request from a shortlisted offshore development vendor?

Three, with at least one being a concluded engagement rather than an active account. Active accounts have ongoing commercial reasons to be diplomatic; churned clients do not. If a vendor cannot produce a single reference whose contract has ended, treat that as a data point in itself — it suggests either a very young firm or a pattern of relationships that end badly enough that clients decline to be referenced.

What's a reasonable change-order rate in the first 90 days of a custom software engagement?

In well-scoped engagements, change-order value in the first quarter typically sits in the single digits as a percentage of original contract value, and most of that is client-initiated scope addition rather than vendor rescoping. Anything above 20% in the first 90 days correlates with the hidden-cost patterns Gartner has documented in poorly scoped T&M work — an 18–25% total-contract uplift — and should trigger a hard conversation before you sign the next phase.

Should I ask references about pricing specifically?

Ask about pricing structure (fixed-bid vs T&M vs hybrid) and change-order patterns, not rates. Rates are not comparable across engagements because scope, seniority mix, and risk allocation differ. What you want from the reference is the shape of the commercial relationship — did the vendor absorb cost when they got discovery wrong, did T&M stay inside the estimated envelope, were there surprise invoices. That signal transfers; a rate card does not.

How do I verify a vendor's SOC 2 attestation is operationally real and not a PDF?

Ask the reference questions 7, 8, and 9 above — what came back on their security questionnaire, how developer off-boarding worked in practice, and how long it took to produce evidence for their own audit. A SOC 2 Type 2 report is a point-in-time attestation; the reference's lived experience over 12+ months tells you whether the controls are enforced day-to-day. Then have your own security team request the full report, the bridge letter, and the list of exceptions directly from the vendor.

What should I do if a vendor refuses to provide references from concluded engagements?

Push back once and ask why. Legitimate reasons exist — NDAs, acquired clients, references who have changed jobs — but the vendor should be able to name the reason. If the answer is vague, assume concluded engagements ended in ways the vendor would rather you not hear about, and either weight the active-account references more skeptically or move the vendor down your shortlist. On a multi-year commitment, you cannot afford to skip this signal.