Custom Questionnaires

Create custom vendor evaluation criteria tailored to your organization’s specific needs. Build questionnaires to assess vendors on dimensions that matter most to you — security, cost, support, domain expertise, etc.

Why Custom Criteria?

Standard vendor templates cover common evaluation areas, but your organization may have unique requirements:

• Industry-specific — Healthcare orgs prioritize HIPAA compliance; financial orgs need SOX support
• Use-case specific — Code generation priorities differ from content generation
• Strategic — You may weight innovation higher than cost, or vice versa
• Constraint-based — You may require data residency, on-premise options, or specific certifications

Custom criteria let you define what “best fit” means for YOUR business.

Creating Custom Criteria

Step 1: Define Evaluation Dimensions

Choose 5-10 key areas relevant to your vendor decision:

Examples:

For a Finance Company:

• Compliance (SOX, financial data handling)
• Security (encryption, access controls, audit trails)
• Cost (API pricing, volume discounts)
• Performance (latency, uptime SLA, accuracy)
• Support (SLA, response time, dedicated account manager)
• Data residency (must be in US for regulatory reasons)

For a Healthcare Company:

• Compliance (HIPAA, BAA availability)
• Data Privacy (patient PII protection)
• Security (encryption, audit logging)
• Performance (latency for real-time diagnosis support)
• Vendor Stability (financial health, customer base size)

For an Engineering-focused Startup:

• Code generation quality (HumanEval score)
• Cost (price per token)
• API maturity (SDK quality, documentation)
• Performance (latency, throughput)
• Community support (forums, examples, third-party integrations)

Step 2: Define Scoring for Each Dimension

For each dimension, define how you’ll score vendors (0-100 scale):

Example: Compliance (for Finance)

Score	Definition	Example
90+	Exceeds requirements	SOX certified, annual audit, dedicated compliance officer
70-89	Meets core requirements	SOX compliant, recent audit, good documentation
50-69	Mostly adequate	SOX compliant but no recent audit, vague documentation
30-49	Missing key controls	Claims SOX compliance but no audit, limited documentation
0-29	Inadequate	Not SOX compliant, no audit, no controls

Example: Cost (for Startups)

Score	Cost/1M tokens	Rationale
100	<$0.50	Excellent value (open-source self-hosted)
80	$0.50-$2	Good value for quality
60	$2-$5	Fair pricing
40	$5-$15	Premium pricing
20	$15-$30	Very expensive
0	>$30	Prohibitive cost

Step 3: Set Weights

Not all dimensions are equally important. Assign weights (% of total score):

Example: Finance Company

Dimension	Weight	Rationale
Compliance	35%	Non-negotiable; regulatory requirement
Security	25%	Critical for data protection
Cost	20%	Important but not dominant
Performance	15%	Needed but 500ms latency is acceptable
Support	5%	Nice-to-have; not critical

Weighted Score = SUM(Dimension Score × Weight)

Step 4: Add Questions/Sub-dimensions (Optional)

Break down complex dimensions into specific questions:

Compliance dimension might include:

• “Is the vendor SOX/HIPAA/GDPR certified?”
• “How recent is their audit?” (within 2 years = full score, older = deduction)
• “Do they provide audit reports to customers?”
• “What happens if they have a security breach?” (contractual language matters)
• “Do they allow customer audits/penetration testing?”

Each sub-question contributes to the overall Compliance score.

Using Custom Criteria

Step 1: Create Questionnaire

1. Go to Vendor Evaluation → Custom Questionnaires
2. Click New Questionnaire
3. Enter name (e.g., “LLM Provider Evaluation for Finance”)
4. Add dimensions (steps 1-4 above)
5. Save as template

Step 2: Invite Vendor Participation

Send questionnaire to vendors:

1. Click Share
2. Enter vendor email addresses
3. They complete the questionnaire (or you fill it in based on their docs)
4. Responses stored in TruthVouch

Step 3: Score Responses

1. Review each vendor’s answers
2. Assign scores (0-100) for each dimension
3. System auto-calculates weighted scores
4. See final ranking

Step 4: Analyze Results

Scoring Summary:

• Vendor A: 82 (strong all-around)
• Vendor B: 78 (good, weak on cost)
• Vendor C: 65 (weak compliance)

Strengths & Weaknesses:

• Vendor A: Excellent compliance (92), good cost (78), excellent support (88)
• Vendor B: Excellent cost (95), weak compliance (62), adequate support (70)
• Vendor C: Best cost (98), poor compliance (45), adequate performance (72)

Recommendation: Vendor A is best fit (balances compliance and cost). Vendor B worth exploring if you can negotiate data residency and compliance improvements.

Example: Custom Evaluation for Data Analytics Use Case

Questionnaire: “AI Analytics Platform Evaluation”

1. Analytical Accuracy (Weight: 35%)

• How does the model perform on your domain? (test with sample data)
• Accuracy on financial data analysis? (RMSE, MAE metrics)
• Ability to explain its reasoning? (transparency score)

2. Data Privacy & Security (Weight: 25%)

• Data residency options (must be US-only for compliance)
• Encryption at rest and in transit?
• Data retention policies? (delete after 30 days)
• HIPAA BAA available?

3. Cost (Weight: 20%)

• API pricing per query
• Data transfer costs
• Volume discounts available?
• Infrastructure overhead if self-hosted

4. Integration & Compatibility (Weight: 15%)

• Does it integrate with our data warehouse (Snowflake, BigQuery)?
• API maturity and documentation quality
• SDK availability (Python, SQL, REST)

5. Support & SLA (Weight: 5%)

• Response time for critical issues
• Dedicated support available?
• Uptime SLA guarantee

Scoring Example

Vendor	Accuracy	Privacy	Cost	Integration	Support	Weighted Total
Weight	35%	25%	20%	15%	5%	—
Vendor A	90	85	60	75	70	79.5
Vendor B	85	95	90	85	80	87
Vendor C	95	40	95	70	75	76

Analysis:

• Vendor B is best overall (87) — strong privacy, cost, and integration
• Vendor A is second (79.5) — excellent accuracy but weak cost
• Vendor C is risky (76) — excellent accuracy and cost but poor privacy (disqualifying for regulated data)

Best Practices

1. Involve Multiple Stakeholders

Get input from:

• Business — Prioritizes ROI and speed to value
• Finance — Focuses on cost and SLA penalties
• Security/Compliance — Emphasizes safety and audit readiness
• Technical — Cares about integration and performance

Different perspectives prevent biased evaluation.

2. Test with Real Data

Don’t just rely on vendor claims:

• Request trial access
• Test with your actual use cases
• Measure accuracy, latency, and cost
• Collect feedback from internal users

3. Document Weighting Rationale

Why is Compliance weighted 35%? Why not 50%?

• Documented rationale prevents later disputes
• Creates audit trail for stakeholder buy-in
• Helps if requirements change (update weighting accordingly)

4. Revisit Criteria Annually

AI vendor landscape changes fast:

• New models, pricing, certifications
• Your business needs evolve
• Competitive offerings improve
• Update questionnaire and re-score existing vendors

Sharing & Collaboration

Share with Team

1. Click Share on a questionnaire
2. Invite team members (Governance, Security, Finance, Engineering)
3. Set permissions (View, Comment, Edit)
4. Team members add notes, questions, scores

Track Decision History

Comments are timestamped and attributed:

• “Per Finance, cost weight should be 20% not 15%” — [Sarah, Mar 15]
• “Compliance is non-negotiable due to SOX” — [Legal, Mar 14]
• “Performance insufficient; need <50ms latency” — [Engineering, Mar 13]

Creates audit trail for compliance reviews.

Related Topics

• Vendor Scoring — Standard evaluation templates
• Vendor Templates — RFP and scorecard templates
• Tools - Comparison — Built-in vendor comparison tool

Next Steps

1. List key decision criteria for your vendor evaluation
2. Define scoring rubric (0-100 per dimension)
3. Set weights prioritizing your needs
4. Create questionnaire in TruthVouch
5. Share with vendors for completion
6. Score and rank candidates
7. Present findings to stakeholders