Claim Extraction Pipeline

Before Brand Intelligence can track narratives and measure accuracy, TruthVouch automatically extracts individual factual claims from AI responses. This page explains how the claim extraction pipeline works.

Claim Extraction Overview

When you ask ChatGPT “What does [company] do?”, the response might contain dozens of claims:

• Company name and founding year
• Product offerings
• Customer counts
• Market position
• Capabilities and differentiators
• Team composition
• Recent news

Brand Intelligence must extract each claim individually, then measure whether it matches your truth nuggets.

Three-Stage Pipeline

Stage 1: Named Entity Recognition (NER)

Goal: Identify entities (people, places, companies, organizations, dates) in the AI response.

How it works: TruthVouch uses machine learning to automatically identify and classify named entities:

• Companies: ChatGPT, TruthVouch, Google
• People: Jane Smith, Elon Musk
• Locations: San Francisco, California, USA
• Dates: Founded in 2020, March 2024
• Numbers: 500 customers, $10M revenue
• Products: “AI Governance Platform”, “Hallucination Shield”

Example:

Input: "TruthVouch, founded in 2020, is an AI governance company
serving 500+ enterprise customers in San Francisco."

NER Output:
- Company: TruthVouch
- Founding Date: 2020
- Company Type: AI governance
- Customer Count: 500+
- Location: San Francisco

Stage 2: Claim Segmentation

Goal: Break sentences into individual, testable claims.

How it works: One sentence might contain multiple claims. TruthVouch automatically segments them:

Example:

Input: "TruthVouch, an AI governance company founded in 2020,
serves 500+ enterprises and integrates with 40+ AI models."

Claim 1: "TruthVouch is an AI governance company"
Claim 2: "TruthVouch was founded in 2020"
Claim 3: "TruthVouch serves 500+ enterprises"
Claim 4: "TruthVouch integrates with 40+ AI models"

Stage 3: Natural Language Inference (NLI)

Goal: Map claims to your truth nuggets and assess semantic equivalence.

How it works: TruthVouch automatically compares each extracted claim to your truth nuggets:

Extracted Claim: "TruthVouch was founded in 2020"
Your Truth Nugget: "Founded in 2020"
Match: EXACT MATCH (entailment)

Extracted Claim: "Serves 500+ enterprises"
Your Truth Nugget: "500+ enterprise customers"
Match: SEMANTIC MATCH (paraphrase)

Extracted Claim: "Integrates with 40+ AI models"
Your Truth Nugget: [No matching nugget]
Match: UNRELATED (no match)

Claim Extraction Examples

Example 1: Product Description

AI Response:

“TruthVouch is an AI governance platform that monitors LLM outputs and enforces policies in real-time. It provides sub-200ms latency and is HIPAA/SOC2 compliant.”

Extracted Claims:

1. “TruthVouch is an AI governance platform”
2. “TruthVouch monitors LLM outputs”
3. “TruthVouch enforces policies in real-time”
4. “Sub-200ms latency”
5. “HIPAA compliant”
6. “SOC2 compliant”

Matching Against Your Truth Nuggets:

• Nugget “AI Governance Platform” → Claim 1 matches (CORRECT)
• Nugget “Sub-200ms enforcement latency” → Claim 4 matches (CORRECT)
• Nugget “SOC 2 Type II certified” → Claim 6 matches (CORRECT)
• Nugget “HIPAA compliant” → Claim 5 matches (CORRECT)
• Nugget “Supports 9+ AI engines” → No matching claim (MISSING)

Example 2: Leadership & History

AI Response:

“Founded by Stanford AI researchers in 2020, TruthVouch’s CEO Jane Smith previously worked at Google. The company is now profitable and has 50 employees.”

Extracted Claims:

1. “Founded by Stanford AI researchers”
2. “Founded in 2020”
3. “CEO is Jane Smith”
4. “CEO previously worked at Google”
5. “Company is now profitable”
6. “Company has 50 employees”

Truth Nugget Matches:

• Nugget “Founded in 2020” → Claim 2 matches (CORRECT)
• Nugget “CEO Jane Smith” → Claim 3 matches (CORRECT)
• Nugget “50+ employees” → Claim 6 partially matches (CORRECT, “50” vs “50+”)
• Nugget “Founded by Stanford AI PhDs” → Claim 1 partially matches (CORRECT but less specific)
• Nugget “Profitable since 2023” → Claim 5 matches topic but MISSING specifics

Accuracy & Limitations

What the Pipeline Gets Right

The extraction pipeline achieves 85-92% accuracy on:

• Named entities (companies, people, locations, dates)
• Product names and versions
• Numeric claims (customer counts, performance metrics)
• Binary claims (yes/no, certified/not certified)

What It Struggles With

Accuracy drops to 60-75% on:

• Implicit claims (requires reading between lines)
• Context-dependent claims (what counts as “enterprise”?)
• Subjective claims (“leader”, “best”, “innovative”)
• Negations (“doesn’t do X” vs “only does Y”)

Example Challenge:

AI says: "TruthVouch is for enterprises, unlike Competitor X which focuses on SMB."

Extracted Claims (ambiguous):
- Claim 1: "TruthVouch is for enterprises" ✓
- Claim 2: "Competitor X focuses on SMB" ✓
- Implicit Claim 3: "TruthVouch doesn't focus on SMB" (inference)

The system extracts Claims 1 & 2 reliably.
Implicit Claim 3 is harder to extract consistently.

Limitations to Know

1. Paraphrasing ambiguity: “500+ customers” vs “over 400 employees” are different metrics, but pipeline might confuse them

2. Negation handling: “We don’t require setup” requires special logic to extract correctly

3. Comparatives: “Cheaper than Competitor A” requires knowing Competitor A’s pricing to validate

4. Context: “Founded when the AI boom started” is vague (when exactly?)

5. Subjectivity: “Industry-leading” is subjective and can’t be fact-checked

Claim Confidence Scores

Each extracted claim gets a confidence score (0-100):

High Confidence (80+):

• Named entities clearly present
• Direct statements without qualifiers
• Example: “Founded in 2020”

Medium Confidence (50-80):

• Implicit or paraphrased claims
• Statements with qualifiers (“roughly”, “approximately”)
• Example: “Founded in the early 2020s”

Low Confidence (<50):

• Highly ambiguous or subject to interpretation
• Claims requiring inference
• Subjective statements
• Example: “Industry-leading platform”

How Claims Feed Into Accuracy Scoring

The overall accuracy score weights high-confidence claims more heavily:

Accuracy =
  (High-confidence matches × 100%) +
  (Medium-confidence matches × 70%) +
  (Low-confidence matches × 40%)
  ─────────────────────────────────
  Total claims extracted

This means:

• Missing high-confidence facts (e.g., “Founded in 2020”) hurts accuracy significantly
• Missing low-confidence claims (e.g., “Innovative”) hurt less
• High-accuracy scores require getting the major facts right

Viewing Extracted Claims

Navigate to Brand Intelligence → Dashboard → Alerts and click any alert to see:

Extracted Claim:

“TruthVouch was founded in 2018”

Your Truth Nugget:

“Founded in 2020”

Confidence: 95% (clear, direct statement)

Match Assessment: INACCURATE (off by 2 years)

Engine: ChatGPT

Date Detected: March 15, 2024

You can flag claims as:

• Correctly extracted: The AI said this, and Brand Intelligence extracted it accurately
• Incorrectly extracted: Brand Intelligence misunderstood what the AI said
• Irrelevant: The claim doesn’t relate to your truth nuggets

This feedback helps train the extraction pipeline.

Improving Extraction Accuracy

You can help the pipeline by:

1. Using specific truth nuggets

   • Good: “Founded in 2020”
   • Bad: “Founded a few years ago”

2. Using consistent terminology

   • If you say “CEO” in your website, use “CEO” in truth nuggets
   • Not “Chief Executive Officer” and then “CEO”

3. Providing enough context

   • Good: “Serves 500+ enterprise customers”
   • Bad: “Enterprise”

4. Flagging extraction errors

   • When you see incorrect extraction, tell us
   • This trains the model for future accuracy

Next Steps

• Narrative Clustering → — How extracted claims form narratives
• Contamination Risk → — How false claims spread
• Accuracy Score → — How accuracy is calculated
• Dashboard → — View extracted claims and alerts