Claim Extraction and Verification

Content Certification uses automated claim extraction and semantic matching to verify factual assertions against your knowledge base. This guide explains the technical process and how you can optimize results.

Verification Pipeline

Stage 1: Claim Extraction

TruthVouch analyzes submitted content and identifies factual assertions:

Example Content:

“TruthVouch is headquartered in San Francisco. Founded in 2023, our platform monitors 9+ AI models including ChatGPT, Claude, and Gemini. Pricing starts at $349/month for the Starter plan.”

Extracted Claims:

1. “TruthVouch is headquartered in San Francisco”
2. “TruthVouch was founded in 2023”
3. “TruthVouch platform monitors ChatGPT”
4. “TruthVouch platform monitors Claude”
5. “TruthVouch platform monitors Gemini”
6. “Starter plan pricing is $349/month”

Claim extraction uses:

• Named Entity Recognition (NER): Identifies organizations, locations, people, dates
• Fact Extraction: Recognizes assertions, relationships, specifications
• Sentence Parsing: Breaks content into logical chunks

Stage 2: Truth Nugget Matching

Each claim is compared against your knowledge base (truth nuggets):

Matching Modes:

• Exact Match: Claim text exactly matches a truth nugget
• Semantic Match: Claim means the same thing as truth nugget (uses embeddings)
• Entailment Check: Claim logically follows from truth nugget data

Example Matching:

Claim	Truth Nugget	Match Type	Status
”Founded in 2023"	"Founding Year: 2023”	Exact	✓ Verified
”HQ in San Francisco"	"Headquarters: San Francisco, CA”	Semantic	✓ Verified
”Monitor ChatGPT"	"Supported Models: OpenAI GPT-4, ChatGPT, …”	Semantic	✓ Verified
”Costs $349/month"	"Pricing: Starter $299/month”	Exact	✗ Drift Detected

Stage 3: Score Calculation

Trust Score is calculated per-claim and aggregated:

Per-Claim Score = {
  100 if verified match
  0 if drift/contradiction detected
  50 if unverified (no matching truth nugget)
}

Certificate Score = average(claim_scores) + confidence_adjustment

Confidence Adjustment:

• Reduces score if claims are ambiguous or partially matched
• Increases score if multiple independent truth nuggets confirm claim

Verification Settings

Strictness Levels

Configure how aggressively claims are verified:

Strict (Default):

• Semantic similarity threshold: 95%
• Requires explicit truth nuggets
• Flags any deviation as drift

Balanced:

• Semantic similarity threshold: 85%
• Accepts minor variations in wording
• Allows reasonable inference from truth nuggets

Lenient:

• Semantic similarity threshold: 75%
• Accepts paraphrased claims
• Infers related facts from broader truths

Configuration:

client.certification.update_settings(
    strictness="balanced",
    unverified_claim_score=50,  # Score for claims without truth nuggets
    auto_revoke_threshold=20    # Revoke if drift > 20%
)

Truth Nugget Categories

Configure which categories of truth nuggets are used for verification:

client.certification.update_verification_scope(
    enabled_categories=[
        "product",
        "pricing",
        "company",
        "capabilities",
        "locations"
    ],
    excluded_categories=[
        "internal_only",
        "pending_review"
    ]
)

Handling Unverified Claims

When a claim has no matching truth nugget:

Option 1: Add Truth Nugget Create a new truth nugget to document this fact:

1. Review flagged claim in certification dashboard
2. Click Add Truth Nugget
3. Create nugget with the verified information
4. Re-verify certificate

Option 2: Accept Unverified Some claims may not need verification:

1. Review flagged claim
2. Click Mark as Approved
3. This adds the claim to approved list (doesn’t affect score)

Option 3: Flag as Hallucination If claim is inaccurate:

1. Click Mark as Incorrect
2. Optionally provide correct information
3. This adds to your hallucination correction history

Drift Detection

When underlying truth nuggets change, TruthVouch detects drift:

Automatic Monitoring:

• Every hour, re-verifies all active certificates
• Checks if truth nuggets have been modified
• Detects additions, deletions, value changes

Drift Types:

• Value Change: Truth nugget value updated (e.g., pricing change)
• Deletion: Truth nugget removed from knowledge base
• Addition: New competing truth nugget created
• Deprecation: Truth nugget marked as outdated

Example:

Certificate issued: "Starter plan costs $349/month"
Truth nugget: "Starter: $349/month"
↓ (Truth nugget updated)
New truth nugget: "Starter: $399/month"
↓
Drift detected: "Underlying fact changed"
Certificate status: "revoked" (if auto_revoke=true)

Verification Reports

Export detailed verification data:

Per-Certificate Report:

report = client.certification.get_verification_report(
    certificate_id="cert-123"
)

print(f"Verified Claims: {report.verified_count}")
print(f"Unverified Claims: {report.unverified_count}")
print(f"Drift Detected: {report.drift_count}")
print(f"Overall Score: {report.trust_score}/100")

for claim in report.claims:
    print(f"  - '{claim.text}'")
    print(f"    Status: {claim.status}")
    print(f"    Matched Truth Nugget: {claim.matched_nugget}")

Batch Report:

batch_report = client.certification.get_batch_verification_report(
    batch_id="batch-123"
)

print(f"Average Score: {batch_report.avg_score}")
print(f"Pass Rate: {batch_report.pass_rate}%")
print(f"Docs Requiring Review: {batch_report.review_required_count}")

Advanced: Custom Verification Logic

For specialized use cases, define custom verification rules:

# Example: Verify claims against external sources
client.certification.add_verification_source(
    type="api",
    name="product_api",
    endpoint="https://api.example.com/products",
    verify_fields=["price", "availability", "specs"]
)

# Now claims about pricing/availability auto-verify against product API

Next Steps

• Trust Score Guide: Understand your overall certificate score
• Auto-Revocation: Configure when certificates revoke
• Monitoring: Set up alerts for verification changes
• Badge Customization: Customize how scores are displayed