FinReflectKG-EvalBench: Faithfulness Evaluation for Financial Knowledge Graph Triplets

A benchmark for evaluating the faithfulness of large language models, built on a financial knowledge graph. It contains over 480,000 triplet evaluations sourced from 101 S&P 100 companies, including judge reasoning and full document context. The dataset was curated by Domyn and was last updated on October 6, 2025.

Use Cases

• Benchmarking LLM faithfulness in extracting financial relationships based on the provided triplet evaluations.
• Evaluating knowledge graph extraction quality using the reflection-based methodology described.
• Training or fine-tuning models for financial information extraction based on the detailed judge reasoning.
• Analyzing the consistency of entity and relation extraction from corporate documents for the 101 covered companies.

Strengths

• Contains over 480,000 evaluation instances, providing a substantial scale for analysis.
• Covers 101 S&P 100 companies, offering a focused scope on major financial entities.
• Includes detailed judge reasoning and full document context for each evaluation, supporting in-depth analysis.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment for specific tasks.
• The dataset uses a CC-BY-NC-4.0 license, which restricts commercial use.

Provenance

Source

Domyn via Hugging Face.

Collection Method

Built on top of the FinReflectKG knowledge graph using a reflection-based methodology.

Freshness

Last updated 2025-10-06 15:42:56; freshness should be verified.