CoDEx: Knowledge Graph Completion Benchmark from Wikidata

CoDEx is a set of knowledge graph completion datasets extracted from Wikidata and Wikipedia, presented at EMNLP 2020. It comprises three knowledge graphs varying in size and structure, includes multilingual entity descriptions, and provides tens of thousands of hard negative triples. The benchmark was created by Tara Safavi of the University of Michigan to improve upon existing datasets in scope and difficulty.

Use Cases

• Benchmarking link prediction models based on the three knowledge graphs of varying size and structure.
• Evaluating triple classification performance using the provided hard negative triples.
• Training multilingual knowledge graph models based on the included entity and relation descriptions.
• Analyzing logical relation patterns within knowledge graphs as described in the empirical analysis.

Strengths

• Includes three distinct knowledge graphs, providing variability in size and structure for evaluation.
• Contains tens of thousands of hard negative triples that are plausible but verified to be false.
• Provides multilingual descriptions for entities and relations, supporting broader model development.
• Benchmarked against five extensively tuned embedding models, establishing baseline performance.

Limitations

• This repository hosts only the relational part; the full dataset with text and pretrained models is hosted elsewhere.
• Column-level documentation is absent; field semantics must be inferred after download.
• Last update date is unknown; freshness unverified.

Provenance

Source

Extracted from Wikidata and Wikipedia.