LettuceDetect: Token-Level Hallucination Annotations for LLM-Generated Code

13,203 training samples provide token-level annotations for hallucinated spans in LLM-generated code responses to real developer tasks from SWE-bench. The dataset, created by KRLabsOrg and last updated in May 2026, uses the unified LettuceDetect v2 taxonomy for character-level labeling. It includes separate splits for training, development, and testing, with 5,039 samples explicitly labeled as hallucinated in the training set.

Use Cases

• Training hallucination detection models based on token-level annotations.
• Benchmarking the performance of code-generating LLMs on real software engineering tasks.
• Analyzing patterns and types of hallucinations in generated code using the LettuceDetect v2 taxonomy.
• Developing fine-tuning or post-processing techniques to improve the factual accuracy of code completions.

Strengths

• Provides 13,203 annotated training samples, offering a substantial base for model development.
• Includes 5,039 explicitly hallucinated samples in the training split, enabling focused study of errors.
• Annotations are at the character level using a unified taxonomy, which suggests detailed labeling.
• Based on real developer tasks from SWE-bench, grounding the data in practical software engineering problems.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count for the test split is incomplete ('1,670…'), and total dataset size is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality and annotation consistency require manual inspection.

Provenance

Source

KRLabsOrg via Hugging Face.

Collection Method

Built by annotating hallucinated spans in LLM-generated code responses to tasks from the SWE-bench dataset.

Freshness

Last updated 2026-05-19 08:42:01; freshness should be verified.