OpenHermes 2.5: 300,466 High-Perplexity Instruction Samples

Housing 300,466 high-perplexity text samples filtered from the OpenHermes 2.5 dataset by Malum0x in March 2026. It consists of the top 30% of records that Qwen2.5-3B-Instruct identified as having the highest cross-entropy loss during scoring.

Use Cases

• Fine-tuning language models on high-difficulty instruction samples to improve performance on edge cases
• Analyzing model loss patterns by inspecting high-perplexity text records
• Curating high-signal training data for small language models

Strengths

• 300,466 filtered samples
• Objective filtering via Qwen2.5-3B-Instruct cross-entropy loss
• Apache 2.0 licensed

Limitations

• High perplexity may capture malformed text or noise rather than just complex logic
• Model-specific bias from the Qwen2.5-3B-Instruct evaluator

Provenance

Source

teknium/OpenHermes-2.5

Collection Method

Filtered from source using cross-entropy loss scoring by Qwen2.5-3B-Instruct (4-bit NF4)

Freshness

Last updated March 2026.