Instruction-Tuning Data For Sub-Billion Parameter Language Models

Smol-SmolTalk contains conversational data specifically curated for training language models with fewer than 1 billion parameters. The dataset was created by HuggingFaceTB and was last updated in February 2025. It was used to develop the instruction-tuned models SmolLM2-360M-Instruct and SmolLM2-135M-Instruct.

Use Cases

• Perform supervised fine-tuning (SFT) on conversational examples to align model outputs with human instructions.
• Apply Direct Preference Optimization (DPO) using this data as a base before training on preference datasets like UltraFeedback.
• Train models on shorter dialogue sequences, as conversations from the 'Smol-Magpie-Ultra' source are intentionally truncated.
• Benchmark small model performance on core instruction-following tasks by excluding specialized data like function calling examples.

Strengths

• Data is explicitly designed for models under 1B parameters, addressing a specific niche.
• Proven utility in creating two publicly released models, SmolLM2-360M-Instruct and SmolLM2-135M-Instruct.

Limitations

• Dataset size, row count, and specific column structure are not provided.
• Excludes certain task-specific data types such as function calling and reduces examples for rewriting and summarization.
• Contains synthetic or processed conversational data, which may not reflect organic human dialogue patterns.

Provenance

Source

HuggingFaceTB on Hugging Face.

Collection Method

Subset adapted from the larger SmolTalk dataset, with modifications for smaller models.

Time Range

null

Freshness

Last updated on 2025-02-06.

Geography

null