DeepSeek DCLM Model Logits Packed to 8192 Sequence Length

DeepSeek DCLM Logits Packed 8192 is a dataset of teacher model logits for knowledge distillation, prepared by arcee-ai. The data is prepacked to a fixed sequence length of 8192 tokens. It was last updated on Hugging Face in December 2025.

Use Cases

• Knowledge distillation training using the compressed teacher logits with a vocab_size of 129,280 to train a smaller student model.
• Analyzing the effect of logit compression techniques, such as the legacy_logit_compression with exact_k=32 and polynomial_degree=8, on model performance.
• Training language models on long sequences by leveraging the prepacked data format with a fixed sequence_length of 8192.
• Benchmarking different compression algorithms (invert_polynomial, with_sqrt_term) for preserving teacher model output distributions.

Strengths

• Data is prepacked to a consistent sequence_length of 8192, simplifying training pipeline setup.
• Logits are processed with a defined compression method (legacy_logit_compression) targeting a reduced k=128 from an original exact_k=32.
• The dataset has a defined and large vocab_size of 129,280 tokens.

Limitations

• The total number of rows, samples, and dataset size are unknown, making it difficult to assess scale.
• The source and composition of the original training data used to generate the logits are not specified, limiting reproducibility.
• Specific data formats and the availability of validation/test splits are not provided.

Provenance

Source

arcee-ai, via Hugging Face.

Collection Method

Logits likely generated from a teacher model (DeepSeek-DCLM) and processed with compression and packing.

Freshness

Last updated on the platform on 2025-12-10.