Tokenlearn Cornstack Docs Coderankembed: Code Document Embeddings for 6 Languages

Mean token embeddings for code documents, produced by the nomic-ai/CodeRankEmbed model for static embedding distillation. The dataset contains 300,000 rows, with 50,000 code documents each from six programming languages sourced from CornStack. It was created by author minishlab and last updated on April 24, 2026.

Use Cases

• Training Model2Vec models for code retrieval based on the described embedding targets.
• Distilling static code embeddings based on the mean token embeddings produced by CodeRankEmbed.
• Benchmarking code representation models across six programming languages.
• Developing cross-lingual code search systems based on the multi-language corpus.

Strengths

• Contains 300,000 total rows, providing a substantial corpus for training.
• Includes code from six distinct programming languages, offering diversity.
• Embeddings are derived from a specific, named model (nomic-ai/CodeRankEmbed).

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

CornStack (nomic-ai)

Collection Method

Created with Tokenlearn for training Model2Vec models; embeddings produced by nomic-ai/CodeRankEmbed.

Time Range

null

Freshness

Last updated 2026-04-24 06:36:39; freshness should be verified.

Geography

null