IndicContextEval: A Benchmark for Audio LLMs Across 8 Indic Languages

IndicContextEval is a benchmark dataset for evaluating context utilisation in audio large language models. It contains 16,884 utterances totaling 55.93 hours of speech from 555 speakers across 8 Indic languages. The dataset, created by AI4Bharat, covers 23 professional domains and includes multiple speech styles and prompt levels.

Use Cases

• Benchmarking context window performance of audio LLMs based on the described 7 prompt levels (L0-L6).
• Training or fine-tuning multilingual speech models based on audio data from 8 Indic languages.
• Evaluating model robustness across different speech styles based on the described 'Read' and 'Extempore' categories.
• Analyzing domain adaptation in speech models based on data from 23 professional domains.

Strengths

• Covers 8 distinct Indic languages: Hindi, Bengali, Telugu, Marathi, Gujarati, Malayalam, Odia, and Urdu.
• Includes 16,884 utterances with a total duration of 55.93 hours from 555 speakers.
• Spans 23 professional domains and two speech styles (Read and Extempore).
• Provides a structured evaluation framework with 7 prompt levels (L0-L6).

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

AI4Bharat (author ai4bharat), hosted on Hugging Face.

Collection Method

Likely collected for benchmarking audio large language models; specific collection method not detailed.

Freshness

Last updated 2026-06-19 14:26:05; freshness should be verified.

Geography

Primarily covers Indic languages, suggesting a focus on the Indian subcontinent.