AI Knowledge-Base Assistant Benchmarks for Computer Science Education Using OER

82 open-licensed Markdown documents form the knowledge base for benchmarking AI assistants in computer science education. The study by Xiaoqing Shen, last updated in June 2026, provides systematic benchmarks for models like Qwen-7B and DeepSeek-MoE, evaluating accuracy, latency, and energy consumption on consumer-grade GPUs. It includes results on multi-hop reasoning tasks and the impact of 4-bit quantization on VRAM usage.

Use Cases

• Benchmarking the accuracy and latency of open-source LLMs for educational question-answering based on the described multi-criteria evaluation framework.
• Evaluating the trade-offs of model quantization on VRAM usage and response utility for deployment on resource-constrained hardware.
• Designing energy-efficient, on-premise knowledge-base systems for computer science education based on the provided deployment guidelines and energy consumption metrics.
• Studying the application of Retrieval-Augmented Generation (RAG) with open educational resources for creating domain-specific assistants.

Strengths

• Benchmarks are based on a knowledge base of 82 structured, open-licensed documents.
• Provides specific performance metrics: Qwen-7B achieved 71.5% overall accuracy with 1.4s latency, and DeepSeek-MoE achieved 79.8% overall accuracy.
• Quantifies hardware efficiency: NF4 4-bit quantization reduced VRAM usage by approximately 38% for both tested models.
• Includes energy consumption measurements, reporting 1.8 mWh per query for the most efficient configuration.

Limitations

• The dataset is a 38.1 KB DOCX file, suggesting it contains a summary or supplementary document rather than the primary benchmark data or knowledge base corpus.
• Row count and column-level documentation for any underlying data are unknown, limiting suitability assessment.
• The description notes the system is characterized as a knowledge-base assistant, not a validated intelligent tutor, and comprehensive pedagogical validation is noted as future work.

Provenance

Source

figshare, author Xiaoqing Shen

Collection Method

Knowledge extraction from 82 open-licensed Markdown documents, followed by systematic benchmarking of LLMs.

Freshness

Last updated 2026-06-02 05:26:34