Unitxt: IBM Catalog for AI Benchmarking and NLP Evaluation

IBM Research maintains Unitxt as a Python-based catalog for AI benchmarking, featuring the world's largest collection of tools and data for end-to-end performance evaluation as of February 2026. It provides a standardized framework for assessing Large Language Models and vision systems across diverse NLP and computer vision tasks.

Use Cases

• Benchmarking LLM performance using the evaluation-metrics catalog
• Standardizing data ingestion for NLP tasks using the nlp-library templates
• Creating custom evaluation pipelines for MLOps workflows using the Python API

Strengths

• Apache-2.0 license for enterprise use
• Maintained by IBM Research
• Supports both vision and NLP domains

Limitations

• Requires Python proficiency to access and implement benchmarks
• Acts as a meta-resource, meaning data availability depends on external source connectivity

Provenance

Source

IBM Research

Collection Method

Curated from various AI benchmarking sources

Freshness

Updated February 2026

Geography

Global