Giving access to synthetic sequences generated from Hidden Markov Models (HMMs) to benchmark in-context learning in language models. It includes the GINC (Generative In-Context) data generation scripts and experimental code used to test the hypothesis that models perform implicit Bayesian inference when processing few-shot prompts.
Use Cases
- Measure in-context learning accuracy by evaluating next-token prediction on sequences with specific latent HMM states.
- Analyze model behavior during implicit Bayesian inference by tracking performance as the number of in-context examples increases.
- Benchmark transformer architectures on synthetic data where the ground-truth generative distribution is known and controllable.
Strengths
- Generated using Hidden Markov Models (HMMs) to create sequences with underlying latent structures.
- Includes scripts to produce the GINC (Generative In-Context) dataset for small-scale benchmarking.
- Designed for sequences where the model must predict the next token based on a few-shot prompt of latent-related examples.