Task-Like Versus Non-Task-Like Text Examples

Featuring prompted text examples labeled as 'task-like' or 'not-task-like'. Task-like examples are sourced from P3, NI, RST, BigBench, FLAN, StackExchange, and UnNatural Instructions, while non-task-like examples are from C4. The C4 examples were sampled to match the length distribution of the task-like sources.

Use Cases

• Train a binary text classifier to distinguish 'task-like' from 'not-task-like' examples based on source dataset labels.
• Analyze the linguistic features of text from 'task-like' sources like P3, NI, RST, BigBench, FLAN, StackExchange, and UnNatural Instructions.
• Validate language model performance on held-out datasets from P3, NI, BigBench, FLAN, StackExchange, and UnNatural Instructions designated for validation.

Strengths

• Examples are explicitly labeled as 'task-like' or 'not-task-like' for clear classification tasks.
• Sampling of C4 examples was performed to match the length distribution of task-like sources, controlling for a key variable.
• Validation data includes held-out datasets from multiple sources (P3, NI, BigBench, FLAN, StackExchange, UnNatural Instructions).

Limitations

• Specific row count, column structure, and file formats are unknown, hindering assessment of scale and usability.
• The description indicates some datasets from P3 were excluded because examples were too long, potentially introducing a source bias.
• Non-tasky validation data from C4 was gathered without matching the length distribution, creating a potential validation set mismatch.

Provenance

Source

Aggregated from P3, Natural Instructions (NI), RST, BigBench, FLAN, StackExchange, UnNatural Instructions, and C4.

Collection Method

Examples were collected and labeled based on their source, with C4 examples sampled to match the length distribution of other sources. Specific datasets were held out for validation.

Freshness

Last updated on 2023-05-31.

Geography

Region tag suggests 'us', but specific spatial coverage is not detailed in the description.