L0DROP: Sparsified Encoder Outputs for Sequence-to-Sequence Models

L0DROP, a stochastic gating layer, prunes 40-70% of Transformer encoder outputs without significant quality loss. The method was tested on two machine translation and two summarization tasks by Biao Zhang of the University of Edinburgh. Pruning yields decoding speedups of up to 1.65x on document summarization and shows systematic preferences for pruning function words and punctuation.

Use Cases

• Improving decoding efficiency in Transformer models based on encoder output sparsification.
• Analyzing model behavior to understand which word types (e.g., function words) are pruned most frequently.
• Exploring rule-based pruning patterns using part-of-speech tags, word frequency, and position information mentioned in the description.

Strengths

• Experimental results show pruning 40-70% of source encodings is possible without significant quality compromise.
• The method yields a decoding speedup of up to 1.65x on document summarization tasks against a standard Transformer.

Limitations

• Row count and dataset size are unknown, which may limit suitability assessment.
• Column-level documentation is absent; field semantics must be inferred after download.
• Last update date is unknown; freshness unverified.

Provenance

Source

University of Edinburgh

Collection Method

Research paper and associated experimental results.