ArXiv and PubMed Scientific Papers with Abstracts and Sections

A collection of long, structured scientific documents sourced from ArXiv and PubMed OpenAccess repositories. The dataset contains two primary features: the full article body and its abstract, with paragraphs separated by '/n', along with section titles. It is provided by the author 'scillm' and was last updated in September 2023.

Use Cases

• Train abstractive summarization models using the article body as input and the abstract as the target summary.
• Analyze the structure and frequency of section_names across different scientific disciplines within the ArXiv and PubMed corpora.
• Build a retrieval-augmented generation system by indexing the article text and using the abstract for query relevance assessment.
• Perform topic modeling on the article feature to discover latent research themes in the combined ArXiv and PubMed archives.

Strengths

• Sourced from two major, authoritative open-access scientific repositories: ArXiv and PubMed.
• Contains structured long-form text with distinct article and abstract features for summarization tasks.
• Dataset size is categorized as between 1 million and 10 million entries, indicating a substantial collection.
• Explicitly tagged for the abstractive summarization task, clarifying its primary intended application.

Limitations

• The exact row count, column definitions beyond the three named features, and file formats are unknown, limiting precise assessment.
• Potential class or domain imbalance between the ArXiv (physics, math, CS) and PubMed (biomedical) source collections.
• Paragraphs are separated by the literal string '/n' instead of actual newline characters, which may require preprocessing for correct parsing.

Provenance

Source

ArXiv and PubMed OpenAccess repositories.

Collection Method

Obtained from the mentioned repositories; specific extraction method not detailed.

Freshness

Last updated on 2023-09-07; no update frequency specified.