Skip to content
Back to Writing

Python Script to Fetch Abstract from doi

1664 words

This script can help you fetch an article's abstract if its DOI is available.

toolPython

In the world of academic research, the Digital Object Identifier (DOI) is a persistent link to a specific piece of scholarly work. Whether you're building a research database, a literature review tool, or just trying to automate data collection, you often need to fetch not just the title or authors, but the abstract of an article. However, this is not always straightforward. DOI links redirect to various publisher websites, each with its own unique and complex structure. Many of these sites rely heavily on JavaScript to load content, making simple HTTP requests insufficient.

Today, we'll walk through a powerful Python script that tackles these challenges head-on. This script uses a combination of requests and selenium to reliably find and extract an article's abstract using only its DOI.

The Complete Script

Here is the complete Python script we will be dissecting. It's designed to be run from the command line, taking a DOI as an argument.

import sys
import time
import re
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
import requests
from urllib.parse import urlparse


def validate_doi(doi: str) -> bool:
    """Validate DOI format."""
    doi_pattern = r'^10\.\d{4,}/.+'
    return bool(re.match(doi_pattern, doi))


def get_abstract_from_doi(doi: str, max_retries: int = 3) -> str:
    """Get abstract from DOI or DOI URL with retry logic and better error handling."""
    # Support both bare DOI (e.g., 10.1007/...) and full URL
    if doi.startswith("http://") or doi.startswith("https://"):
        doi_url = doi
    else:
        if not validate_doi(doi):
            raise ValueError(f"Invalid DOI format: {doi}")
        doi_url = f"https://doi.org/{doi}"

    r = requests.get(doi_url, allow_redirects=True, timeout=30)

    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--disable-gpu")

    for attempt in range(max_retries):
        driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
        try:
            driver.get(r.url)
            time.sleep(2)  # Wait for page to load

            elements = driver.find_elements(By.XPATH, '//*[contains(text(), "bstract:")]')
            if len(elements) == 0:
                elements = driver.find_elements(By.XPATH, '//*[contains(text(), "bstract")]')

            elements = [elem for elem in elements if len(elem.text.strip()) >= 1]
            if not elements:
                if attempt < max_retries - 1:
                    time.sleep(2)
                    continue
                raise RuntimeError("Abstract element not found")

            element = min(elements, key=lambda x: len(x.text.strip()))
            characters = len(element.text.strip())
            while len(element.text.strip()) < 2 * characters:
                element = element.find_element(By.XPATH, "./..")
            abstract_text = element.text.strip()
            return abstract_text
        except Exception as e:
            if attempt < max_retries - 1:
                time.sleep(2)
                continue
            raise e
        finally:
            driver.quit()


def main():
    doi = None
    if len(sys.argv) > 1:
        doi = sys.argv[1]
    else:
        doi = "https://doi.org/10.1007/s11191-025-00677-6"

    try:
        print(get_abstract_from_doi(doi))
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()

How It Works: A Step-by-Step Guide

Let's break down the magic behind this script.

1. Prerequisites and Setup

The script relies on a few external Python libraries to handle web requests and browser automation.

  • requests: A simple, elegant HTTP library for making web requests.
  • selenium: The primary tool for automating web browsers. It allows our script to interact with a webpage just like a human user would, which is essential for pages that load content dynamically with JavaScript.
  • webdriver-manager: A handy utility that automatically downloads and manages the correct browser driver for selenium. [1][2]

You can install these libraries using pip: [3]

pip install selenium webdriver-manager requests

2. Validating the DOI

Before we do anything else, it's good practice to ensure our input is valid. The validate_doi function uses a regular expression to check if the provided string follows the standard DOI format, which typically starts with 10. followed by a prefix and a suffix.

def validate_doi(doi: str) -> bool:
    """Validate DOI format."""
    doi_pattern = r'^10\.\d{4,}/.+'
    return bool(re.match(doi_pattern, doi))

3. The Core Function: get_abstract_from_doi

This is where the main logic resides.

Resolving the URL:
The script first takes the input, which can be a bare DOI (10.1007/...) or a full URL (https://doi.org/...). It constructs a full doi.org URL if needed and then uses requests.get() to follow all redirects. This gives us the final URL of the article on the publisher's website.

if doi.startswith("http://") or doi.startswith("https://"):
    doi_url = doi
else:
    # ... validation ...
    doi_url = f"https://doi.org/{doi}"

r = requests.get(doi_url, allow_redirects=True, timeout=30)

Headless Browsing with Selenium:
Next, it fires up a headless instance of Chrome using selenium. "Headless" means the browser runs in the background without a visible UI, which is perfect for automation. The script navigates to the publisher's URL obtained in the previous step.

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
# ... other options for stability ...

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get(r.url)
time.sleep(2)  # Wait for dynamic content to load

Finding the Abstract:
This is the cleverest part of the script. Finding the abstract isn't as simple as looking for a specific HTML tag, because every website is different. The script employs a smart heuristic:

  1. It first uses XPath to find all HTML elements on the page that contain the text "Abstract" (or "abstract").
  2. It assumes the actual heading for the abstract section will be one of the shortest elements found (e.g., just the word "Abstract"). It selects this element.
  3. It then enters a loop, repeatedly moving up to the parent element (./.. in XPath). It stops when the amount of text within the current parent element is significantly larger than the heading itself. The assumption is that this parent element must contain both the heading and the full text of the abstract.
# Find elements containing "Abstract"
elements = driver.find_elements(By.XPATH, '//*[contains(text(), "bstract")]')

# Find the shortest one, likely the heading
element = min(elements, key=lambda x: len(x.text.strip()))
characters = len(element.text.strip())

# Traverse up the DOM to find the container
while len(element.text.strip()) < 2 * characters:
    element = element.find_element(By.XPATH, "./..")
abstract_text = element.text.strip()

Robustness:
The script includes a retry loop (for attempt in range(max_retries):) and a finally block to ensure the browser instance is always closed, even if errors occur. This makes it more resilient to network issues or unexpected page structures.

How to Use the Script

  1. Save the code above as a Python file, for example, fetch_abstract.py.
  2. Make sure you have the required libraries installed (pip install selenium webdriver-manager requests). [1][4]
  3. Run the script from your terminal, passing the DOI or DOI URL as a command-line argument:
# Using a bare DOI
python fetch_abstract.py 10.1007/s11191-025-00677-6

# Using a full URL
python fetch_abstract.py https://doi.org/10.1007/s00440-002-0236-0

Examples in Action

Let's test the script with two DOIs.

Example 1: Generative AI in Science Education

  • DOI: 10.1007/s11191-025-00677-6

Running the script produces the following output:

Abstract
This systematic review study investigated the research on the use of generative artificial intelligence (GenAI) tools in science education. The Web of Science database was screened by study titles and abstracts. In total, 41 peer-reviewed articles were included. The researchers created the coding scheme in light of GenAI use in education and science education literature. Results revealed that with 12 studies, most of the research was conducted in chemistry education. The highest frequency of the intended use was to evaluate GenAI tools’ performance in tasks (e.g., solving physics problems), whereas very few studies aimed at enriching science teaching (n = 1). Regarding student-centeredness, only five studies let learners use the technology without any direction, whereas in seven studies, learners’ use was guided. Concerning the level of use, in 14 studies, GenAI use was at a replacement level. The assessment was primarily focused on the instructional component in 23 papers. Most of the studies conducted with GenAI tools did not include participants (n = 20). Quantitative methods (n = 17) were preferred over qualitative ones (n = 10). Based on the results, the use of GenAI tools in science education seems to be in its infancy. At this stage, interdisciplinary partnerships between technology and science educators are necessary. In future research, science educators should focus more on using GenAI tools to enrich science instruction (e.g., how to enhance learners’ arguments, how GenAI tool use affects learners’ argument formation, and how GenAI tools can enrich the history and nature of science instruction). [3]

Example 2: Wired Spanning Forest

  • DOI: 10.1007/s00440-002-0236-0

Running the script with this DOI gives us:

Abstract
We show that a.s. all of the connected components of the Wired Spanning Forest are recurrent, proving a conjecture of Benjamini, Lyons, Peres and Schramm. Our analysis relies on a simple martingale involving the effective conductance between the endpoints of an edge in a uniform spanning tree. We believe that this martingale is of independent interest and will find further applications in the study of uniform spanning trees and forests. [4]

Conclusion and Limitations

This Python script provides a robust and clever solution to the common problem of fetching article abstracts. By combining the strengths of requests for URL resolution and selenium for rendering dynamic web pages, it can successfully navigate the varied landscape of academic publisher websites.

However, it's important to acknowledge its limitations. The heuristic used to find the abstract content—traversing up the DOM from the "Abstract" heading—works for many sites (like Springer), but it is not foolproof. Some websites might have unusual HTML structures that could confuse the script. For a truly industrial-strength solution, you might need to write specific parsers for major publishers or integrate more advanced content extraction libraries.

Nonetheless, for many use cases, this script is an excellent and highly effective tool to have in your research automation toolkit.