Speed and memory

import resource

import pdfplumber

texts = []
with pdfplumber.open("samples/real/federal-register-2020-17221.pdf") as pdf:
    for page in pdf.pages:
        texts.append(page.extract_text())
        page.close()          # drop this page's parsed objects and cached text map

peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024    # KB on Linux
print(len(texts), "pages,", sum(map(len, texts)), "characters, peak memory", round(peak), "MB")

15 pages, 58492 characters, peak memory 55 MB

How we measured

The test documents repeat eight pages from this site’s samples (an inspection report with a table, a two-column article, a three-page bank statement and an invoice) to 1, 10, 100 and 1,000 pages. That is a typical mix of text, tables and ruling lines. Every page has its own content stream, as in a real document, so the parser cannot reuse work between pages.

Each measurement ran in a fresh Python process, so its peak memory (maximum resident set size, including Python itself at about 34 MB) belongs to that run alone. Times are wall-clock, the median of three runs, except the two slowest 1,000-page runs, which ran once. The script, lab/bench/perf.py, builds the documents and writes every number on this page.

Without close(), memory grows by roughly 3.7 MB per page and never comes back until the PDF is closed. With it, memory stays flat. Time per page is about 45 ms with close() and, on long documents, about twice that without it. Part of that gap is Python’s garbage collector, which keeps re-scanning the growing pile of objects: with the collector switched off, the 1,000-page run without close() took 78 s instead of 93 s (a single run, same machine). The rest is the cost of allocating and holding gigabytes of small objects.

Why memory grows

pdf.pages is a list that holds every Page object, and each page caches what it has parsed: every character, line and rectangle as a dictionary, plus the text map that extract_text() builds. A character dictionary has 22 keys, so a dense page costs a few megabytes; in our test, about 3.7 MB per page. Nothing is released while the PDF is open.

page.close() drops those caches. The page object stays usable: touch it again and it re-parses. pdf.close() (or leaving the with block) closes all pages and the file. Two version notes: page.close() returned in 0.10.4 after being absent from some earlier releases, and since 0.11.1 it also clears the text map cache, so use a current version (releases).

Derived pages share the cache of the page they came from. A page.crop(...) keeps the parent’s objects alive, so close the parent, not just the crop.

Where the time goes

Per page, averaged over the 100-page document, with each step measured on its own after parsing:

Parsing is about 91% of the cost of getting a page’s text. It happens once per page, the first time you touch page.chars, page.objects or any extraction method: pdfminer.six interprets the content stream and pdfplumber turns every object into a dictionary. Everything after that works on those dictionaries and is cheap. Three practical consequences:

• Text options do not make you faster. Tolerances, layout=True and the like change a step that costs a few milliseconds. The parse dominates.
• Cropping does not save time. A crop filters the parent page’s objects, so the page is parsed in full anyway. Extracting text from half the page was, if anything, slower than from the whole page, because of the filtering pass.
• Repeated calls are free. The second identical extract_text() took 3 microseconds: the text map is cached. Calling extraction methods several times on one page costs little; re-opening the file costs a lot.

Tables cost little on top of the parse here: extract_tables() added about 2.8 ms per page. Pages with thousands of short line segments (charts, maps, dense forms) take far longer, because the table finder compares edges with each other. If you only need text from such pages, do not call the table methods.

Use several processes

pdfplumber runs in one thread, and in the standard CPython build, threads do not run its pure-Python parsing in parallel. Separate processes do. Split the page numbers across workers, and let each worker open the file itself:

from concurrent.futures import ProcessPoolExecutor

import pdfplumber

PATH = "samples/real/federal-register-2020-17221.pdf"

def extract(page_numbers):
    # each worker opens the file itself; PDF and Page objects cannot be pickled
    with pdfplumber.open(PATH, pages=page_numbers) as pdf:
        out = []
        for page in pdf.pages:
            out.append((page.page_number, page.extract_text()))
            page.close()
        return out

if __name__ == "__main__":      # needed where workers are spawned (Windows, macOS)
    with pdfplumber.open(PATH) as pdf:
        n = len(pdf.pages)
    workers = 3
    chunks = [list(range(i + 1, n + 1, workers)) for i in range(workers)]
    with ProcessPoolExecutor(workers) as pool:
        results = sorted(r for part in pool.map(extract, chunks) for r in part)
    in_order = [p for p, _ in results] == list(range(1, n + 1))
    print(len(results), "pages; page order restored:", in_order)

15 pages; page order restored: True

Two workers nearly halve the time, and three bring it to 2.7×. A fourth made it slightly slower on this 4-core cloud machine, in all three runs. The parent process, the operating system and anything else running need a share of the CPU too, so one worker per core is not automatically best. Measure on your own hardware, starting at one fewer worker than you have cores.

Interleaving the page numbers (worker 1 gets pages 1, 4, 7 …) spreads heavy and light pages evenly, which matters in reports where the tables sit together. For many files, parallelise across files instead: one file per task, as many workers as cores, and nothing to reassemble.

Opening a file costs little: about 313 ms to open the 1,000-page document and read its first page. Passing pages=[...] to pdfplumber.open() saves little of that (257 ms), because pdfplumber still walks the whole page tree. It is still the cleanest way to give each worker its share.

When pdfplumber is the wrong tool for speed

Reading the same 1,000 pages with PyMuPDF took 0.82 s at 58 MB, about 55 times faster than pdfplumber with page.close(). PyMuPDF and pypdfium2 are compiled; pdfplumber and pdfminer.six are pure Python and build a dictionary for every character.

That is the trade: pdfplumber gives you every character’s position, font and colour, and the table finder built on them. If you need plain text from millions of pages, use a faster library for that pass, and pdfplumber for the pages where layout and tables matter (comparison, PyMuPDF vs pdfplumber, which also covers the licence difference).