(cid:x) instead of text

See it on a real file

cid-font.pdf is a page we generated and then broke in the same way real files are broken: its embedded font keeps its glyphs but has lost both of its Unicode tables. It renders normally. pdfplumber, faithfully, cannot tell what the letters are:

import pdfplumber

for name in ["invoice", "cid-font"]:
    with pdfplumber.open(f"samples/{name}.pdf") as pdf:
        chars = [c for page in pdf.pages for c in page.chars]
        cid = sum(c["text"].startswith("(cid:") for c in chars)
        print(f"{name:9} {cid}/{len(chars)} chars are (cid:x)  ->", repr(pdf.pages[0].extract_text()[:28]))

invoice   0/851 chars are (cid:x)  -> 'Hollow Pine Supply Co. INVOI'
cid-font  141/141 chars are (cid:x)  -> '(cid:81)(cid:117)(cid:97)(ci'

Why it happens

Text in a PDF is a list of codes in a font. For simple fonts the codes are usually ordinary character codes. For CID-keyed fonts (/Type0, used for large character sets and by many PDF generators for everything), each code is a CID: a number that picks a glyph. To turn a CID back into a letter, a reader needs one of:

• a /ToUnicode map in the font dictionary (the normal way), or
• a mapping it can derive from the embedded font program’s own tables, or
• a standard character collection it knows (for example Adobe-Japan1).

When none is available, pdfminer.six (which pdfplumber uses to parse fonts) emits (cid:N). You can confirm that from the font dictionary:

import pdfplumber
from pdfminer.pdftypes import resolve1

with pdfplumber.open("samples/cid-font.pdf") as pdf:
    page = pdf.pages[0]
    fonts = resolve1(page.page_obj.resources.get("Font", {}))
    for name, ref in fonts.items():
        font = resolve1(ref)
        print(name, resolve1(font.get("BaseFont")), "| subtype:", resolve1(font.get("Subtype")),
              "| encoding:", resolve1(font.get("Encoding")), "| ToUnicode:", "ToUnicode" in font)

F1 /'EVXJRZ+DejaVuSans' | subtype: /'Type0' | encoding: /'Identity-H' | ToUnicode: False

Type0, Identity-H and no ToUnicode: the combination that produces (cid:x). Producers do this to save space or, occasionally, to discourage copying; subsetting tools sometimes drop the map by accident.

Fix 1: chr(N), when CIDs are Unicode

Some generators number glyphs by their Unicode code point, so (cid:81) really is chr(81), Q. The quick fix, widely suggested on Stack Overflow, works for those files:

import re
import pdfplumber

def decode_cids(text):
    """Replace (cid:N) with chr(N). Only correct when the PDF's CIDs are Unicode code points."""
    return re.sub(r"\(cid:(\d+)\)", lambda m: chr(int(m.group(1))), text)

def looks_like_text(text):
    letters = sum(ch.isalpha() or ch.isspace() for ch in text)
    return letters / max(len(text), 1) > 0.7

with pdfplumber.open("samples/cid-font.pdf") as pdf:
    raw = pdf.pages[0].extract_text()

guess = decode_cids(raw)
print(guess.splitlines()[0])
print("plausible:", looks_like_text(guess))

Quarterly revenue rose in every region.
plausible: True

Our sample was made by one of those generators, so it decodes perfectly. Most (cid:x) files are not like this. When CIDs are glyph indexes, chr() produces confident-looking garbage. The looks_like_text() check is a minimal safeguard; better still, compare against a few words you can read on the page.

Fix 2: OCR the page

OCR does not need the font’s mapping: it reads the glyphs as an image. --force-ocr rasterises each page and replaces the broken text layer:

ocrmypdf --force-ocr -l eng broken.pdf readable.pdf

import ocrmypdf
import pdfplumber

# --force-ocr rasterises every page and replaces its text with OCR output
ocrmypdf.ocr("samples/cid-font.pdf", "cid-ocr.pdf", force_ocr=True, language="eng", progress_bar=False)

with pdfplumber.open("cid-ocr.pdf") as pdf:
    print(pdf.pages[0].extract_text().splitlines()[:2])

['Quarterly revenue rose in every region.', 'Invoice total: 1,118.23 USD']

Both lines came back exactly. On real documents expect occasional OCR errors (see the measured results on scanned PDFs), and remember that --force-ocr also rasterises any vector graphics on the page, so table ruling lines become pixels: use the borderless table recipe for tables afterwards.

Things that do not help

A note on other libraries

We ran pypdf 6.21.4 and PyMuPDF 2.1 on cid-font.pdf, and both returned the correct text. They do not have the missing table either; when a font has no Unicode map they treat the codes as Unicode, the same guess as the chr(N) fix above. On a file whose codes are not Unicode, the guess fails quietly. cid-subset.pdf uses subset fonts where the fi and fl ligatures were given codes 1 and 2, then lost their maps:

import pdfplumber
import pymupdf
import pypdf

path = "samples/cid-subset.pdf"     # subset fonts whose codes are not Unicode, maps removed
line = lambda text: next(l for l in text.splitlines() if l.startswith("English"))

print("pdfplumber:", repr(line(pdfplumber.open(path).pages[0].extract_text())))
print("pypdf     :", repr(line(pypdf.PdfReader(path).pages[0].extract_text())))
print("PyMuPDF   :", repr(line(pymupdf.open(path)[0].get_text())))

pdfplumber: 'English: The office (cid:1)les were (cid:2)agged for review.'
pypdf     : 'English: The office \x01les were \x02agged for review.'
PyMuPDF   : 'English: The office \x01les were \x02agged for review.'

pdfplumber flags the two glyphs it cannot decode; the other two return invisible control characters (\x01, \x02) that will slip into your data unnoticed. A second library is still a cheap test, but check its output for control characters.

Partial (cid:x), where a few symbols are codes but most text is fine, is common with special fonts for bullets, ticks and arrows. There it is usually safe to replace or drop the few known codes.

Check your own file

The PDF Doctor reports the share of (cid:x) characters per page, along with scanned pages and duplicated text. It runs pdfplumber in your browser; your file is not uploaded.

Run pdfplumber to see extract_text() for this page.

table_settings =   # defaults