This is a follow-up on the previous article PDF to image conversion workaround.

Motivation

The motiviation stays the same: Some PDF files are not rendered correctly in ReadEra. It’s not the whole file, some pages look fine, but other don’t. Very annoying.

Initial analysis

With the help of the tool RUPS, I could see the structure of the PDF file. I tried to compare the problematic file with a file, that works, and figured: The problematic file is a bit more complex in structure: The content of the page just refers to an object in the resources - so there is at least one more level of indirection. Maybe there is a way to simplify the PDF file without loosing content.

Finding the issue with GitHub’s Copilot

I first used Copilot, to generate a script to flatten the PDF structure, e.g. inlining Form XObjects. While the script kind of worked, it didn’t solve the problem. I went on asking more about the PDF streams - RUPS showed the the PDF code. I learned, that the problematic PDF used manual kerning, so - maybe removing this would solve the problem? After asking Copilot to not only remove kerning in the content directly in the page but also in referenced objects, it worked - but still didn’t solve the rendering problem.

I also learned, that there are some marked-content tags. But removing them didn’t solve the problem either.

Reducing the levels of nested graphics contexts also didn’t help.

At some point, I tried a suggestion using ghostscript to convert the PDF file into a older PDF version via gs -o fixed.pdf -sDEVICE=pdfwrite -dCompatibilityLevel=1.3 input.pdf. Opening that file with my desktop PDF viewer (GNOME’s Papers), it showed the same rendering problem: The text was not rendered. But I noticed, I could select the text. This now resulted in the correct syndrom description: invisible text until selected.

It appears, that my previous attempts where XY problem solutions - the actual problem was not, that the structure of the PDF file is too complex, but something else. So, simplifying the structure didn’t solve the problem (as I noticed).

Since I gave Copilot a couple of snippets of the PDF file, it guessed, that the symptom might be caused by the text rendering fill mode 4 (Tr 4). The generated script worked and for the first time, actually solved the initial problem: The PDF file was rendered correctly now - and unlike the older workaround, the text is still text and can be used to select and search (the PDF has not been converted into images).

See the full chat transcript.

Workaround script

The script now traverses each page in the PDF file, and each object in the resources of a file and looks out for Tr 4 operators. It then inserts directly after that a Tr 0 operator, which changes the text rendering mode again back to zero - which simply means “Fill text”. The text rendering modes are descripted in PDF 1.7 specification, chapter 9.3.6 “Text Rendering Mode”. Mode 4 means, “Fill text and add to path for clipping”. This apparently doesn’t work correctly.

The full script is listed below, but can also be downloaded directly as v1_force_text_render_mode_fill.py.

v1_force_text_render_mode_fill.py 
#!/usr/bin/env python3
"""
Force PDF text rendering mode to 0 (fill) by rewriting `n Tr` inside BT...ET
for page content streams and Form XObjects recursively.

This targets viewer bugs where text with Tr 4..7 (clipping variants) becomes
invisible unless selected.

Requirements:
  pip install pikepdf

Usage:
  python force_text_render_mode_fill.py in.pdf out.pdf
  python force_text_render_mode_fill.py in.pdf out.pdf --debug
"""

import sys
from typing import List, Set, Tuple
import pikepdf
from pikepdf import Name, Stream, Array, Dictionary

WS = b" \t\r\n\x0c\x00"
DELIMS = b"[]<>()/%"


class Logger:
    def __init__(self, enabled=False):
        self.enabled = enabled
    def log(self, msg):
        if self.enabled:
            print(msg, file=sys.stderr)


def read_literal_string(data: bytes, i: int) -> int:
    n = len(data)
    j = i + 1
    depth = 1
    while j < n and depth > 0:
        c = data[j]
        if c == ord('\\'):
            j += 2
            continue
        if c == ord('('):
            depth += 1
        elif c == ord(')'):
            depth -= 1
        j += 1
    return min(j, n)


def read_hex_string(data: bytes, i: int) -> int:
    n = len(data)
    j = i + 1
    while j < n and data[j] != ord('>'):
        j += 1
    if j < n:
        j += 1
    return j


def read_dict(data: bytes, i: int) -> int:
    n = len(data)
    j = i + 2
    depth = 1
    while j < n and depth > 0:
        c = data[j]
        if c == ord('('):
            j = read_literal_string(data, j)
            continue
        if c == ord('<'):
            if j + 1 < n and data[j + 1] == ord('<'):
                depth += 1
                j += 2
                continue
            j = read_hex_string(data, j)
            continue
        if c == ord('>') and j + 1 < n and data[j + 1] == ord('>'):
            depth -= 1
            j += 2
            continue
        j += 1
    return j


def read_array(data: bytes, i: int) -> int:
    n = len(data)
    j = i + 1
    depth = 1
    while j < n and depth > 0:
        c = data[j]
        if c == ord('('):
            j = read_literal_string(data, j)
            continue
        if c == ord('<'):
            if j + 1 < n and data[j + 1] == ord('<'):
                j = read_dict(data, j)
                continue
            j = read_hex_string(data, j)
            continue
        if c == ord('['):
            depth += 1
        elif c == ord(']'):
            depth -= 1
        j += 1
    return min(j, n)


def read_name(data: bytes, i: int) -> int:
    j = i + 1
    n = len(data)
    while j < n and data[j] not in WS + DELIMS:
        j += 1
    return j


def read_word(data: bytes, i: int) -> int:
    j = i
    n = len(data)
    while j < n and data[j] not in WS + DELIMS:
        j += 1
    if j == i:
        j += 1
    return min(j, n)


def is_number_word(tok: bytes) -> bool:
    try:
        float(tok.decode("latin1"))
        return True
    except Exception:
        return False


def rewrite_force_tr0(data: bytes, logger: Logger, label: str) -> Tuple[bytes, int]:
    out = bytearray()
    i = 0
    n = len(data)

    in_text = False
    # pending numeric operand candidates (byte ranges already emitted)
    recent_words: List[bytes] = []
    tr_changes = 0

    while i < n:
        c = data[i]

        # pass-through tokens that may contain delimiter chars
        if c in WS:
            j = i + 1
            while j < n and data[j] in WS:
                j += 1
            out.extend(data[i:j])
            i = j
            continue

        if c == ord('%'):
            j = i + 1
            while j < n and data[j] not in b"\r\n":
                j += 1
            if j < n and data[j] == ord('\r'):
                j += 1
                if j < n and data[j] == ord('\n'):
                    j += 1
            elif j < n and data[j] == ord('\n'):
                j += 1
            out.extend(data[i:j])
            i = j
            continue

        if c == ord('('):
            j = read_literal_string(data, i)
            out.extend(data[i:j])
            i = j
            continue

        if c == ord('<'):
            if i + 1 < n and data[i + 1] == ord('<'):
                j = read_dict(data, i)
            else:
                j = read_hex_string(data, i)
            out.extend(data[i:j])
            i = j
            continue

        if c == ord('['):
            j = read_array(data, i)
            out.extend(data[i:j])
            i = j
            continue

        if c == ord('/'):
            j = read_name(data, i)
            out.extend(data[i:j])
            i = j
            continue

        j = read_word(data, i)
        tok = data[i:j]

        # Track BT/ET context
        if tok == b"BT":
            in_text = True
            recent_words.clear()
            out.extend(tok)
            i = j
            continue
        if tok == b"ET":
            in_text = False
            recent_words.clear()
            out.extend(tok)
            i = j
            continue

        if in_text and tok == b"Tr":
            # Replace immediate preceding numeric operand with 0 where feasible.
            # We do this by appending canonical " 0 Tr" and relying on consumers;
            # but to avoid duplicate operand confusion, only do this when recent
            # token was a number (common pattern: "4 Tr").
            if recent_words and is_number_word(recent_words[-1]):
                # easiest safe rewrite: emit "0 Tr" and skip emitting current tok
                # BUT previous number already emitted. So instead overwrite logic
                # isn't possible in streaming; canonical workaround is append:
                # " Tr 0 Tr" -> final mode becomes 0.
                out.extend(tok)
                out.extend(b" 0 Tr")
                tr_changes += 1
                logger.log(f"[{label}] forced Tr->0")
                recent_words.clear()
            else:
                out.extend(tok)
            i = j
            continue

        out.extend(tok)

        # track recent word operands only inside text objects
        if in_text:
            if tok and tok not in (b"BT", b"ET"):
                recent_words.append(tok)
                if len(recent_words) > 4:
                    recent_words.pop(0)

        i = j

    return bytes(out), tr_changes


def is_form_xobject(obj) -> bool:
    if not isinstance(obj, Stream):
        return False
    st = obj.get(Name("/Subtype"), None)
    ty = obj.get(Name("/Type"), None)
    return st == Name("/Form") or (ty == Name("/XObject") and st == Name("/Form"))


def page_content_bytes(page) -> bytes:
    c = page.get(Name("/Contents"), None)
    if c is None:
        return b""
    if isinstance(c, Stream):
        return c.read_bytes()
    if isinstance(c, Array):
        parts = []
        for s in c:
            if isinstance(s, Stream):
                parts.append(s.read_bytes())
        return b"\n".join(parts)
    return b""


def set_page_content_bytes(pdf: pikepdf.Pdf, page, data: bytes) -> None:
    page[Name("/Contents")] = pdf.make_stream(data)


def process_form_recursive(form: Stream, visited: Set[Tuple[int, int]], logger: Logger, label: str) -> Tuple[int, int]:
    changed_streams = 0
    tr_changes_total = 0

    objgen = getattr(form, "objgen", None)
    if objgen and objgen in visited:
        return 0, 0
    if objgen:
        visited.add(objgen)

    original = form.read_bytes()
    rewritten, tr_changes = rewrite_force_tr0(original, logger, label)
    if rewritten != original:
        form.write(rewritten)
        changed_streams += 1
    tr_changes_total += tr_changes

    resources = form.get(Name("/Resources"), None)
    if isinstance(resources, Dictionary):
        c, t = process_forms_in_resources(resources, visited, logger, label)
        changed_streams += c
        tr_changes_total += t

    return changed_streams, tr_changes_total


def process_forms_in_resources(resources: Dictionary, visited: Set[Tuple[int, int]], logger: Logger, parent_label: str) -> Tuple[int, int]:
    changed_streams = 0
    tr_changes_total = 0

    xobj = resources.get(Name("/XObject"), None)
    if not isinstance(xobj, Dictionary):
        return 0, 0

    for name_obj, xo in xobj.items():
        if is_form_xobject(xo):
            label = f"{parent_label}/XObject{name_obj}"
            c, t = process_form_recursive(xo, visited, logger, label)
            changed_streams += c
            tr_changes_total += t

    return changed_streams, tr_changes_total


def process_pdf(infile: str, outfile: str, debug: bool) -> None:
    logger = Logger(enabled=debug)
    with pikepdf.open(infile) as pdf:
        modified_streams = 0
        tr_changes_total = 0
        visited_forms: Set[Tuple[int, int]] = set()

        for idx, page in enumerate(pdf.pages, start=1):
            label = f"page[{idx}]"
            original = page_content_bytes(page)
            if original:
                rewritten, tr_changes = rewrite_force_tr0(original, logger, f"{label}/Contents")
                if rewritten != original:
                    set_page_content_bytes(pdf, page, rewritten)
                    modified_streams += 1
                tr_changes_total += tr_changes

            resources = page.get(Name("/Resources"), None)
            if isinstance(resources, Dictionary):
                c, t = process_forms_in_resources(resources, visited_forms, logger, label)
                modified_streams += c
                tr_changes_total += t

        pdf.save(outfile)

    print(f"Wrote {outfile} (modified streams: {modified_streams}, Tr forced to 0: {tr_changes_total})")


def main():
    if len(sys.argv) < 3:
        print("Usage: python force_text_render_mode_fill.py in.pdf out.pdf [--debug]")
        sys.exit(2)

    infile = sys.argv[1]
    outfile = sys.argv[2]
    debug = "--debug" in sys.argv[3:]

    process_pdf(infile, outfile, debug)


if __name__ == "__main__":
    main()
</code></pre>
</details>

The script could be optimized to replace `Tr 4`, but for may purpose, it's good enough, what it does.
You'll need the python library [pikepdf](https://github.com/pikepdf/pikepdf).


## References
* <https://stackoverflow.com/questions/3549541/how-can-i-visually-inspect-the-structure-of-a-pdf-to-reverse-engineer-it>
* <https://github.com/itext/rups/releases/tag/26.01>
* PDF Specification: <https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf>