recursive call loop error in pypdf 4.1.0 and above

[ssjkamei]

I am running on Python 3.11.4.

It seems that an error occurs due to a loop of recursive calls when parallel processing using ProcessPoolExecutor.
I would like to know if there is a workaround.

Here is the code where the error occurs. Occurs in pypdf 4.1.0 or 4.2.0.

from concurrent.futures import ProcessPoolExecutor
from pypdf import PdfReader, PdfWriter
from multiprocessing import freeze_support

parallel_num = 4


def _page_numbering_pdf(pdf_file: PdfReader):
    pdf_file_writer = PdfWriter()
    for page in pdf_file.pages:
        pdf_file_writer.add_page(page)
    return pdf_file_writer


merge_base = PdfReader("test.pdf")

if __name__ == '__main__':
    freeze_support()
    with ProcessPoolExecutor(max_workers=parallel_num) as executor:
        pdf_file_writer = PdfWriter()
        result_pdf_writer_list = executor.map(
            _page_numbering_pdf,
            [merge_base]
        )

The file can be anything.
test.pdf

The following is the error.

  File "xxx\site-packages\pypdf\generic\_base.py", line 284, in get_object
    return self.pdf.get_object(self)
           ^^^^^^^^
  File "xxx\\site-packages\pypdf\generic\_base.py", line 301, in __getattr__
    return getattr(self._get_object_with_check(), name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "xxx\\site-packages\pypdf\generic\_base.py", line 290, in _get_object_with_check
    o = self.get_object()
        ^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded

[stefan6419846]

You are defining pdf_file_writer both globally and locally which is discouraged, although not related to this issue.

The real issue seems to be that you are passing a shared reader to the parallel code. There is no real way to ensure that each worker catches the reader in the same state, most likely leading to the issue you see. I just did a quick test which initializes the reader inside _page_numbering_pdf instead and got no issue.

I did not yet do any further research on what changed here in pypdf, but I personally would refrain from using such a pattern altogether.

[ssjkamei]

Hello stefan6419846.
Thank you for your answer.

I am actually processing several hundred pages of PDFs in separate parts. This process is divided into several parallel processing parts, so I am passing it as PdfReader or PdfWriter for processing.
Would it be preferable to save the error in this case as a pdf once per process and pass it on, rather than something to be corrected?

I am not sure if I am not understanding what you are talking about.

[stefan6419846]

I am clearly no expert for multiprocessing stuff and when and to which extent pypdf has thread-safe objects. By default, I would not rely on this and thus not share PdfReader or PdfWriter objects across multiple workers.

This might be related to #2642, although I am not sure.

[ssjkamei]

Thank you!

We will work around it by not passing PdfReader. I'll make sure to consider integrating the process so that there are no performance issues in doing so.

I will leave this discussion open to some extent to see if this is a bug, and close it if there are no other responses.

[ssjkamei]

Since no one seems to be particularly interested in this, I'll close this question.
An infinite loop occurs when using multiprocessing with PdfReader or PdfWriter.

• There is no discussion about whether it should be thread-safe.
• This can be avoided by reading with PdfReader in multiple threads, but it is necessary to save it as a pdf first.