Create PDF by streaming data

[loust333]

Hello @gettalong 😊

I was playing around with the plugin. I have a use case where a PDF for the entire year needs to be generated which contains every employee and data about their payslips.

Is there a way to progressively write in a StringIO or TempFile and, after writing, resetting the Composer (keeping styles but forgetting pages) ? And afterwards to concatenate the new pages to the Tempfile ?

I tried to merge, but the issue is that it still loads all the objects using that method.. 😅

Did I miss something ?

[gettalong]

Hello @loust333,

It is currently not possible to do that kind of streaming because creating the pages is separate from writing the PDF. However, it might be possible to save some memory by finalizing pages earlier. I will have to look at this to see how this could be done (similar to how forgetting the Canvas instances in the raw text benchmark - https://github.com/gettalong/hexapdf/blob/master/benchmark/raw_text/hexapdf.rb#L18-L21 - works).

Do you have some generalized example code for your use-case that you can share?

[gettalong]

Thanks @loust333! I will have a look at it over the weekend and see what I can do.

[gettalong]

Okay, I had a quick look at it:

• First of all: The report looks very nice! 👍
• I changed a bunch of lines to remove Rails syntax but that shouldn't influence the result in any way.
• I was able to reproduce your results.
• However, using Benchmark.memory produces quite some overhead, so I commented out the lines and ran the script using /usr/bin/time to get runtime and memory usage. On difficulty hard (=30) it took 13s and 78MiB without YJIT and 7s and 88MiB with YJIT. The generated report has 271 pages and that memory usage seems okay to me in this case.
• Just to have something to compare with, I set the difficulty to 100. That generated a report with 2401 pages and a size of abou 26MiB, took with YJIT about 82s and used about 436MiB memory.

I will have a more in-depth look later to see if it is possible to optimize this.

[loust333]

That's weird. Without any changes, to hexapdf, it uses only that much memory.

Can you share the code so that I can try out in rails ? Is it because of the benchmark memory ? 😵

[gettalong]

I just added the needed require statements and changed #present? and #second to their plain Ruby counterparts. That made the script run-able without Rails, giving your results. Then I commented out the Benchmark.measure lines (56/57 plus their respective end lines).

I haven't looked at 'benchmark-memory' but in my experience all memory measurement tools have some overhead, especially when many objects are involved as in this case. The difference in memory usage on the medium difficulty level was ~200MiB without Benchmark.memory and ~1400MiB with it.

[loust333]

Thanks for the explanation, i will try out and give you more feedback. 🙂

[loust333]

Hi @gettalong,

I tested out, and it was indeed the Benchmark. memory which was creating the overhead. After removing, the generation of PDF was completed without any issue. 😅

Thanks a lot. I just need to test 8 more PDF documents. 😩

[gettalong]

Let me know if you need any help!

For what's worth:

• You can save some memory by creating a style for :small_right, i.e. style(:small_right, base: :small, text_align: :right) which saves some memory (on difficulty 10 it saved about a quarter of Style objects and 6000 less Arrays as well as 11000 less Hashes)

• You can also reduce memory and increase performance by removing the cell_style from container.table(cells, cell_style: { border: { width: 0 }, padding: [1, 2 ,1, 1] }) do |args| and moving it as first part into args, i.e. args[0..-1, 0..-1] = { border: { width: 0 }, padding: [1, 2 ,1, 1] }.

• Two small hints: You don't need to specify style: :base since that is the default. And if part of the data to #formatted_text looks like {text: "..."} you don't need the hash and can just use "...". For example, line 107 {text: "#{employee_data[:name]} (", style: :base}, could also be written as "#{employee_data[:name]} (",.

• I will update HexaPDF to allow step sizes for ranges, i.e. (1..11).step(2), when defining table cell styles. This allows you to define the table cell styles using just 4 instructions instead of repeating the same one for all uneven rows/columns).

• I found another small optimization for HexaPDF itself that saves memory and is more performant by not relying on HexaPDF::Document::Layout#method_missing.

[loust333]

Thanks a lot @gettalong. It worked like a charm.

I will continue testing the implementation of HexaPDF in all the others PDFs. 🥱

One little remark that I encounter during the test phase:

cells << [
    layout.text(income_line[:code] || '', style: :small),
    layout.text(income_line[:amount] || '', style: :small),
  ]
  
composer.table(cells)

OR

composer.text(redundant_translations[:title], style: :h2)

If the variable is nil an error is raised. Sometimes the variable may be nil. To avoid the issue || '' is needed in most calls.

We can keep this discussion open for those that test the methods with Benchmark.memory 😂