"No bytes were copied", derivative generation broken

[kayakr]

Happening on a local development VM (vagrant, Virtualbox, playbook via Ansible) after updating to PHP 7.4 and Drupal 9.2.5.
When adding new content, e.g. PDF as Digital document, Drupal watchdog reports:
Symfony\Component\HttpKernel\Exception\HttpException: No bytes were copied to public://2021-09/50355-Extracted Text.txt in Drupal\islandora\MediaSource\MediaSourceService->updateFile() (line 206 of /var/www/html/drupal/web/modules/contrib/islandora/src/MediaSource/MediaSourceService.php).

Previously encountered by Eli Zoller, and Jordan Dukart, discussed at https://islandora.slack.com/archives/CM5PPAV28/p1619194482331900

I encountered it on a different instance but it somehow resolved itself. I want to ensure this is logged as an issue since it has been encountered by three parties so far.

In this case, I can revert the VM to PHP 7.2, Drupal 8.9.18 and derivatives are ok. So it's something attributable or triggered by changes to PHP version, PHP config, or Drupal core and islandora* code changes (not microservices, or camel, etc. AFAICT)

The error occurs when the derivative file is sent to Drupal as an HTTP PUT. This is handled by islandora/src/MediaSource/MediaSourceService.php:

• MediaSourceService->putToNode()

• MediaSourceService->updateFile()

  • $content_length = stream_copy_to_stream($resource, $destination);
    • where $resource is Resource from $request->getContent(TRUE) and $destination is Resource from fopen($uri, 'wb');

• sudo tail -f -n100 /opt/karaf/data/log/karaf.log
claw.dev:80 127.0.0.1 - - [08/Sep/2021:20:47:30 +0000] "PUT /node/50355/media/extracted_text/500 HTTP/1.1" 400 79898 "-" "Apache-HttpClient/4.5.3 (Java/1.8.0_292)"

[kayakr]

• Has this happened to you? If so, please share details of PHP versions, code versions, environment, etc.
• I wonder if this is unique to Playbook VMs and the update journey from PHP 7.2 to PHP 7.4

[seth-shaw-unlv]

In case this point of reference helps: I have not seen this happen. I'm on PHP 7.4, but not Drupal 9.

[adam-vessey]

Not touching Ansible; however, it appears we've encountered it (or something extremely) similar. Appears to be related to interactions between chunked encoding and how PHP's being run:

• https://stackoverflow.com/a/30924257
• https://stackoverflow.com/a/68156157
• https://bz.apache.org/bugzilla/show_bug.cgi?id=57087

So, Crayfish generates the derivative, and streams it back to the connector, the derivative connector PUTs it back to Drupal stream-wise (not knowing the exact length, and so using chunked encoding), and gets into this situation with the bogus length of 0.

[kayakr]

@adam-vessey Thanks for the links; the handling (or lack of handling) of transfer-encoding: chunked in Apache via FPM/FastCGI seems very relevant. However, in my VM, I updated to Apache/2.4.48 and added SetEnv proxy-sendcl 1 to /etc/apache2/apache2.conf and restarted fpm, but I still get 0 bytes copied.
The transfer-encoding: chunked header is gone, and header content-length:0 is added.

[kayakr]

Hmm, do I'd actually need to run Apache as a proxy to spool the chunked transfer and generate a valid content-length. Might be easier to try nginx instead...
https://stackoverflow.com/questions/54444220/php-7-fpm-does-not-show-request-body-with-chunked-encoding

[kstapelfeldt]

Problem needs to be addressed in ISLE, maybe

[kayakr]

I resolved this on my local VM by reverting the switch to PHP FPM in favour of Apache 2 Handler.

• sudo a2dismod proxy_fcgi
• sudo a2disconf sudo a2disconf php7.4-fpm
• sudo a2enmod php7.4
• sudo systemctl reload apache2
  Now derivatives are working.
  @kstapelfeldt AFAIK isle-dc uses nginx so may not be affected by this Apache + php-fpm issue.

[noahwsmith]

@dannylamb is going to look into this for us, Whitman and JHU, likely next week. The resulting fix will be made in the community version of ISLE as well as the individual projects in question.

[rosiel]

I'm still getting this in a fresh playbook. Is 'switching to PHP FPM' a solution that could be baked into the playbook?

[whikloj]

We ran into and (unfortunately) didn't find this issue for a couple days.

My understanding is that Apache will spool the content to calculate the length if required, so if you are using a patched version of Apache (>= 2.4.47) and have SetEnv proxy-sendcl 1 then this issue should be resolved.

Our campus IT is spinning us a RHEL 9 box with 2.4.53 which we will attempt Apache with php-fpm on again. Will report back.