request#show: incompatible character encodings: ASCII-8BIT and UTF-8

[laurentS]

This seems very similar to #5783 but hits a different part of the code base.

I've been flooded with crash logs like below, related to one single request.
It seems that the incoming message has an unusual encoding, which resulted in

FoiAttachment.find("the_id").unmasked_body.encoding
  Disk Storage (3.2ms) Downloaded file from key: y98h2zagecn1nsrp7cuddv7eub88
=> #<Encoding:ASCII-8BIT>

This in turn caused the censor rule method below to crash:

  app/models/censor_rule.rb:61:in `gsub'
  app/models/censor_rule.rb:61:in `apply_to_text'
  lib/themes/dada-france-theme/lib/model_patches.rb:552:in `block in apply_text_masks'
  lib/themes/dada-france-theme/lib/model_patches.rb:552:in `each'
  lib/themes/dada-france-theme/lib/model_patches.rb:552:in `reduce'
  lib/themes/dada-france-theme/lib/model_patches.rb:552:in `apply_text_masks'
  lib/alaveteli_text_masker.rb:41:in `apply_masks'
  app/jobs/foi_attachment_mask_job.rb:37:in `mask'
  app/jobs/foi_attachment_mask_job.rb:20:in `perform'
  app/jobs/foi_attachment_mask_job/uniqueness.rb:19:in `perform_once_now'
  app/models/foi_attachment.rb:110:in `body'
  app/models/foi_attachment.rb:121:in `body_as_text'
  app/models/incoming_message.rb:358:in `_convert_part_body_to_text'
  app/models/incoming_message.rb:349:in `get_main_body_text_internal'
  lib/themes/dada-france-theme/lib/model_patches.rb:386:in `custom_text_for_admins'
  lib/themes/dada-france-theme/lib/model_patches.rb:410:in `custom_text_unfolded_for_admins'
  lib/themes/dada-france-theme/lib/model_patches.rb:432:in `get_main_body_text_unfolded'
  lib/themes/dada-france-theme/lib/model_patches.rb:450:in `get_body_for_html_display'
  lib/themes/dada-france-theme/lib/views/request/_attachments.html.erb:54
  lib/themes/dada-france-theme/lib/views/request/_attachments.html.erb:46
  lib/themes/dada-france-theme/lib/views/request/_incoming_correspondence.html.erb:32
  app/views/request/events/_response.html.erb:1
  app/views/request/_correspondence.html.erb:5
  lib/themes/dada-france-theme/lib/views/request/show.html.erb:42
  lib/themes/dada-france-theme/lib/views/request/show.html.erb:40
  app/controllers/request_controller.rb:100:in `block (3 levels) in show'
  app/controllers/request_controller.rb:97:in `block in show'
  lib/alaveteli_localization.rb:46:in `with_locale'
  app/controllers/request_controller.rb:50:in `show'
  lib/themes/dada-france-theme/lib/controller_patches.rb:211:in `show'
  app/controllers/application_controller.rb:116:in `record_memory'
  lib/deeply_nested_params.rb:15:in `call'
  lib/strip_empty_sessions.rb:14:in `call'

I fixed the crash by changing CensorRule.apply_to_text as follows:

  def apply_to_text(text_to_censor)
    return nil if text_to_censor.nil?

    text_to_censor.gsub(
		to_replace(text_to_censor.encoding),
		replacement.dup.force_encoding(text_to_censor.encoding)
	)
  end

I have tested manually by checking a variety of requests, and this seems to work, but I'd welcome a more thorough testing approach on this :)

Despite the presence of my patches in the stack trace, it does not seem to be the cause of the crash, these patched parts only modify which censor rules are applied (more specifically, they disable the rule for email addresses for the request author and site admins).

Disclaimer 2: this bug has only appeared on 1 incoming message out of ~15k, so I don't know if it's worth putting much more effort into it (as long as I haven't made things worse with my patch!).