Return Path bounce processing

[LotisHunters]

(This is my first time opening an issue/writing a bug report on GitHub, so I apologize for the format of this or any missing information. I can provide any additional information if needed)

Description of issue

According to the docs here:
https://docs.postalserver.io/other/auto-responders-and-bounces

Specifying the returnpath route is meant to allow for routing of bounces that don't match any outbound messages.
"Once added, messages sent to your return path that aren't detected as bounces will be sent to HTTP endpoint you choose."

When specifying the returnpath route, those messages properly get sent to my webhook which is great. However, according to the next section, bounce messages that can be linked to an outgoing message should not be delivered to that return path and should instead be mapped as bounces.
"A note about bounces
Messages that Postal detects as being bounces for a message you have already sent will not be delivered to your return path route. The original message will be updated and a MessageBounced webhook event will be triggered."

When I specify the returnpath route, all bounce messages sent to the return path address (including bounce messages that can be mapped to outgoing messages) get delivered to my webhook, meaning that bounce message processing in Postal stops working entirely.

Expected behavior

According to the docs, specifying the returnpath route should only match the messages that can't be detected as a bounce for an outgoing message.

Environment details

• OS: Ubuntu 22.04
• Version 2.1.2

[willpower232]

Does this not mean that all your bounces happened relatively quickly so that Postal was able to identify which messages they were related to?

[LotisHunters]

Does this not mean that all your bounces happened relatively quickly so that Postal was able to identify which messages they were related to?

I have bounces that come in very quickly, and some that come in hours later.
Often it's the ones that come in hours later that just hard fail since they can't be matched to an outgoing message.

The way I identify bounces is by going to outgoing messages and filtering by "status: bounced"
Before enabling the returnpath route, there were bounced messages being identified every couple of minutes.
After enabling the returnpath route, there hasn't been a single outgoing message identified as "bounced".
In that timeframe, my webhook for receiving the returnpath messages has received a LOT of bounces.

Before setting up the returnpath route, about 1/4th of the bounces would hard fail since they couldn't be mapped to outgoing messages. They seem very important to process so that's why I'm trying to route just the unmatchable bounces that were sent to the returnpath for manual review/external automation.
Now that I have the webhook set up, every single bounce (mappable or not) going to the return path are getting sent to my webhook, and my application that was set up to watch for mapped bounces is no longer getting anything.

[LotisHunters]

I just disabled the returnpath route and set up a webhook endpoint for the MessageBounced event.
Doing this makes it so that bounces are still mapped correctly (I saw a few come in and get mapped) and it sent a webhook to my endpoint.
Unmapped bounces hard failed, which is expected since there is no returnpath route.

I can get just mapped bounces to route to my endpoint, but there doesn't seem to be a way to properly get just unmapped bounces to route to an endpoint without impacting all bounce processing

[chrisdeeming]

@LotisHunters hope you're doing okay.

I think I'm experiencing the same issue as you except from my perspective, what I want to see is MessageBounced events sent to my webhook.

Messages that Postal detects as being bounces for a message you have already sent will not be delivered to your return path route. The original message will be updated and a MessageBounced webhook event will be triggered.

I did have a return path route, which I have now deleted.

I have an outgoing email which is bouncing after a few minutes of being sent. In the activity log for the original outgoing email I get this:

And the incoming email is being processed:

But I do not get a webhook.

Am I correct in saying I should get a webhook?

My webhook is set up correctly and is receiving MessageDeliveryFailed events but I never get any MessageBounced events.

Does anyone know why this might be?

[chrisdeeming]

Here's what I believe is the relevant logs while the bounce happens. This looks like an error but I'm not certain.

worker-1    | [8] [2024-01-15T15:24:40.053] INFO -- : [XAC0LUSMDM] [1::3851 3883] Bounce linked with message 3726
worker-1    | [8] [2024-01-15T15:24:40.054] INFO -- : [GELL2NGHSY] Started processing SendWebhookJob job
worker-1    | [8] [2024-01-15T15:24:40.109] INFO -- : [XAC0LUSMDM] Finished processing UnqueueMessageJob job in 0.06937057s
worker-1    | [8] [2024-01-15T15:24:40.112] WARN -- : [GELL2NGHSY] Psych::DisallowedClass: Tried to load unspecified class: ActiveSupport::HashWithIndifferentAccess
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/class_loader.rb:99:in `find'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/class_loader.rb:28:in `load'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/to_ruby.rb:426:in `resolve_class'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/to_ruby.rb:290:in `visit_Psych_Nodes_Mapping'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/visitor.rb:30:in `visit'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/visitor.rb:6:in `accept'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/to_ruby.rb:35:in `accept'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/to_ruby.rb:320:in `visit_Psych_Nodes_Document'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/visitor.rb:30:in `visit'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/visitor.rb:6:in `accept'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych/visitors/to_ruby.rb:35:in `accept'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/lib/ruby/3.2.0/psych.rb:334:in `safe_load'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/coders/yaml_column.rb:53:in `yaml_load'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/coders/yaml_column.rb:26:in `load'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/type/serialized.rb:22:in `deserialize'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/type/helpers/mutable.rb:8:in `cast'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute.rb:175:in `type_cast'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute.rb:43:in `value'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute.rb:155:in `changed_from_assignment?'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute.rb:60:in `changed?'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute_mutation_tracker.rb:79:in `attribute_changed?'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute_mutation_tracker.rb:45:in `changed?'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute_mutation_tracker.rb:15:in `block in changed_attribute_names'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute_mutation_tracker.rb:15:in `select'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activemodel-6.1.7.3/lib/active_model/attribute_mutation_tracker.rb:15:in `changed_attribute_names'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/attribute_methods/dirty.rb:145:in `changed_attribute_names_to_save'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/associations/association.rb:197:in `initialize_attributes'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/associations/association.rb:322:in `block in build_record'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/core.rb:527:in `initialize'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/inheritance.rb:72:in `new'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/inheritance.rb:72:in `new'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/reflection.rb:157:in `build_association'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/associations/association.rb:321:in `build_record'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/associations/collection_association.rb:354:in `_create_record'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/associations/has_many_association.rb:137:in `_create_record'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/associations/association.rb:209:in `create!'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/associations/collection_proxy.rb:364:in `create!'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /opt/postal/app/app/models/webhook_request.rb:41:in `block in trigger'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/relation/delegation.rb:88:in `each'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/activerecord-6.1.7.3/lib/active_record/relation/delegation.rb:88:in `each'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /opt/postal/app/app/models/webhook_request.rb:40:in `trigger'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /opt/postal/app/app/jobs/send_webhook_job.rb:22:in `perform'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /opt/postal/app/lib/postal/worker.rb:64:in `receive_job'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /opt/postal/app/lib/postal/worker.rb:96:in `block in join_queue'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/bunny-2.20.3/lib/bunny/consumer.rb:56:in `call'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/bunny-2.20.3/lib/bunny/channel.rb:1842:in `block in handle_frameset'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/bunny-2.20.3/lib/bunny/consumer_work_pool.rb:108:in `block (2 levels) in run_loop'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/bunny-2.20.3/lib/bunny/consumer_work_pool.rb:103:in `loop'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/bunny-2.20.3/lib/bunny/consumer_work_pool.rb:103:in `block in run_loop'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/bunny-2.20.3/lib/bunny/consumer_work_pool.rb:102:in `catch'
worker-1    | [8] [2024-01-15T15:24:40.113] WARN -- : [GELL2NGHSY]    /usr/local/bundle/gems/bunny-2.20.3/lib/bunny/consumer_work_pool.rb:102:in `run_loop'
worker-1    | [8] [2024-01-15T15:24:40.113] INFO -- : [GELL2NGHSY] Finished processing SendWebhookJob job in 0.059291199s

[chrisdeeming]

My issue is related to this, it seems:
#2477