Possible to have duplicate email addresses in merge_data with different fields?

[chrisgrande]

Normally I would construct the merge_data as a dictionary like so:

For example I'm getting a values list like this:
{('Bob', 'bob@example.com', '3Z3HRJG'), {('Annie', 'annie@example.com', '3ZHSN76'), ('Bob', 'bob@example.com', 'KSB76GH')

And converting them to a dict in the right format for merge_data:

buyers = {}
for buyer in self.buyers:
    buyers[buyer[1]] = {'name': buyer[0], 'order_code': buyer[2]}
return buyers

However in this instance I have duplicate emails due to different orders and need each one to get an email with different metadata but basically the same merge_global_data. Is this not an instance where I can use a batch send operation and should send each email separately? I'm using Postmark as my ESP.

Thank you for any assistance! 😃

[medmunds]

Short answer: no, if you want to send two messages with different merge data to the same email address, you'll need to do that as two separate sends, not a single batch send.

Longer answer: this is a limitation in Anymail's batch sending. Anymail's merge_data is structured as a dict keyed by recipient email, which means each recipient address can only appear once in it. (Unless you play games with case sensitivity, which, please don't. BTW, Anymail inherited the merge_data dict approach from the predecessor Djrill project.)

Postmark's batch send API (and it looks like every other ESP except Mailgun) actually allows the same recipient to appear multiple times with different data, generating multiple messages to the same recipient. But there's not a way to communicate that to Anymail through merge_data. And Anymail's esp_extra is the usual "escape hatch" to get to raw ESP functionality, but I don't think it can be used to work around this.

Since repeat batch recipients are so widely supported, it might be nice to add this to Anymail. It's a little tricky, because Django's EmailMessage object isn't really a great match for describing a batch send. (Really, a batch should take a list of EmailMessage objects, but that's a whole other can of worms.) If we stick with the EmailMessage's to field providing the list of batch recipients, then maybe merge_data could optionally be a list of data dicts in the same order as to? Or [(to_email, merge_data), ...] pairs? Or a MultiValueDict? Open to ideas.

[chrisgrande]

Thank you for the response — I had the "come to code" moment with the dict structure and it just not seeming possible with how merge_data works. I didn't think of the unholy case sensitivity, don't worry, I won't poke that.

As for the future of the functionality, with all of the machinery using the EmailMessage fields to generate the recipients list, and if you wanted to stick with that as the source of "truth" for batch messages, using a MultiValueDict might make the most sense. With each addresses additional metadata as a list of dicts on the single key.

Speaking of cans of worms, it seems like when using merge_data or some new successor it would make sense to not even take the to field into account and just use that merge data structure as the source of truth. Maybe some new MultiValueDict batch_message_data that can have it's own path to get a list of EmailAddress objects to pass on to the rest of the system.

[medmunds]

it would make sense to not even take the to field into account and just use that [new successor] structure as the source of truth

Yep. Yes. Yes!

I'm increasingly convinced that the current "merge_data converts a single EmailMessage into a batch send" is… just not the optimal design for batch sending. (It's confusing. It necessitates sticking orange "Caution" boxes in the docs. It leads to some really scary, hard-to-maintain backend code with novella-length comments.)

So I'm inclined to not try to shoehorn in support for repeat batch recipients on top of the existing merge_data, and instead start working towards a separate, new send_batch_email(message_list) API that would eventually replace (deprecate) merge_data. Like a richer version of Django's send_mass_mail(). Embrace those canned worms.