IMAP 'UID FETCH' response parsing error

[BasTossings]

I am currently investigating a serious IMAP issue that seems to plague many email clients that indirectly utilize libetpan through MailCore2, including my installation of Mailspring.

The specific issue in Mailspring is mentioned here: The Eternal Sync Bug. But i've read other mail clients utilizing MailCore2 have similar problems.

In my case, if fails when fetching older messages from my iCloud mail account using UID FETCH, for example:

10 UID FETCH 11153:11156,[redacted for brevity] (UID FLAGS ENVELOPE INTERNALDATE BODY.PEEK[HEADER.FIELDS (References)])

To which the server responds with:

* 7027 FETCH (UID 11153 MODSEQ (366744470000000) FLAGS (\Seen) ENVELOPE ("Tue, 30 Jan 2018 23:39:12 +0100 (CET)" "Redacted" (("Redacted" NIL "automatisch" "Redacted")) (("Redacted" NIL "automatisch" "Redacted")) (("Redacted" NIL "automatisch" "Redacted")) ((NIL NIL "redacted" "me.com")) NIL NIL NIL "<700089605.6840893.1517351952703.JavaMail.www-data@mp-be012>") INTERNALDATE "30-Jan-2018 22:39:15 +0000" BODY[HEADER.FIELDS (References)] {0}
)
* 7028 FETCH (UID 11154 MODSEQ (366744470000000) FLAGS (\Seen) ENVELOPE ("Tue, 30 Jan 2018 22:55:59 +0000 (UTC)" "Redacted" (("Redacted" NIL "info" "redacted")) ((NIL NIL "info" "redacted.com")) (("Redacted" NIL "info" "redacted")) ((NIL NIL "redacted" "me.com")) NIL NIL NIL "<783557266.1517352959090.JavaMail.root@8d68e8bf002b>") INTERNALDATE "30-Jan-2018 22:56:03 +0000" BODY[HEADER.FIELDS (References)] {0}
)
[redacted for brevity]
10 OK UID FETCH completed

The parser faults on the output at mailimap_response_tagged_parse with a MAILIMAP_ERROR_PARSE because it does not recogize the first character of the response (*) as a tag, leading to mailimap_uid_fetch also returning with a MAILIMAP_ERROR_PARSE.

The call stack at that point is:

mailimap_tag_parse <-- fails here
mailimap_response_tagged_parse
mailimap_response_done_parse
mailimap_response_parse_progress
mailimap_response_parse_with_context
mailimap_parse_response
mailimap_uid_fetch_qresync_vanished
mailimap_uid_fetch_changedsince
mailimap_uid_fetch

According to the RFC (at 6.4.8), UID FETCH has an untagged response, so I am assuming the reason libetpan calls mailimap_response_tagged_parse in this case is just to parse the status response at the end (in this case 10 OK UID FETCH), assuming the untagged response has already been parsed.

Now, in mailimap_response_parse_progress, before mailimap_response_done_parse is called, it calls mailimap_struct_multiple_parse_progress, which returns MAILIMAP_NO_ERROR, yet does not seem to parse any untagged data (or any data for that matter).

I will continue to investigate further, but any assistance would be greatly apprectiated!

[BasTossings]

Ok, some progress. I'm now looking at mailimap_msg_att_parse_progress where it tries to parse the message after the message sequence number and FETCH identifier (* 7027 FETCH) have been successfully parsed.

The call stack is now:

mailimap_msg_att_parse_progress
mailimap_message_data_parse_progress
mailimap_response_data_parse_progress
mailimap_cont_req_or_resp_data_parse_progress
mailimap_struct_multiple_parse_progress
mailimap_response_parse_progress
mailimap_response_parse_with_context
mailimap_parse_response
mailimap_uid_fetch_qresync_vanished
mailimap_uid_fetch_changedsince
mailimap_uid_fetch

It passes mailimap_oparenth_parse successfully, then calls mailimap_struct_spaced_list_parse_progress and that too returns with no error.

However, it now expects a closing parenthesis ) when the stream is still at ENVELOPE (, and this is where it fails. So it seems ENVELOPE is unsupported at this point.

I'll dive deeper into mailimap_struct_list_parse_progress next.

[BasTossings]

And more progress: it seems to choke on an ENVELOPE's messageid containing quotes:

"<"392889836.11.1529401004417.JavaMail.Redacted"@Redacted>"

mailimap_env_message_id_parse uses mailimap_nstring_parse which encounters the first " after the string opens and concludes that that must be the end of the string and therefore the messageid.

Consequently, mailimap_cparenth_parse fails because the next character in the stream is not a ) concluding the ENVELOPE but the continuation of the messageid, namely <

[BasTossings]

Looking at the RFC, the formal syntax does not seem to allow DQUOTE's in the envelope messageid (except when properly escaped, which in this case, they are not):

env-message-id  = nstring
nstring         = string / nil
nil             = "NIL"
string          = quoted / literal
literal         = "{" number "}" CRLF *CHAR8 ; Number represents the number of CHAR8s
quoted          = DQUOTE *QUOTED-CHAR DQUOTE
QUOTED-CHAR     = <any TEXT-CHAR except quoted-specials> / "\" quoted-specials
TEXT-CHAR       = <any CHAR except CR and LF>
quoted-specials = DQUOTE / "\"

But alas, here we are.

[BasTossings]

@dinhvh if you could add me as a contributor I can make a PR with a workaround

[BasTossings]

FYI: I have also created an issue on Apple’s developer site since it’s an iCloud issue and eventually, Apple should fix it on their side:

iCloud IMAP server bug

[dsanghan]

@BasTossings could you please share the fix?

[BasTossings]

Hey @dsanghan, the fix consists of a number of changes in "src/low-level/imap/mailimap_parser.c".

I'd rather just push my branch but here it is anyways: mailimap_parser.zip