[HOLD for payment 2023-09-07] [$4000] Inconsistent front-end email validation across the product

[kavimuru]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Action Performed:

(Chat)

1. Create a new chat
2. Enter an invalid email (e.g. jaj@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdicdjejajasjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfisjksksjsjssskssjskskssksksksksskdkddkddkdksskskdkdkdksskskskdkdkdkdkekeekdkddenejeodxkdndekkdjddkeemdjxkdenendkdjddekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdicdjejajasjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdi.cdjd
3. Start the chat

(Login/SignUp)

1. Try to login with an invalid email like jaj@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdicdjejajasjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfisjksksjsjssskssjskskssksksksksskdkddkddkdksskskdkdkdksskskskdkdkdkdkekeekdkddenejeodxkdndekkdjddkeemdjxkdenendkdjddekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdicdjejajasjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdi.cdjd

(Add Contact Method)

1. Navigate to Settings > Profile > Contact Method > New Contact Method
2. Try to add an invalid email like jaj@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdicdjejajasjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfisjksksjsjssskssjskskssksksksksskdkddkddkdksskskdkdkdksskskskdkdkdkdkekeekdkddenejeodxkdndekkdjddkeemdjxkdenendkdjddekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdicdjejajasjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdi.cdjd

Expected result:

Front-end validation should prevent these emails from being added and creating unnecessary network requests. Additionally, consistent validation should be used in all areas where we validate emails in the product (e.g. pasting an invalid email into an existing chat still formats it like a valid email)

Backend email validation uses PHP's email validation filter FILTER_VALIDATE_EMAIL as well as the following regex (for emails and phone numbers) [\w.'#%+-]+@[a-zA-Z\d.-]+.[a-zA-Z]{2,}|^+?[1-9]\d{1,14}$)

Actual results:

Emails are successfully submitted, only to be subsequently rejected by the backend

Workaround:

N/A

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android / native
• Android / Chrome
• iOS / native
• iOS / Safari
• MacOS / Chrome / Safari
• MacOS / Desktop

Version Number: v1.3.0-0
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation

bandicam.2023-04-13.13-58-25-321.mp4

Expensify/Expensify Issue URL:
Issue reported by: @chiragxarora
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1681358100970919

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01878e05ad15a14027
• Upwork Job ID: 1648410114624286720
• Last Price Increase: 2023-08-10
• Automatic offers:
• aimane-chnaif | Reviewer | 26133961
• jjcoffee | Contributor | 26133964

[MelvinBot]

Triggered auto assignment to @adelekennedy (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[MelvinBot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[MelvinBot]

@adelekennedy Whoops! This issue is 2 days overdue. Let's get this updated quick!

[adelekennedy]

I can reproduce! Seems very similar to #17388

[MelvinBot]

Job added to Upwork: https://www.upwork.com/jobs/~01878e05ad15a14027

[MelvinBot]

Current assignee @adelekennedy is eligible for the External assigner, not assigning anyone new.

[MelvinBot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @aimane-chnaif (External)

[MelvinBot]

Triggered auto assignment to @NikkiWines (External), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[chiragxarora]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Invalid email validation in chats, leading to the wrong strings markdowns being transformed to email anchor tags by using mailto: protocol.

After the transformation, when we click the link rendered in chats, it also opens that invalid email in external email app.

What is the root cause of that problem?

Root cause of the problem is wrong regex applied in expensify's helper library called Expensify-common. Here the regex expressions are defined in the file lib/CONST.jsx/ as

https://github.com/Expensify/expensify-common/blob/777177937a77a0edfea5bdba2252eff35da9f9db/lib/CONST.jsx#L333

Since a valid email can have at most 254 characters, this regex failed to put a limit on length of total string which is causing the problem

What changes do you think we should make in order to solve the problem?

We should add a clause to the regex above which can limit the total length of valid email.
We can do this by adding this additional condition (?=.{1,254}$)

After adding the length condition:

Before that(in production right now):

Thankyou!

EDIT: grammar
PS: I'm sorry if something's off in the detailing of proposal, this is my first proposal to Expensify

[jjcoffee]

Proposal

Please re-state the problem that we are trying to solve in this issue.

We want email validation to be consistent across the app and between FE and BE.

What is the root cause of that problem?

We are using two different sets of regex patterns on the FE, neither of which fully matches the behaviour of the BE. The one that is used when validating emails when starting a new chat can be traced back to EMAIL_BASE_REGEX:

https://github.com/Expensify/expensify-common/blob/e93e1eb448ad6bdbde911fd6239f70d5e749635e/lib/CONST.jsx#L3

and then there's the markdown one here, which eventually is used to converts emails to links in-chat:

https://github.com/Expensify/expensify-common/blob/e93e1eb448ad6bdbde911fd6239f70d5e749635e/lib/CONST.jsx#L333

This duplicating of regexes appears to be a result of this PR.

What changes do you think we should make in order to solve the problem?

I think it makes sense to have one master rule which fully matches BE behaviour, which would be used both for starting chats and for figuring out when a string in a message needs to be converted to an email link. This just keeps things simple going forwards, but I can see there could be an argument for looser constraints for the chat markdown conversion (as it's less critical if it gets it wrong).

The BE uses a combination of FILTER_VALIDATE_EMAIL and additional regex ([\w\.'#%+-]+@[a-zA-Z\d\.-]+\.[a-zA-Z]{2,}) that is more restrictive than the PHP filter in terms of which characters it accepts.

If we look at the regex of FILTER_VALIDATE_EMAIL extracted from PHP source, we can see that it does a mixture of character validation and various lengths. To summarise it will validate:

• Overall length of 254 chars
• Address part (before the @) is limited to 64 chars
• The domain part as having 63 chars in each label (parts between the dots .)
• The domain part can't start or end in a hyphen (e.g. @example-.com or @-example.com)

We don't need to care about the details of the filter's character validation as the additional regex the BE uses is more restrictive (e.g. BE won't accept "{" as a valid character, whereas the PHP filter would), so we just want to apply the BE's additional regex when it comes to which characters we accept.

In order to match all the above behaviour, I think we should update EMAIL_BASE_REGEX to this:

New regex
(?=((?=[\\w'#%+-]+(?:\\.[\\w'#%+-]+)*@)[\\w\\.'#%+-]{1,64}@(?:(?=[a-z\\d]+(?:-+[a-z\\d]+)*\\.)(?:[a-zA-Z\\d-]{1,63}\\.)+[a-zA-Z]{2,63})(?= |_|\\b))(?<end>.*))\\S{3,254}(?=\\k<end>$)

This combines the length limits in FILTER_VALIDATE_EMAIL and the character limits in the extra BE regex (as well as the TLD being >2 length), so this will fully match the BE's behaviour as described above.

The MARKDOWN_EMAIL constant should be updated to refer to the base regex: MARKDOWN_EMAIL: EMAIL_BASE_REGEX (or it can be removed and references updated to EMAIL_BASE_REGEX).

The autoEmail rule needs to be updated to this:

(?![^<]*>|[^<>]*<\\/)(?:(?<= |[^\\w'#%+-])|^)${CONST.REG_EXP.MARKDOWN_EMAIL}(?![^<]*(<\\/pre>|<\\/code>|<\\/a>))

This ensures that the length count starts at either whitespace, start of line or non-email character. Without this you'd get the part of the email within the length limit converted and the rest left as plain text.

The BE considers _example@test.com a valid email which conflicts a bit with markdown italic underscores, so dealing with that requires an extra processing step. See below for the fix.

Show fix for emails with leading underscores

It's easy enough to handle simple cases like _example@test.com_ (just match the email within the underscores by consuming any matching start/end underscores), but it gets more complex for multi-line mixed examples, e.g.

_email@test.com is my main email, you can also try _email2@test.com

or:
_email3@test.com_

where it becomes less clear which parts should be italicised and we can't simply handle this in one regex lookup.

The easiest solution is just not to accept leading underscores in markdown since it's really rare to actually see an email starting with an underscore. Slack gets around this by just ignoring leading underscores, for example:

However if we want to fix it, we need to add a process callback to the email rule, like this:

process: (textToProcess, replacement) => {
    const emailRegex = new RegExp(
        `(?![^<]*>|[^<>]*<\\/)(?:\\b)${CONST.REG_EXP.MARKDOWN_EMAIL}(?![^<]*(<\\/pre>|<\\/code>|<\\/a>))`,
        'gim'
    );
    return this.modifyTextForEmails(emailRegex, textToProcess, replacement);
},
replacement: '<a href="mailto:$1">$1</a>',

This callback will run the italic markdown rule to find any italic markdown underscores, and then run the email replace rule within the italics, before running the email replace on the rest of the text.

We can move the existing markdown italic regex into a constant at the top of ExpensiMark.js:

const MARKDOWN_ITALICS_REGEX = new RegExp(/(\b_+|\b)(?!_blank")_((?![\s_])[\s\S]*?[^\s_])_(?![^\W_])(?![^<]*(<\/pre>|<\/code>|<\/a>|<\/mention-user>|_blank))/g);

Then the callback function is:

modifyTextForEmails(emailRegex, textToCheck, replacement) {
    // Replace emails within italic markdown underscores first
    const italicRegex = MARKDOWN_ITALICS_REGEX;
    let match = italicRegex.exec(textToCheck);
    let replacedText = '';
    let startIndex = 0;

    while (match !== null) {
        // `match[1]` contains additionally captured leading underscores which are valid email chars
        // `match[2]` contains the text within the underscores (markdown italics).
        const italicText = `${match[1]}${match[2]}`;

        // We only run the email replace if there are emails within the markdown italics.
        if (emailRegex.test(italicText)) {
            // MARKDOWN_ITALICS_REGEX captures any extra leading underscores, so we compare match group lengths to
            // determine the index of the first underscore, taking into account the closing underscore.
            const offset = match.index + (match[0].length - italicText.length - 1);

            replacedText = replacedText.concat(textToCheck.substr(startIndex, (offset - startIndex)));

            const emailText = italicText.replace(emailRegex, replacement);

            replacedText = replacedText.concat(emailText);
            startIndex = match.index + (italicText.length + 1);
        } else {
            replacedText = replacedText.concat(textToCheck.substr(startIndex, ((match.index + match[0].length) - startIndex)));
            startIndex = match.index + (match[0].length);
        }

        // Now we move to the next match that the js regex found in the text
        match = italicRegex.exec(textToCheck);
    }
    if (startIndex < textToCheck.length) {
        replacedText = replacedText.concat(textToCheck.substr(startIndex));
    }

    // Now we run the normal email regex on the rest
    replacedText = replacedText.replace(emailRegex, replacement);

    return replacedText;
}

Tests

All the following test suite emails can be validated against the PHP filter + BE regex in this quick PHP script I threw together.

Fixes to existing test suites

There are several tests that need to be removed as they test for emails that the BE considers invalid. The following email patterns need to be removed from this test:

• a~b@gmail.com
• a*b@gmail.com
• a~*_b@gmail.com

There are also several tests that include the following invalid emails in the same file:

• concierge?@expensify.com
• concierge!@expensify.com
• concierge1?@expensify.com

We would also need to modify the result of this test as _abc@gmail.com would be considered a valid email by BE whereas def@gmail.com_ is not so the correct result is:

result = '<a href="mailto:_abc@gmail.com">text</a><br />'
        + '[text](<a href="mailto:def@gmail.com">def@gmail.com</a>_)';

This test also needs to be updated toBeFalsy as $test@gmail.com is an invalid email according to the BE.

New test cases

Several new tests need to be added to Str-test.js. These can replace Str.isValidEmailMarkdown tests as these two functions run exactly the same regex now. Proposed tests are below. They can be validated against PHP + the BE's regex in this PHP script.

Show Str-test.js new tests

describe('Str.isValidEmail', () => {
    it('Correctly identifies valid emails', () => {
        expect(Str.isValidEmail('abc@gmail.com')).toBeTruthy();

        // Domain length (63 chars in each label)
        expect(Str.isValidEmail('test@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com')).toBeTruthy();
        expect(Str.isValidEmail('abc@abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.km')).toBeTruthy();
        expect(Str.isValidEmail('abc@co.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.km')).toBeTruthy();

        // Address length (64 chars)
        expect(Str.isValidEmail('sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@test.com')).toBeTruthy();

        // Overall length (254 chars)
        expect(Str.isValidEmail('averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com')).toBeTruthy();

        // Domain with lots of dashes
        expect(Str.isValidEmail('sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asj-j-s-sjdjdjdjd-jdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdke.com.ab.net.aa.bb.cc.dd.ee')).toBeTruthy();
        expect(Str.isValidEmail('abc@g---m--ai-l.com')).toBeTruthy();

        // Domain with repeated labels of 63 chars
        expect(Str.isValidEmail('test@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.com')).toBeTruthy();

        // TLD >=2 chars
        expect(Str.isValidEmail('abc@gmail.co')).toBeTruthy();
        expect(Str.isValidEmail('a@a.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijk')).toBeTruthy();

        // Very short address
        expect(Str.isValidEmail('a@example.com')).toBeTruthy();

        // xn-- style domain name
        expect(Str.isValidEmail('test@xn--diseolatinoamericano-76b.com')).toBeTruthy();

        // Unusual but valid prefixes
        expect(Str.isValidEmail('-test@example.com')).toBeTruthy();
        expect(Str.isValidEmail('_test@example.com')).toBeTruthy();
        expect(Str.isValidEmail('#test@example.com')).toBeTruthy();
        expect(Str.isValidEmail('test.+123@example.com')).toBeTruthy();
        expect(Str.isValidEmail('-test-@example.com')).toBeTruthy();

        // Invalid chars
        expect(Str.isValidEmail('$test@gmail.com')).toBeFalsy();
        expect(Str.isValidEmail('!test@gmail.com')).toBeFalsy();
        expect(Str.isValidEmail('"test"@gmail.com')).toBeFalsy();
        expect(Str.isValidEmail('~abc@gmail.com~')).toBeFalsy();
        expect(Str.isValidEmail('abc@gmail.com~')).toBeFalsy();
        expect(Str.isValidEmail('test@example_123site.com')).toBeFalsy();
        expect(Str.isValidEmail('test{@example.com')).toBeFalsy();
        expect(Str.isValidEmail('test..new@example.com')).toBeFalsy();
        expect(Str.isValidEmail('test@example-.a.com')).toBeFalsy();
        expect(Str.isValidEmail('test@example......a.com')).toBeFalsy();

        // Invalid period location
        expect(Str.isValidEmail('.test@example.com')).toBeFalsy();
        expect(Str.isValidEmail('.test.new@example.com')).toBeFalsy();
        expect(Str.isValidEmail('test.@example.com')).toBeFalsy();

        // Domain too long (>63 chars in each label)
        expect(Str.isValidEmail('test@averylongdomainpartoftheemailthatwillgooverthelimitasitismorethan63chars.com')).toBeFalsy();
        expect(Str.isValidEmail('abc@abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890a.km')).toBeFalsy();
        expect(Str.isValidEmail('abc@co.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890a.km')).toBeFalsy();

        // Address too long (>64 chars)
        expect(Str.isValidEmail('averylongaddresspartoftheemailthatwillgovoerthelimitasitismorethan64chars@example.com')).toBeFalsy();

        // Overall length too long
        expect(Str.isValidEmail('sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com.a.aa.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjasfffa')).toBeFalsy();

        // Incorrect domains start/end
        expect(Str.isValidEmail('test@example-.com')).toBeFalsy();
        expect(Str.isValidEmail('test@-example-.com')).toBeFalsy();

        // TLD too short
        expect(Str.isValidEmail('test@example.a')).toBeFalsy();

        // TLD too long
        expect(Str.isValidEmail('a@a.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijkl')).toBeFalsy();
    });
});

We can also add similar tests to ExpensiMark-HTML-test.js, as well as a new test for the more complex italic markdown underscore case mentioned above:

Show ExpensiMark-HTML-test.js new tests

// Check emails within other markdown
test('Test emails within other markdown', () => {
    const testString = '> test@example.com\n'
    + '```test@example.com```\n'
    + '`test@example.com`\n'
    + '_test@example.com_ '
    + '__test@example.com_';
    const result = '<blockquote><a href="mailto:test@example.com">test@example.com</a></blockquote>'
    + '<pre>test@example.com</pre>'
    + '<code>test@example.com</code><br />'
    + '<em><a href="mailto:test@example.com">test@example.com</a></em> '
    + '<em><a href="mailto:_test@example.com">_test@example.com</a></em>';
    expect(parser.replace(testString)).toBe(result);
});

// Check email regex's validity at various limits
test('Test markdown replacement for valid emails', () => {
    const testString = 'A simple email: abc@gmail.com, '
    + 'or a very short one a@example.com '
    + 'hitting the maximum domain length (63 chars) test@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com '
    + 'or the maximum address length (64 chars) sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@test.com '
    + 'overall length of 254 averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com '
    + 'domain with many dashes sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asj-j-s-sjdjdjdjd-jdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdke.com.ab.net.aa.bb.cc.dd.ee '
    + ' how about a domain with repeated labels of 63 chars test@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.com '
    + 'max length email with italics '
    + '_averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com_ '
    + ' xn-- style domain test@xn--diseolatinoamericano-76b.com '
    + 'or a more complex case where we need to determine where to apply italics markdown '
    + '_email@test.com\n_email2@test.com\n\nemail3@test.com_ '
    + 'some unusual, but valid prefixes -test@example.com '
    + ' and _test@example.com '
    + 'a max length email enclosed in brackets '
    + '(averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com) '
    + 'max length email with ellipsis ending '
    + 'averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com... '
    + 'try a markdown link with a valid max-length email '
    + '[text](sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com.a.aa.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjasfff)';
    const result = 'A simple email: <a href="mailto:abc@gmail.com">abc@gmail.com</a>, '
    + 'or a very short one <a href="mailto:a@example.com">a@example.com</a> '
    + 'hitting the maximum domain length (63 chars) <a href="mailto:test@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com">test@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com</a> '
    + 'or the maximum address length (64 chars) <a href="mailto:sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@test.com">sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@test.com</a> '
    + 'overall length of 254 <a href="mailto:averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com">averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com</a> '
    + 'domain with many dashes <a href="mailto:sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asj-j-s-sjdjdjdjd-jdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdke.com.ab.net.aa.bb.cc.dd.ee">sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asj-j-s-sjdjdjdjd-jdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdke.com.ab.net.aa.bb.cc.dd.ee</a> '
    + ' how about a domain with repeated labels of 63 chars <a href="mailto:test@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.com">test@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekasgasgasgasgashfnfn.com</a> '
    + 'max length email with italics '
    + '<em><a href="mailto:averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com">averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com</a></em> '
    + ' xn-- style domain <a href="mailto:test@xn--diseolatinoamericano-76b.com">test@xn--diseolatinoamericano-76b.com</a> '
    + 'or a more complex case where we need to determine where to apply italics markdown '
    + '<em><a href="mailto:email@test.com">email@test.com</a><br />'
    + '<a href="mailto:_email2@test.com">_email2@test.com</a><br /><br />'
    + '<a href="mailto:email3@test.com">email3@test.com</a></em> '
    + 'some unusual, but valid prefixes <a href="mailto:-test@example.com">-test@example.com</a> '
    + ' and <a href="mailto:_test@example.com">_test@example.com</a> '
    + 'a max length email enclosed in brackets '
    + '(<a href="mailto:averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com">averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com</a>) '
    + 'max length email with ellipsis ending '
    + '<a href="mailto:averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com">averylongaddresspartthatalmostwillreachthelimitofcharsperaddress@nowwejustneedaverylongdomainpartthatwill.reachthetotallengthlimitforthewholeemailaddress.whichis254charsaccordingtothePHPvalidate-email-filter.extendingthetestlongeruntilwereachtheright.com</a>... '
    + 'try a markdown link with a valid max-length email '
    + '<a href="mailto:sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com.a.aa.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjasfff">text</a>';
    expect(parser.replace(testString)).toBe(result);
});

test('Test markdown replacement for invalid emails', () => {
    const testString = 'Replace the valid email part within a string '
    + '.test@example.com '
    + '$test@gmail.com '
    + 'test..new@example.com '
    + 'Some chars that are not allowed within emails '
    + 'test{@example.com '
    + 'domain length over limit test@averylongdomainpartoftheemailthatwillgooverthelimitasitismorethan63chars.com '
    + 'address over limit averylongaddresspartoftheemailthatwillgovoerthelimitasitismorethan64chars@example.com '
    + 'overall length too long '
    + 'sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com.a.aa.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjasfffa '
    + 'invalid domain start/end '
    + 'test@example-.com '
    + 'test@-example-.com '
    + 'test@example.a '
    + '[text](sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com.a.aa.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjasfffa)';
    const result = 'Replace the valid email part within a string '
    + '.<a href="mailto:test@example.com">test@example.com</a> '
    + '$<a href="mailto:test@gmail.com">test@gmail.com</a> '
    + 'test..<a href="mailto:new@example.com">new@example.com</a> '
    + 'Some chars that are not allowed within emails '
    + 'test{@example.com '
    + 'domain length over limit test@averylongdomainpartoftheemailthatwillgooverthelimitasitismorethan63chars.com '
    + 'address over limit averylongaddresspartoftheemailthatwillgovoerthelimitasitismorethan64chars@example.com '
    + 'overall length too long '
    + 'sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com.a.aa.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjasfffa '
    + 'invalid domain start/end '
    + 'test@example-.com '
    + 'test@-example-.com '
    + 'test@example.a '
    + '[text](sjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjab@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.com.a.aa.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcj.asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjasfffa)';
    expect(parser.replace(testString)).toBe(result);
});

[chiragxarora]

Hi @jjcoffee ,
You're right about the duplicate regex rules and those two can be combined into one which is being used in the backend (if it is being used).
I believe probability of a user coming in and typing a very long email like string is not much but having a limit on length makes our app more consistent with any modern day chat application in the market and also RFC compliant at the same time. Again possibility of someone entering a very long character email like string is low but if we render it as a link which opens in an email app, that would reflect as overall bad aesthetics and validation. Since we cannot accurately validate the TLDs without the DNS lookup, the least we can do is validation on length part on frontend.

[aimane-chnaif]

I agree that it's inconsistency issue to validate emails differently in various parts (login, chat, add contact, search, etc) with different regexes.
I think we should also match validations between frontend and backend.
@NikkiWines can you please share all email validation regexes used in backend along with where they're used?

[abdulrahuman5196]

I am coming from the PR which introduced MARKDOWN_EMAIL regex.

I am not sure if understand the problem statement correctly or what is trying to be fixed here.

1. The problem statement expects "should not parse the email address as a valid one for abc@abc.abc".
2. And the proposal and slack conversation is talking about limiting email length to RFC standard.
3. And we are also checking to make common regex for emails(this was also talked in the original PR which introduced MARKDOWN_EMAIL).

I am confused here, Are we trying to solve all 3 above or something specifically? Could you kindly provide this information so I can provide better information with regards to the original MARKDOWN_EMAIL regex. @aimane-chnaif

[aimane-chnaif]

@abdulrahuman5196 abc@abc.abc is valid email. What the issue mentioned is very long email:

@chiragxarora as you're a reporter, can you share example invalid email here so that other can copy&paste and reproduce easily?

[chiragxarora]

Sure @aimane-chnaif

Jaj@asjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdicdjeJajasjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfisjksksjsjssskssjskskssksksksksskdkddkddkdksskskdkdkdksskskskdkdkdkdkekeekdkddenejeodxkdndekkdjddkeemdjxkdenendkdjddekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdicdjeJajasjjssjdjdjdjdjdjjeiwiwiwowkdjdjdieikdjfidekjcjdkekejdcjdkeekcjcdidjjcdkekdiccjdkejdjcjxisdjjdkedncicdjejejcckdsijcjdsodjcicdkejdi.cdjd

[Antasel]

@adelekennedy I've sent again

[jjcoffee]

@adelekennedy Thanks, applied to the job!

[EliteDevSolution]

Hello, Expensive Team

import React, { useState } from 'react';
import { View, TextInput, Button, Alert } from 'react-native';
import validator from 'validator';

const EmailValidationExample = () => {
  const [email, setEmail] = useState('');

  const handleInputChange = (text) => {
    setEmail(text);
  };

  const handleSubmit = () => {
    if (validator.isEmail(email)) {
      // Valid email, proceed with submission
      Alert.alert('Success', 'Email is valid');
      // Add code to send the email to the server here
    } else {
      Alert.alert('Error', 'Invalid email address');
    }
  };

  return (
    <View>
      <TextInput
        placeholder="Enter email"
        value={email}
        onChangeText={handleInputChange}
      />
      <Button title="Submit" onPress={handleSubmit} />
    </View>
  );
};

export default EmailValidationExample;

In this example, we're using the validator.js library to validate the email. Make sure to install it using npm install validator.

The handleInputChange function updates the email state as the user types. When the user presses the submit button, the handleSubmit function is called. It checks if the email is valid using validator.isEmail(email).

On a successful validation, you can then proceed to submit the email to the server. Make sure that server-side validation is also implemented to ensure that only valid emails are accepted.

Remember, this is a basic example and you may need to adapt it to fit your specific application structure and requirements. Additionally, you should handle edge cases and potential error scenarios appropriately.

For server-side validation, you can use PHP's FILTER_VALIDATE_EMAIL or the provided regex in your backend code to further ensure that only valid emails are accepted.

[melvin-bot]

📣 @EliteDevSolution! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:

1. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
2. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
3. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
   
   Format:

Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>

[melvin-bot]

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

[aimane-chnaif]

@EliteDevSolution as a new contributor, please review our contributing guideline. This issue doesn't have Help Wanted label so not open for proposals.

[EliteDevSolution]

Hello, Expensive Team

const validator = require('validator');

const phoneNumber = '+1334567890';

if (validator.isMobilePhone(phoneNumber, 'any')) {
  console.log('Valid phone number');
} else {
  console.log('Invalid phone number');
}

I think that no problem with phone number verification too.

[Antasel]

@adelekennedy 24 hours passed. Still didn't get invitation on my email. Could you please ping the inbox person again for me?
Thanks in advance

[adelekennedy]

@Antasel did you respond to Ben?

[EliteDevSolution]

@adelekennedy 24 hours passed. Still didn't get invitation on my email. Could you please ping the inbox person again for me?
Thanks in advance

[adelekennedy]

Have you replied to Ben? I added the screenshot of the email exchange above

[Antasel]

the invitation link was sent long ago, but in the spam mailbox. and replied to Ben about it
now I joined there. thanks @adelekennedy

[adelekennedy]

Payouts due:

Issue Reporter: $250 @chiragxarora (Upwork)
Contributor: $2000 @jjcoffee & @Antasel (Upwork)
Contributor+: $4000 @aimane-chnaif (Upwork)

Eligible for 50% #urgency bonus? N

Upwork job is here.

[adelekennedy]

@chiragxarora will you apply to the Upwork job for the reporting bonus?

[chiragxarora]

Pls send offer, I don't have connects:(

[adelekennedy]

What's your Upwork information?

[adelekennedy]

[Antasel]

@chiragxarora please confirm your upwork information

[jjcoffee]

@adelekennedy Thanks, offer accepted!

[Antasel]

@adelekennedy accepted offer, thanks

[chiragxarora]

Yes this is it @adelekennedy , pls send the offer here

[adelekennedy]

Offer sent @chiragxarora

[chiragxarora]

Offer accepted! @adelekennedy

[aimane-chnaif]

@adelekennedy I haven't received payment yet. Can you please check?

[adelekennedy]

oh @aimane-chnaif I'm so sorry I thought you were in the NewDot payment beta as well, I'll update the summary above and send you an offer through Upwork

[aimane-chnaif]

I already accepted Melvin offer

[aimane-chnaif]

@adelekennedy bump ^

[adelekennedy]

@aimane-chnaif just ended the contract