Email verification/confirmation bugs

[mdmADA]

Dataverse 5.6 (dataverse.harvard.edu)
Tested on harvard's dataverse to see if I could replicate issues on ADA test dataverse.

What steps does it take to reproduce the issue?

1. Create new user account with a valid email address
2. Upon clicking 'Create', user receives email that user account has been created
   • the verify email link with token is in that email
   • do not click on it
3. In Dataverse, once the account is created, you are automatically logged in
4. Go to 'Account Information'
   • email is showing as 'Not Verified'
   • click the 'Verify email' button
   • popup saying 'A verification email has been sent to . Please check your inbox.'
   • no email is received

I assume because there is already a token in the confirmemaildata table that that is the reason why a new email is not sent to the user. However it is confusing to users as the popup is telling them something should happen and it doesn't. The user may not read the initial email sent to them on account creation (if the email goes to junk or since the user is automatically logged in upon account creation, the user just starts navigating through the UI) so having popups with messages that reflect the true situation (email already sent, email is sent) is beneficial.

• When does this issue occur?
  On email-based account signup

• Which page(s) does it occurs on?
  Account creation/Account information

• What happens?
  See description above ^^^

• To whom does it occur (all users, curators, superusers)?
  Users who have created an account with email

• What did you expect to happen?
  If I click 'Verify Email', if the popup says I will get an email, then I expect to get an email.
  If the initial token in the confirmemaildata table is causing the new email not to be sent, I would expect the popup to say 'An email with validation instructions has already been sent to you. Please check your inbox and junk folders.' (or something similar).

Which version of Dataverse are you using?
This was on dataverse.harvard.edu (5.6)

Any related open or closed issues to this bug report?
There are several issues for shibboleth and other authentication-related services. Not sure if there are any for this particular issue.

---

This is @pdurbin adding some screenshots and other notes as I work on this.

Here's the popup described above (I repositioned it a bit for clarity). This message is somewhat misleading. It makes it sound like and email was JUST sent by you clicking the button. What really happened is that the database was checked and an existing token was found. That existing token/link should have already been emailed to the user when they signed up:

When you click a link to verify your email, you get a success message like this:

If you then navigate to your account page, here's how a verified email looks (that is, you got the email and clicked the link):

If the user doesn't have a token yet (because it expired, for example), they get a different message when they click "Verify Email":

[sbarbosadataverse]

@scolapasta this is similar. Can I amend this issue to include this verify fail issue?

[sbarbosadataverse]

To update this issue with more improvements:
verify email link received by account creators does not always work, and when account creators receive a "failed to verify" message, they reach out to HDV support which created unnecessary support inquiries, especially since we can't resolve the failed verification.

[TaniaSchlatter]

It sounds like resolving this issue may involve editing user messaging, per the description @mdmADA added. We should sort out the current functionality, and evaluate messaging.

[TaniaSchlatter]

Discussed with @pdurbin. Desired functionality is to always send a new email when the "Verify" button is clicked.

1. The most recently sent token replaces all previous tokens.
2. Change the button text to "Send Verification Email"
3. Eliminate the popup. The confirmation message on the page serves as feedback.

[pdurbin]

I created draft pull request #8579 to address the original problem reported by @mdmADA above. It implements the solution that @TaniaSchlatter and I discussed and that she summarized above. Please see the pull request for lots of screenshots and discussion. In short, the popup is gone, the button has been renamed to "Send Verification Email", and the button always sends a fresh email with a fresh link. This should reduce some confusing. Thanks for the detailed bug report, @mdmADA.

I say "draft" pull request because we've been using this issue (#8227) as a placeholder for other improvements we might wanto make to confirm/verify email. So the pull request is in draft until we've decided we've done enough hacking and we're ready for code review.

I took a look through old issues and here's what I found.

Some history... Confirm email was originally added in pull request #3299. My pull request above is basically reverting the idea of adding a popup in #3438 (comment) in pull request #7516. Oh well. Times change. 😄 Pull request #7033 added the ability to create a user via API and NOT send an email about it. It addressed #6915 which originally included additional ideas such as the ability to set the email as confirmed via API.

#3300 is about adding some sort of restrictions on users who haven't verified their email. Perhaps they aren't allowed to created datasets, for example. This is a good idea but out of scope for the current effort.

#5663 is about Shib users and verified email. What we say in the guides is "Users of the Institutional Log In option are not required to verify their email address because the institution providing the email address is trusted." However, as the issue explains, only new Shib accounts have verified emails. Shib accounts that were upgraded from DVN 3.x are not verified and still show a "Verify Email" button. (I can see this for my own account on Harvard Dataverse.) I'd be happy to spend a little time trying to fix this as part of the current effort but I'm thinking I'll follow up with a second pull request rather than muddying the waters with the first one. I'll know more once I'm in there. I can definitely see why the bug is happening at least and I'll leave a screenshot and a comment or two on the original issue.

#6679 is about having OIDC auth also set "email verified" to true like Shib does. Having never hacked on the OIDC code, this seems like a larger effort, especially if we go down one of the proposed routes of refactoring all of the auth providers so that they can indicate if they set email as verified or not. I'm declaring this out of scope for the current effort.

#8582 echos the comment above by @sbarbosadataverse about the "verify email" link not working, confusing messaging and notifications. This issue was originally opened at https://github.com/IQSS/dataverse.harvard.edu/issues/121 but I just moved it to the main repo (with Sonia's permission). @TaniaSchlatter once Sonia has sent me some RT tickets and/or I've had a chance to dig into this more, I'd like to run any messaging changes by you. I believe that all the new messaging (changes we recently worked on) is captured in screenshots and emails in pull request #8579.

So, in summary, what I see as in scope:

• needs to be reproduced (RT tickets might help) and possible messaging changes: Notification for email verification: notifications should include a more useful message so users understand what they are seeing #8582
• probably a small effort, not blocked, I'll propose a plan in the issue: Shibboleth users who predate Shibboleth are assumed to be email-verified but aren't #5663