fix(users): GitHub strategy missing email #1250

mleanos · 2016-03-06T12:47:46Z

Fixes an issue with an empty/missing/null Email coming from GitHub's
OAuth call response.

Also, introduces the sparse index option on the User model's Email
field. This will ensure that we can have multiple User documents without
the Email field.

Related #1145

mleanos · 2016-03-06T12:50:20Z

I'll add some tests to this PR, sometime tomorrow.

mleanos · 2016-03-06T20:23:06Z

Added a test to ensure the sparse option is working properly.

@lirantal WDYT?

lirantal · 2016-03-06T21:37:37Z

I think it's overall ok, although I'm wondering if there's really a case where GitHub will return no email for a user?

mleanos · 2016-03-06T21:45:31Z

If a user chooses the "Don't show my email address" option in their Profile settings for Public email. Then GitHub won't return the email address. Also, Twitter doesn't send user's email at all.

lirantal · 2016-03-06T21:46:34Z

Which only make you wonder how twitter ever worked, right? :)
Ok, let's see if anyone else has anything else to contribute

mleanos · 2016-03-06T21:53:34Z

Twitter worked, because we allow empty/missing email fields in our User model. The issue will only show itself if there's an attempt to save a new User without the email field. Thus, mongodb treats both of those as the same, and won't allow it since we have the unique index setting on that field; there's a violation of the unique constraint in that case.

We don't run into this issue with local testing, since most of us just use one User for any particular Social Account. I guess User's of this framework haven't ran into this issue, or reported it. Until @farajfarook did.

lirantal · 2016-03-08T22:02:06Z

That's what I was referring to - that probably users of MEAN.JS who used GitHub did get this problem for such cases where the user opted out from sharing the email detail.

Anyway, it looks like a correct way of handling it.
LGTM.

@codydaig maybe another pair of eyes?

lirantal · 2016-03-08T22:02:39Z

@mleanos BTW you are definitely able to run some manual tests with local users signup, GitHub as well as another strategy, right? just to make sure we're not breaking anything larger here.

mleanos · 2016-03-09T00:18:13Z

@lirantal I've been going through the Social Accounts implementation & testing. I've noticed some issues with the other providers as well. This fix is part of a bigger initiative to get the Social Providers working 100%. I have a list of issues I've come across, but haven't organized it yet.

I have tested this with the other provider's and there aren't any issues, other than the other aspects of the Social Provider's implementation that are broken. However, this does also take care of the issue raised in #1145.

I did extensive testing with the sparse option, and it won't have any adverse effects. The only nuance is that when this is integrated into an existing application, the index on the email field in the MongoDB instance would need to be dropped. The next time the application is run, Mongoose will rebuild the index using the sparse option. I added a comment in the Schema for that.

Also, the test I added will ensure that the sparse index works; regardless of which Provider is being used.

lirantal · 2016-03-10T09:38:23Z

@mleanos ok, that's clear.
Do users have to manually drop the index or will mongodb rebuild it automatically for them?
If it's manually, I'm not sure users will understand that straight away - is it possible to catch the error from mongodb on that and show a message to guide them?

mleanos · 2016-03-10T22:30:22Z

User's do have to manually drop the index. However, we can probably add a Gulp task that is executed with one of our other tasks, that start the server; default, prod, test. Do you think that would work?

Another option might be to add something to our mongoose server config, that ensures the index is dropped if the sparse option is not set for it. Or we simply display a message in this case. I haven't looked into that, but I think it will be possible. One caveat to this option is that this check will be performed every time the database is started; I'm not sure how I feel about that.

I'd like to make it as easy as possible for our users. However, it might be more trouble for us than it's worth. As it stands, it's technically not a breaking change. Users will only run into any issue when a duplicate user is saved that violates the indexes unique constraint; in this case, a missing/empty email field.

I'm happy to do more work on this to make it easier on our users. But I just wanted to throw those options out there. And the inline comment might suffice for now.

lirantal · 2016-03-11T17:21:19Z

We would like to be unobtrusive but in the same time not break things horribly.
This has been a recurrent issue for us with other updates we had before like fixing the users table's salt which was accidentally binary and that messed things up and was a breaking change too when we fixed it.

I would like to suggest that we have an UPGRADE.md (and possibly accompanying wiki page or whatever else helps as well) that states the big changes that require user involvement when the version changes.

@ilanbiala WDYT?

ilanbiala · 2016-03-11T22:55:40Z

If we do like Angular does and put in our Changelog the breaking changes and the fixes, then I think that is sufficient.

lirantal · 2016-03-12T13:13:49Z

sounds good to me.
@mleanos can you include this kind of update with this PR about the index change and instructions of what should be done?

mleanos · 2016-03-13T02:15:11Z

I added a UPGRADE.md to this PR. However, I may need help with the markup. I don't have a way of testing the markup locally, since I'm on Windows. And AFAIK Jenkins isn't supported on Windows.

lirantal · 2016-03-13T13:27:39Z

UPGRADE.md

+  * If you are upgrading an existing MEANJS application, and have not changed this field, you will already have an existing index on the Email field of your Users collection. If this is the case, you will need to follow these steps for the `sparse` option to be applied.
+
+
+    1. Manually drop the Email field's index in your Users collection, on your applications MongoDB instance.


@mleanos
Can we provide actual copy&paste instructions on how to do it to ease the users?

Sure. I can add the MongoDB commands to drop the index, and then the commands to start the application.

lirantal · 2016-03-13T13:28:02Z

@mleanos added my comment and also you can see how the markup renders through GitHub, check this URL for your file: https://github.com/mleanos/mean/blob/fix/users-github-strategy-email/UPGRADE.md

mleanos · 2016-03-13T17:40:30Z

@lirantal I know how to test it once it's in a remote GitHub repo. I just can't view the rendered markup locally. It's not a problem. I can just push to my fork, and test there.

mleanos · 2016-04-06T01:05:56Z

@lirantal @ilanbiala @codydaig I updated this PR to include a companion upgrade script, for dropping the index using Mongoose. I also updated the UPGRADE readme to include directions on how to use the script.

How does the UPGRADE.md file look to y'all?

coveralls · 2016-04-06T01:08:16Z

Coverage remained the same at 70.615% when pulling 07b56f6c3802a41f1be539431b9111aecf5af86f on mleanos:fix/users-github-strategy-email into bc0b4a6 on meanjs:master.

coveralls · 2016-04-06T01:18:26Z

Coverage remained the same at 70.615% when pulling 07b56f6c3802a41f1be539431b9111aecf5af86f on mleanos:fix/users-github-strategy-email into bc0b4a6 on meanjs:master.

codydaig · 2016-04-06T07:23:01Z

scripts/upgrade-users-sparse-index.js

+
+var mongoose = require('mongoose'),
+  chalk = require('chalk'),
+  mg = require('../config/lib/mongoose');


I think this should be mongoose instead of mg to be consistent with the rest of the project.

mongoose is defined above

mongooseConfig, then? mg is a bit fuzzy.

coveralls · 2016-04-09T04:13:57Z

Coverage remained the same at 70.615% when pulling 570b69b74cf7e5c5588004e46f23b6fdfeeedd69 on mleanos:fix/users-github-strategy-email into bc0b4a6 on meanjs:master.

codydaig · 2016-04-09T18:56:38Z

LGTM

lirantal · 2016-04-10T17:36:12Z

Great, let's add.

ilanbiala · 2016-04-10T20:20:21Z

scripts/upgrade-users-sparse-index.js

+  // get a reference to the User collection
+  var userCollection = db.connections[0].collections.users;
+
+  console.log(chalk.yellow('*************** Removing index "' + _indexToRemove + '" from the User collection ***************'));


Is this line not over 80 characters?

@ilanbiala Yes. I don't really like what I did with the messages. I'll clean them up.

Just to clarify, are you concerned with the length of the line of the code, or the output of the console log?

Mostly length, I think the content is pretty good.

Alright. I can break these up into multiple lines of code.

coveralls · 2016-04-13T08:29:18Z

Coverage remained the same at 70.615% when pulling 9c57a79d4fdfea28cd230864b6a4463acc61d639 on mleanos:fix/users-github-strategy-email into f2a6bf9 on meanjs:master.

coveralls · 2016-04-13T08:41:05Z

Coverage remained the same at 70.615% when pulling 9c57a79d4fdfea28cd230864b6a4463acc61d639 on mleanos:fix/users-github-strategy-email into f2a6bf9 on meanjs:master.

mleanos · 2016-04-13T08:58:38Z

@ilanbiala I've cleaned up those long lines of code. LGTY?

ilanbiala · 2016-04-13T15:03:43Z

scripts/upgrade-users-sparse-index.js

+    }
+  } else {
+    console.log(chalk.green(message));
+    console.log(chalk.green('The next time your application starts, Mongoose will rebuild the index "' + _indexToRemove + '".'));


This line is too long still.

Forgot that one :|

trendzetter · 2016-04-18T21:38:00Z

👍

ilanbiala · 2016-04-20T15:14:33Z

modules/users/tests/server/user.server.model.tests.js

@@ -193,6 +193,29 @@ describe('User Model Unit Tests:', function () {

    });

+    it('should not index missing email field, and not enforce unique index constraint (sparse index)', function (done) {


Also slightly long, this is 120 chars

ilanbiala · 2016-04-20T15:16:20Z

@mleanos strangely we don't have a linting rule for line length. Adding that would be nice for the future so we can catch extra long lines more easily.

Fixes an issue with an empty/missing/null Email coming from GitHub's OAuth call response. Also, introduces the `sparse` index option on the User model's Email field. This will ensure that we can have multiple User documents without the Email field. Adds a server-side User model test for the sparse index setting on the email field. Confirms that User documents without the email field are not indexed, illustrating the sparse option on the schema's email field works properly. Added the dropdb task to the Gulp test:client & test:server tasks, to ensure we have a clean database & that any indexes are rebuilt; this will ensure any Schema changes (in this case the email index is rebuilt using the sparse index option) are reflected when the database is started again. Added a UPGRADE.md for tracking important upgrade information for our user's to be aware of, when we introduce potentially breaking changes. Included an explanation of the Sparse index being added, and how to apply it to an existing MEANJS application's database. Adds a script for dropping the `email` field's index from the User collection. Related meanjs#1145

coveralls · 2016-04-28T22:46:51Z

Coverage remained the same at 70.615% when pulling bd64737 on mleanos:fix/users-github-strategy-email into e3cd65f on meanjs:master.

mleanos · 2016-04-28T23:08:47Z

@ilanbiala I shortened the lines in the upgrade script to be less than 80 chars. As for the test line you mentioned, I shortened it as much as possible but it's still about 105 chars long. I didn't see any other way of shortening that line. Quite a bit of our describe() lines are longer than 80 chars.

If this looks good to you now, I'll merge.

ilanbiala · 2016-04-29T02:58:44Z

LGTm.

- Intentionally omits setting email in constructor to trigger defaults when creating user. Handles cases where email is not authorized/given by provider. Related to issue meanjs#1250

* fix(users) patch OAuth default email issue - Intentionally omits setting email in constructor to trigger defaults when creating user. Handles cases where email is not authorized/given by provider. Related to issue #1250

mleanos force-pushed the fix/users-github-strategy-email branch from 7fdd0e4 to 41fa2e6 Compare March 6, 2016 19:37

lirantal assigned mleanos Mar 8, 2016

lirantal added this to the 0.5.0 milestone Mar 8, 2016

lirantal added the issue:bug label Mar 8, 2016

mleanos force-pushed the fix/users-github-strategy-email branch from 41fa2e6 to 1909b5f Compare March 13, 2016 01:58

lirantal reviewed Mar 13, 2016
View reviewed changes

mleanos mentioned this pull request Apr 5, 2016

fix(user-Model): social signup email fix #1293

Closed

mleanos force-pushed the fix/users-github-strategy-email branch 2 times, most recently from 4333aab to 07b56f6 Compare April 6, 2016 00:59

codydaig reviewed Apr 6, 2016
View reviewed changes

ilanbiala reviewed Apr 10, 2016
View reviewed changes

mleanos force-pushed the fix/users-github-strategy-email branch 2 times, most recently from a3a21d9 to 9c57a79 Compare April 13, 2016 08:15

ilanbiala reviewed Apr 13, 2016
View reviewed changes

ilanbiala reviewed Apr 20, 2016
View reviewed changes

mleanos force-pushed the fix/users-github-strategy-email branch from 9c57a79 to bd64737 Compare April 28, 2016 22:34

mleanos merged commit 4906611 into meanjs:master Apr 29, 2016

mleanos mentioned this pull request Sep 10, 2016

fix(users) Stops error on signin/signup #1495

Merged

Wuntenn mentioned this pull request Sep 11, 2016

fix(users) Handle missing email - OAuth #1501

Merged

		* If you are upgrading an existing MEANJS application, and have not changed this field, you will already have an existing index on the Email field of your Users collection. If this is the case, you will need to follow these steps for the `sparse` option to be applied.


		1. Manually drop the Email field's index in your Users collection, on your applications MongoDB instance.

		@@ -193,6 +193,29 @@ describe('User Model Unit Tests:', function () {

		});

		it('should not index missing email field, and not enforce unique index constraint (sparse index)', function (done) {

fix(users): GitHub strategy missing email #1250

fix(users): GitHub strategy missing email #1250

Conversation

mleanos commented Mar 6, 2016

mleanos commented Mar 6, 2016

mleanos commented Mar 6, 2016

lirantal commented Mar 6, 2016

mleanos commented Mar 6, 2016

lirantal commented Mar 6, 2016

mleanos commented Mar 6, 2016

lirantal commented Mar 8, 2016

lirantal commented Mar 8, 2016

mleanos commented Mar 9, 2016

lirantal commented Mar 10, 2016

mleanos commented Mar 10, 2016

lirantal commented Mar 11, 2016

ilanbiala commented Mar 11, 2016

lirantal commented Mar 12, 2016

mleanos commented Mar 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lirantal commented Mar 13, 2016

mleanos commented Mar 13, 2016

mleanos commented Apr 6, 2016

coveralls commented Apr 6, 2016

coveralls commented Apr 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Apr 9, 2016

codydaig commented Apr 9, 2016

lirantal commented Apr 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Apr 13, 2016

coveralls commented Apr 13, 2016

mleanos commented Apr 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trendzetter commented Apr 18, 2016

Choose a reason for hiding this comment

ilanbiala commented Apr 20, 2016

coveralls commented Apr 28, 2016 • edited Loading

mleanos commented Apr 28, 2016

ilanbiala commented Apr 29, 2016

coveralls commented Apr 28, 2016 •

edited

Loading