feat(config): Mongo Seed 2.0

[mleanos]

Adds a more configurable and easily extended MongoDB Seed feature.

Adds additional options at the collection, and collection item level to
allow more control over how each environment config handles the seeding
feature.

Collection level options:
skip: { when: {} }: Provides a configurable way to skip seeding a
particular collection based on results of a Mongoose query.
logResults: true/false: Overrides global Seed logResults settings.

Document level option:
overwrite: true/false: Specificy whether or not to overwrite a
document if it already exists.

Fixed tests due to SeedDB changes.

[mleanos]

This is a modified version of changes that I've made to an existing MEANJS application. I found it rather hard to use the existing SeedDB feature. I found myself wanting more control over how the User's were added to the database, and to have the ability to add more User's.

In addition to the limitation I found, I also wanted to be able to seed other data models. This implementation might be a little wonky with the options that I've provided. However, something along these lines would be very for users of this framework. I'm open for suggestions on improvements here.

In my other project, I also created a couple Gulp tasks that I can run the SeedDB feature from. This made it much easier for me to use both in my dev & production environments. I'll add the Gulp tasks to this PR, if this gets any positive feedback.

[mleanos]

@lirantal Any ideas on the coverage? I don't see how it could have decreased, let alone by that much. I can't make sense of the Coveralls report either.

[lirantal]

The generalization of it is interesting, I'm ok with that.
Looked at the coverage report as well and the changes related to the frontend routing which seems odd, as you mentioned.

[simison]

I found it rather hard to use the existing SeedDB feature. I found myself wanting more control over how the User's were added to the database, and to have the ability to add more User's.

+1 — I never merged that feature from upstream to Trustroots cuz it seemed kinda cumbersome. Instead we've used our own which we wrote before seeding was available in mean.js. Ours is quite hacky so I would be glad to give this a test.

I'd like having this as a command npm run seed rather than as a variable/global/whatnot.

[mleanos]

@lirantal @simison I've added Gulp tasks (& npm scripts) to run the seeding. I've left the "whatnot" implementation there, but I'm not opposed to completely removing it in this PR. Now that there are NPM scripts to run these operations, users of this framework can add them to their own builds/workflows.

I'm glad to see positive feedback here. Now moving on, there are a couple remaining issues that I could use input on..

I haven't been able reconcile the missing User reference in the Article seed. Currently, we can't guarantee when either seeding operations will complete, so we can't just look up a User. One way we could solve this limitation is by changing the SeedDB configuration object around to include the Users model in a "run this first" setting; something like:

seedDB: {
    seed: process.env.MONGO_SEED === 'true',
    options: {
      logResults: process.env.MONGO_SEED_LOG_RESULTS !== 'false'
    },
    prerequisite: {
      collections: [{
        model: 'User',
        docs: [{
          data: {
            username: 'local-admin',
            email: 'admin@localhost.com',
            firstName: 'Admin',
            lastName: 'Local',
            roles: ['admin', 'user']
          }
        }, {
          // Set to true to overwrite this document
          // when it already exists in the collection.
          // If set to false, or missing, the seed operation
          // will skip this document to avoid overwriting it.
          overwrite: true,
          data: {
            username: 'local-user',
            email: 'user@localhost.com',
            firstName: 'User',
            lastName: 'Local',
            roles: ['user']
          }
        }]
      }]
    },
    collections: [{
      model: 'Article',
      options: {
        // Override log results setting at the
        // collection level.
        logResults: false
      },
      skip: {
        // Skip collection when this query returns results.
        // e.g. {}: Only seeds collection when it is empty.
        when: {} // Mongoose qualified query
      },
      docs: [{
        data: {
          title: 'First Article',
          content: 'This is a seeded Article for the development environment'
        }
      }]
    }]
  }

Seems a little weird to me. Another way might be to check for the existence of the User model in the SeedDB collections setting, and ensure that operation completes before any other model is seeded.

Any thoughts on that? Any suggestions would be very welcome. @simison How did you handle this case over at Trustroots?

Lastly, do y'all think these are breaking changes? Any customized SeedDB settings (in the previous format) that any of our users are relying on would be invalid based on the new config structure. I just feel that not many users are using this feature as it currently stands; this is, in part, why I've introduced these changes.

[mleanos]

@lirantal @simison I've added a solution to the above mentioned issue with the Article's user reference. I think this is a good way to handle it, but I'm open to other ideas.

Tests are passing, and if I these latest changes are approved, I'll add couple more tests.

[mleanos]

BTW, I'd like to see #1818 go in before this.

[lirantal]

To be honest I think the seeding is a bit over complex and very coupled to what's going on there now with User and Articles (like this return collection.model && collection.model !== 'User';)

Maybe we can solve it on the seedDB configuration object instead of actual code?

P.S. I'm curious why you removed the arrow functions?

[mleanos]

@lirantal Do you think the seeding is complex and coupled in this PR?

Another approach I thought about was to add a "task" (function) to the config/lib/seed service for each Model that a user of MEANJS wants to seed. Then still have a static Mongoose method on each model to seed an actual document.

pseudo code:

// config/lib/seed.js

function start(config) {
  var userSeeds = [config.seedDB.options.seedUser, config.seedDB.options.seedAdmin]
    .map(function (item) {
      return seed('User', item);
    });
  var otherSeeds = [config.seedDB.options.seedOtherModel]
    .map(function (item) {
      return seed('OtherModel', item);
    });
  var articleSeeds = [config.seedDB.options.seedArticle1, config.seedDB.options.seedArticle2]
    .map(function (item) {
      return seed('Article', item);
    });
  var moreSeeds = [config.seedDB.options.yetAnotherModel]
    .map(function (item) {
      return seed('YetAnotherModel', item);
    });

  // Process Seeds in strict order
  Promise
    .all(userSeeds) // User seeds would most likely go first in most use cases
    .then(function () {
      return Promise.all(otherSeeds);
    })
    .then(function () {
      return Promise.all(articleSeeds);
    })
    .then(function () {
      return Promise.all(moreSeeds);
    })
    .then(function () {
      console.log('All Seeds Complete');
    })
    .catch(function (err) {
      console.log('Error occurred while seeding database!');
      console.log(err);
    });
}

function seed(modelName, data) { // we could add an options param here too
  var Model = mongoose.model(modelName);

  // Model.seed instance method is responsible for deciding whether to override 
  // existing documents, or any other model specific logic.
  return Model.seed(data, config.seedDB.options.logResults); // returns Promise
}

Anytime a MEANJS user wants to seed a new model, they can easily add it to the seed service, and make sure there's a seed instance method on that Mongoose model schema.

Maybe we can solve it on the seedDB configuration object instead of actual code?

To solve what? And what did you have in mind?

I removed the arrow functions based on Mikael's comments in the mongoose-upgrade PR, when he said that we should probably have a bigger discussion before we start adding ES6 features.

[lirantal]

sorry if I wasn't clear before - I meant that processing the user seed first before the articles should not be really coupled to the seeding execution because then someone who doesn't need that exactly will need to edit seed.js

[mleanos]

@lirantal I think in most cases, whatever data model's are added to a project, the User model would need to be seeded first. Anytime a model's field references a User, it would be missing if there's no user yet. The coupling of the User & Article models seems appropriate since MEANJS's Article's require a User reference.

As for a MEANJS user editing seed.js, I had that same position as you. However, this framework is meant as an example of best practices & a way to get an app up & running fast. A MEANJS user doesn't have to worry about figuring out how to do certain things that are common among most applications. This is also true of database Seeding. Once a user starts to add their own models, they would just need to modify seed.js to suit their own needs. I don't see this as much difference from the rest of the application's boilerplate code/configurations.

Maybe we can solve it on the seedDB configuration object instead of actual code?

Did you have anything specific in mind?

EDIT: Did you mean that we could enforce the ordering using the configuration object? If so, this might allow our seeding feature to remain agnostic to specific data models?? I could see a interesting implementations that could allow this to work. WDYT?

Should we just not seed Articles? If so, our user's would have little value in our Seeding feature other than to seed a couple users.

Liran, what did you think of my previous example with multiple seeds & the ordering? It seems like a pretty clean implementation. I know it's still tightly coupled, but we might not have any choice.

[lirantal]

yeah @mleanos I was referring to putting that logic in the configuration object, this way user's can remove it easily without needing to alter code.
We're aligned with the framework target and audience, but just saying that not everyone's use-case includes users and logins. Some users may use it for non-user related stuff.

Let me know if you want to follow-up on the configuration object to control seed ordering.

[mleanos]

@lirantal Should we just consider the order of the seed collections, and data, in the config object to be the order of the seed?

[lirantal]

that's what I think, yes.

[mleanos]

Sounds good.. Thanks. Let me work on those changes.

[mleanos]

@lirantal I've updated this branch to enforce the order of the seed operations based on the configuration. Let me know what you think.

I also added two new tests to confirm the seed ordering is enforced.

[lirantal]

LGTM, thanks.

[mleanos]

@simison Have you had a chance to test this feature out? I'm curious if it would work for you use case.

There are convenient Gulp tasks for seeding data with dev, test, and prod environment settings. You'll just have to ensure the seedDB settings have been updated in your env configs 😸

npm run seed
npm run seed:test // could be used as a "dry run" test
npm run seed:prod

[simison]

@mleanos nope didn't have time and not sure when I'll have more time to really dig into this. Looking forward though!

[mleanos]

This can probably go in now.. Feel free to merge. Or I can in a couple of days, if I don't hear any additional feedback.

[simison]

@lirantal @mleanos good for merge?

[mleanos]

Yes. It's ready to me.

I hesitated too see if anyone came along and also wanted to make sure this went in with a major release bump.

Good to go 😀

[simison]

@mleanos cool, so you can squash and merge as you probably know best how the commit message should be written for this.

also wanted to make sure this went in with a major release bump.

You mean v1.0.0? As far as I know meanjs doesn't follow semver. Next up would be v0.6.0.

[mleanos]

Meant minor bump 🤗 You're correct!

[mleanos]

I'll merge later today. Been busy last few days. Just moved and still getting settled in. Love seeing all the contributions happening now 👍