Calling populate on embedded doc of embedded doc

[dmmalam]

Hey,
I have the following schemas ( simplified for issue)

User = new Schema
login:String

A = new Schema
emb: [B]
comments: [Comments]

//and following embedded schemas

B = new Schema
comments: [Comments]

Comments = new Schema
creator:
type: ObjectId
ref: 'User'
message: String

Now while populate works on A
eg A.find().populate('comments.creator')

it doesn't work on the double nested embedded doc
eg A.find().populate('emb.comments.creator')

Any ideas?

[eneko]

The way I am doing that currently is populating the nested items and then iterating through them to populate the children. I was planing on implementing a solution for populating multiple nested levels, but it get very tricky.

[paulshapiro]

Hi eneko,

How are you doing that?

When you do your first populate, you get returned the nested items but when you run through them to populate their children, are you able to save the populated children? It never sticks when I try to do it.

Some code would be awesome.

Thanks,
Paul

[paulshapiro]

@eneko nevermind, just needed .toObject() ;)

[eneko]

Actually Paul, I figured populate() works on queries that return multiple elements, so there is no need to iterate through the child objects. But yes, you are right, you need to call .toObject if you want the properties to stick.

Here is an example:

// Ideally should be Parent.findOne().populate('children').populate('children.grandchildren').run();

function loadParentWithChildrenAndGrandChildren(parentId) {

  // Load parent without children references (children is array of ObjectId)
  Parent.findOne({ id: parentId }, { children: 0 }, function(err, parent) {
    if (err || !parent) return next(new Error("Parent not found: " + parentId));

    // Load children for this parent, populating grandchildren (no need to load parent reference)
    Children.find({ parent: parent._id }, { parent: 0 })
      .populate('grandchildren', [], { })
      .run(function(err, children) {
        if (err) return next(new Error("Could not load children: " + parentId));

        var result = parent.toObject();
        result.children = children;
        next(null, result);
      });
  });

}

[paulshapiro]

Ah, thanks for your detailed answer. Your mongoose calls are more specific than mine and I could learn from your approach. As it happens my particular application required a recursive populate, so for now I may be stuck doing this manually.

I end up doing something like this.

var viewWithId_forDisplay = function(id, callback) {
if ( !id || typeof id === 'undefined' || id.toString().match(app_utils.emptyReg) ) {
    throw new Error('Bad user ID.');
}

View.findById(id)
.run(function(err, obj) {
    if ( err || !obj || !obj.subviews || !obj.subviews.length ) return callback(err, obj);

    obj = obj.toObject();

    // recursive subview fetch
    async.map( obj.subviews, viewWithId_forDisplay, function(err, results) {
        obj.subviews = results;

        return callback(err, obj);
    } );

})
}

[dmmalam]

Does this now work in the latest mongoose release?

[dboundz]

So I think this is the issue I'm experiencing. Each Comment object has an embedded User object. Comments are embedded as an array in Activity. A query for activities (as below) does not populate the User object on each comment.

var get_activities = function (callback) {
  var _this = this
  Activity.find({}, [], {sort:{ _id: -1 }})
  .populate('user')
  .populate('comments')
  .run(function (error, activities) {
    callback(error, activities)
  })
}

var User = new Schema({
    id: { type: String, required: true, lowercase: true, index: { unique: true } }
  , email_address: { type: String, required: true, index: true }
  , name: { type: String, required: true }
  , first_name: { type: String,  required: true } 
  , last_name: { type: String, required: true }
  , user_name : { type: String, required: true, lowercase: true, index: { unique: true } }
  , avatar_url: { type: String, required: true }
  , bio: String
  , following: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
  , followers: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
})

var Activity = new Schema({
    id: { type: Number, required: true, index: { unique: true } }
  , user: { type: Schema.ObjectId, ref: 'User', required: true }
  , recipients: [{ type: String, index: { unique: true } }]
  , type: String  
  , body: { type: String, required: true }
  , timestamp: { type: Date, required: true }
  , likes: [{ type: Schema.ObjectId, ref: 'User', index: { unique: true } }]
  , comments: [{ type: Schema.ObjectId, ref: 'Comment' }]
})

var Comment = new Schema({
    id: { type: String, required: true, index: { unique: true } }
  , timestamp: { type: Date, required: true }
  , body: { type: String, required: true }
  , user: { type: Schema.ObjectId, ref: 'User', required: true } 
}) 

[dethe]

Is there any plan for populate to support nested paths to handle cases like this?

[aheckmann]

yes we'd like to at some point. it could quickly become a performance issue but yeah we should support it.

[dethe]

For nested objects I need to iterate over them manually, loading them, which seems like more of a performance issue and clutters up my server code. So I'm +1 for making my job easier ;-)

In the meantime I will continue, and also address it by splitting sections of the page into separately loadable objects that don't require such deeply nested data on each call.

[Qard]

I'd be all for something like this;

Parent.findById(1).populate('child').run(function (err, parent) {
  parent.child.populate('grandchild', function (err) {
    console.log(parent.child.grandchild)
  })
})

It'd actually be pretty handy to have populate available on the document like that. You could keep diving as far as you need.

[aheckmann]

@Qard you read my mind.

[eneko]

@Qard that is how mongoose-relationships works. I used that module for a while at first, but ended up doing things manually. Maybe Mongoose should integrate it into the core, or at least parts of it like calling populate on arrays.

[diversario]

Has anything like #601 (comment) been implemented yet?

[mikeuduc]

I'm wondering the same thing. I'm trying to figure out how to populate embedded doc child references. @DBounds I have an analogous models. Have you resolved this issue: #601 (comment). I can't find mongoose-relationships

[mikemoser]

The information in this thread has been very helpful in understanding the current state of .populate(), the current limitation related to nested population of grandchildren and how .toObject() in combination with manual lookup can be an option to workaround it.

I'm wondering if there is a technique we could use that does not require .toObject(). I ask b\c after we populate our model and the grandchildren we'd still like to have a Document and proper types so we can use functions like parent.childarray.id(x) or modify the values and call .save().

Any help here would be greatly appreciated.

note: We've tried using parent.set('child.grandchild', value), however that seems to cause some issues with the integrity of the Document and we are no longer able to read values from that path (error is Invalid ObjectId when trying to read parent.child or parent.child.grandchild).

[aheckmann]

related: http://stackoverflow.com/questions/11137239/querying-nested-embedded-documents-with-mongoose

[vartana]

@aheckmann Any ideas on this yet?

[aheckmann]

been busy. i want it in 3.0 final

[vartana]

@aheckmann Awesome cant wait. It's will be really useful in real applications. But I do want to give you kudos for the great job on what you have accomplished so far.

[jsalonen]

Great awesomeness! I will use this feature as soon it is available.

[JoshuaGross]

Hey folks, I wrote my own layer around Mongoose that supports sub-population. It's not architecturally ideal since it's a wrapper around a wrapper, but it does support subpopulate the way we want it to work. Is anyone interested in seeing it if I clean it up and release it?

[jsalonen]

Definitely interested! I'm not sure if I'm the right person to do the patching though, but any ideas on how to work around this issue right now are very welcome!

[JoshuaGross]

Right now it's a hack, albeit a pleasant hack from my perspective :) Hopefully it can be used to patch Mongoose but it's a monkey-patch at the moment. I'll write some more tests, document my code, and come back here within the next week.

[hackfrag]

Will this be added in version 3.0 ?

[aheckmann]

@hackfrag yes

[aheckmann]

Here's the plan for populate in 3.6: #1292

[dexcell]

+1 We need this. Hopefully released soon, (because the issue already started year ago)

[jamlfy]

I think by popular vote, won this proposal!

[aheckmann]

done. https://github.com/LearnBoost/mongoose/wiki/3.6-pre-release-notes

[jsalonen]

@aheckmann Thank you a lot!!

Also love lean() too so this seems really superb so far!

[andyburke]

Excited for 3.6, this is a good fix, thanks!

[jechenique]

Thanks mr!

On Mar 5, 2013, at 5:26 PM, Andy Burke notifications@github.com wrote:

Excited for 3.6, this is a good fix, thanks!

—
Reply to this email directly or view it on GitHub.

[donnut]

@aheckmann Great, thank you!

[jamlfy]

Thanks!!! Great Job!!!

[flockonus]

That's awesome! I believe it can help with my case. How should I do in this scenario:

var UserSchema, WineRating, WineSchema, mongoose;

UserSchema = new mongoose.Schema({
  wine_ratings: {
    type: [
      {
        type: mongoose.Schema.Types.ObjectId,
        ref: 'WineRating'
      }
    ]
  }
});
mongoose.model("User", UserSchema);

WineRating = new mongoose.Schema({
  wine: {
    type: mongoose.Schema.Types.ObjectId,
    ref: 'Wine'
  }
});
mongoose.model("WineRating", WineRating, 'wine_ratings');


WineSchema = new mongoose.Schema({
  name: String
});
mongoose.model("Wine", WineSchema);

mongoose.model("User").findById(user._id).populate('wine_ratings.wine').exec(function(err, user) {});
/*
gets exception: 
TypeError: Cannot call method 'path' of undefined
    at search (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1830:28)
    at search (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1849:22)
    at Function._getSchema (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1856:5)
    at populate (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1594:22)
    at Function.Model.populate (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/model.js:1573:5)
    at Query.findOne (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/query.js:1633:11)
    at exports.tick (/Users/flockonus/workspace/az/api/node_modules/mongoose/lib/utils.js:393:16)
*/

Is there something wrong? I am on "3.6.0rc0"

[skotchio]

I have the same issue: #1377

[yangsu]

@flockonus @vovan22 I ran to the same issue. This is how I resolved it.

[joeytwiddle]

I was trying to get post.populate("comments comments._creator") working in our project, and had a little success with the following tweaks.

It doesn't seem to work for deeper queries, and the first change breaks some of the existing tests, but I hope it may be of interest for anyone working on this.

https://gist.github.com/joeytwiddle/6129653

[joeytwiddle]

Since my patch kinda sucks, I have tried to contribute by writing a test case instead! #1603

[merhawie]

joeytwiddle, i am having the exact same problem and I hope you get it fixed - I am trying to recursively populate but I get the can't find path error too. I don't know if this is by design ...

[joeytwiddle]

Yes the infamous 601. It is by design, at least for the moment. The latest Mongoose release does support deep population, but only within one schema.

In our project we need to do deep population across different models quite often, so we have written a helper function:

https://gist.github.com/joeytwiddle/6129676

This allows you to populate descendants of one document to any depth you like! It still requires one extra callback after you have fetched the document, but only one. For example:

deepPopulate(blogPost, "comments comments._creator comments._creator.blogposts", {sort:{title:-1}}, callback);

I hope the use of doc.constructor to get the model will not prove problematic in the future!

(Thanks to Sunride and the German government!)

[merhawie]

@joeytwiddle - thank you so much! It worked like a charm. Now I will try to figure out a way to have it do n-deep populate.

[xizhao]

Any updates on this?

[digitaljohn]

@aheckmann I understand your theory on this, but I do disagree that it is an anti-pattern.

Sometimes you do need to perform expensive operations, its inevitable in complex systems and you do need to collect disparate documents together. For example, you may want to do one expensive operation when an document VERY VERY rarely changes and cache the results for performance.

Why do you believe one level of population is 'ok', but more than this is a sign of an anti-pattern?

[buunguyen]

I know this is an old thread, but I've just created a plugin that make it very easy to populate models at any level of depth. I'm posting here in case anyone's interested: https://github.com/buunguyen/mongoose-deep-populate.

The usage is very straightforward, for example:

post.deepPopulate('votes.user, comments.user.followers, ...', cb);
Post.deepPopulate(posts, 'votes.user, comments.user.followers', cb);

Please check out the plugin repo for more information.

[lbeschastny]

@buunguyen nice job!

[joaom182]

Not working for me.

models/missionParticipation.js

var deepPopulate = require('mongoose-deep-populate');
var mongoose = require('mongoose');
var Types = mongoose.Schema.Types;

var missionParticipationSchema = new mongoose.Schema({
        user: {
            type: String,
            default: ''
        },
        mission: {
            type: Types.ObjectId,
            ref: 'Mission'
        },
        images: [{
            type: Types.ObjectId,
            ref: 'Image'
        }]
    }, {
        toJSON: {
            getters: true,
            virtuals: true
        },
        toObject: {
            getters: true,
            virtuals: true
        }
    });

    missionParticipationSchema.plugin(deepPopulate, {
        whitelist: [
            'images',
            'mission',
            'mission.images.poster',
            'mission.images.banner'
        ]
    });

var MissionParticipation = mongoose.model('MissionParticipation', missionParticipationSchema);

module.exports = MissionParticipation;

services/missionParticipationService.js

MissionParticipation.find({user: userID}).deepPopulate('mission.images.poster mission.images.banner').exec(function (err, missionParticipationsDocs) {
    // do the magic.
});

And i get this error on console

TypeError: Object #<Query> has no method 'deepPopulate'

[Swissbite]

@joaom182
I did not yet use deepPopulate, but recording to the docs on http://npm.taobao.org/package/mongoose-deep-populate I would assume that the correct call should be:

MissionParticipation.find({user: userID}, function (err, participations) {
  MissionParticipation.deepPopulate(participations, 'mission.images.poster mission.images.banner', function(err) {
    if (err) {
      //handle it
      return void 0;
    }
    //do your magic stuff. with participations, which are populated in place in the examples 
  })
})

Regards

[joaom182]

I found another way, but I'm concerned about the performance, I try to make a comparison.

The other way uses the async module

MissionParticipation.find({
   user: userID
}).populate('mission').exec(function (err, missionParticipationsDocs) {
   if (err)
      return; // handle error

   async.forEach(missionParticipationsDocs, function (mp, callback) {
      mp.mission.populate('images.poster', 'images.banner', 'prize', function (err, result) {
         callback();
      });
   }, function (err) {
      // forEach async completed
      if(err)
         return; // handle error
      resolve(missionParticipationsDocs);
   });
});

[buunguyen]

@joaom182 you're a bit too fast :). Although I added code to bring deepPopulate to Query, I delayed pushing a new version on NPM so that I could test a bit more.

I've just pushed the new version (0.0.7). So this syntax you used should work after you update the dependency:

MissionParticipation
  .find({user: userID})
  .deepPopulate('mission.images.poster mission.images.banner')
  .exec(cb);

[joaom182]

@buunguyen Awesome!

[vkarpov15]

Can y'all open future issues in the mongoose-deep-populate repo please? Makes everyone's life a little easier :)

[ghost]

@buunguyen Awesome! The best thing that happened to my project and You made my day! Thanks!