Populate Virtuals fails when localField is inside array (with reproduction code and data)

[opablo]

Prerequisites

• I have written a descriptive issue title
• I have searched existing issues to ensure the bug has not already been reported

Mongoose version

8.1.1

Node.js version

20.5.0

MongoDB server version

7.0.4

Typescript version (if applicable)

na

Description

While trying to use "Populate Virtuals" I noticed that the feature that makes it work for sub-elements and, I imagined, also sub-arrays... fails to work in this latter case.

I created an isolated piece of data+code to make sure I was not getting confused... so I can share that as a reproduction snippet.

I also checked this same code with previous versions of mongoose just in case it could be a recent bug... but I confirmed it fails to delived my expected results on all these versions:
8.1.1 - 7.6.8 - 6.12.2
And the part that works for me also fails while going down to
5.13.22
which means that the feature that I'm trying to make work started to appear in 6.x version

Steps to Reproduce

// Mongo Data:
//
// Collection: units
// { unitId: 'in', label: 'Inches' }
// { unitId: 'ft', label: 'Feet' }
// { unitId: 'yr', label: 'Yard' }
//
// Collection: reports:
// {
//   reportId: 'r1',
//   singleMeasurement:
//     { value: '1', unitId: 'in' },
//   multipleMeasurements: [
//     { value: '2', unitId: 'ft' },
//     { value: '3', unitId: 'yr' }
//   ]
// }

import mongoose from 'mongoose'

const unitSchema = new mongoose.Schema({
  unitId: { type: String },
  label: { type: String }
})

const reportSchema = new mongoose.Schema({
  reportId: { type: String },
  singleMeasurement: {
    value: { type: Number },
    unitId: { type: String } // This WORKS to assist creating 'unit' attribute in 'singleMeasurement' obj
  },
  multipleMeasurements: [{
    value: { type: Number },
    unitId: { type: String } // This FAILS to assist creating 'unit' attribute in 'multipleMeasurements' obj array
  }]
})

reportSchema.virtual('singleMeasurement.unit', { ref: 'Unit', localField: 'singleMeasurement.unitId', foreignField: 'unitId' })
reportSchema.virtual('multipleMeasurements.unit', { ref: 'Unit', localField: 'multipleMeasurements.unitId', foreignField: 'unitId' })

reportSchema.set('toJSON', { virtuals: true })
reportSchema.set('toObject', { virtuals: true })

// // // // // // // // // // // // // // //

const reportModel = mongoose.model('Report', reportSchema, 'reports')
const unitModel = mongoose.model('Unit', unitSchema, 'units')

// // // // // // // // // // // // // // //

await mongoose.connect('mongodb://username:password@localhost:27017/database')
const report = await reportModel.findOne({ reportId: 'r1' })

await report.populate('singleMeasurement.unit')
await report.populate('multipleMeasurements.unit')

console.log('\n\n singleMeasurement: ' + report.singleMeasurement.toString())
// ^^^  this WORKS; by including the 'unit' attribute inside the object

console.log('\n\n multipleMeasurements: ' + report.multipleMeasurements.toString())
// ^^^  this FAILS; by missing the 'unit' attribute inside the array

Expected Behavior

In the code sample I print out 2 cases; one working and one not working.

This is the obtained-vs-expected results of each case:

// report.singleMeasurement.unit      => Expected: Object => Received: Object      => OK
// report.multipleMeasurements[].unit => Expected: Object => Received: undefined   => FAIL

[opablo]

Something interesting that I also noticed is that... while sniffing the activity in the db using these 2 mongo commands:

// before running the snippet...
db.setProfilingLevel(2, -1)

// after running the snippet...
db.system.profile.find({}, { "command.find": 1, "command.filter": 1 }).sort( { ts : -1 } ).pretty()

I noticed that mongoose DID the query to gather the data of the 2 'units' required to populate the second .populate()...
check this out... after running the snippet

(following report is in reverse time order)

  {
    command: {   <=== after the .populate('multipleMeasurements.unit')
      find: 'units',
      filter: { unitId: { '$in': [ 'ft', 'yr' ] } }
    }
  },
  {
    command: {   <=== after the .populate('singleMeasurement.unit')
      find: 'units',
      filter: { unitId: { '$in': [ 'in' ] } }
    }
  },
  { 
    command: {   <=== after the reportModel.findOne({ reportId: 'r1' })
      find: 'reports', 
      filter: { reportId: 'r1' }
    }
  }

which means that the feature is "trying" to work... gathering the data but then not being able to plug it in the model

[vkarpov15]

I think the issue is due to the toString() output. Changing the following:

console.log('\n\n multipleMeasurements: ' + report.multipleMeasurements.toString())

to the following:

console.log('\n\n multipleMeasurements: ', report.toObject({ virtuals: true }).multipleMeasurements.map(m => m.unit))

shows the correct output:

 multipleMeasurements:  [
  {
    _id: new ObjectId('65bd0b25a9014d903d07c896'),
    unitId: 'ft',
    label: 'Feet',
    __v: 0,
    id: '65bd0b25a9014d903d07c896'
  },
  {
    _id: new ObjectId('65bd0b25a9014d903d07c897'),
    unitId: 'yr',
    label: 'Yard',
    __v: 0,
    id: '65bd0b25a9014d903d07c897'
  }
]

I'll look into why the toString() function seems to not handle virtuals.

[opablo]

wow... you are right @vkarpov15.

instruction:

console.log(report.multipleMeasurements.toString())

outputs:

{
  _id: new ObjectId('65be7dc86001a9fa5d69e676'),
  value: 2,
  unitId: 'ft'
},{
  _id: new ObjectId('65be7dc86001a9fa5d69e677'),
  value: 3,
  unitId: 'yr'
}

MISSING THE 'unit' attribute... BUT.... :

instruction:

console.log(report.multipleMeasurements[0].unit.toString())

outputs:

{
  _id: new ObjectId('65bbfa650f99b74136f0e336'),
  unitId: 'ft',
  label: 'Feet'
}

which proves your point... the data is there in the model.. it's just the serialization part that is missing to expose it

amazing

so... this part:

reportSchema.set('toJSON', { virtuals: true })
reportSchema.set('toObject', { virtuals: true })

seems to be the one that works in one case... and fails to work in the other case
as if such setting it not being propagated to child objects through an array model