Relationships

[mquandalle]

One of the natural evolution of the collection2/simpleSchema/autoForm stack is the ability to define some relationships between models. This feature is build in most popular frameworks (Rails [1], Django [2], Symfony [3]...) and there are no good package for this purpose on Meteor right now.

I'm sure you already think of this a lot @aldeed. Could you share what are your plans to support relations at the current state?

Some of the questions raised by this topic:

• What relations do we want? For instance rails split the one to many relation into belongs_to and has_one. Do we need this kind of distinction?
• Do we automatically add fields to collections when we need, or do we let the user define explicitly fields names for relationships with other models?
• How organize the code related to relations, is it more a collection2 or a simpleSchema functionality?
• How do we manage autoForms with relations?
• API to declare a relation? Something like this?

    Manufacturers = new Meteor.Collection2('manufacturers', { schema: {
        name: {
            type: String
        }
    }});

    Cars = new Meteor.Collection2('manufacturers', {schema: {
        manufacturer: {
            // Associate a relationship type with another model/collection2
            oneToMany: Manufacturers 
        },
        color: {
            allowedValues: ['white', 'black']
        }
    }});

• How do we validate a object with relations? Could we add constraints on relations (for instance we might want to limit a blog post to 5 related tags, where posts and tags are both models)?

I think a lot a Meteor users will be interested by this topic, so feel free to post suggestions here :-)

[1] : http://guides.rubyonrails.org/association_basics.html
[2] : https://docs.djangoproject.com/en/1.6/topics/db/models/#relationships
[3] : http://symfony.com/doc/current/book/doctrine.html#entity-relationships-associations

[aldeed]

Good topic, @mquandalle. You're correct that I've already been coming up with ideas about this. I just pushed a new file, RELATIONSHIPS.md, that outlines my current line of thinking, which should answer all of your questions. I'm very interested to hear others' ideas because I'm almost certain that there are use cases not yet supported or not yet clearly documented in that file. Feel free to do PR to master branch on that markdown file, too. My goal is to perfect the documentation before doing the development.

I consider this primarily a collection2 feature, although a few tiny changes might be necessary in simpleSchema. SimpleSchemas already support nesting, so this will be sort of an advanced variation of that with specific handling for DB operations. Obviously autoform changes will be necessary, too, but they should hopefully be minimal if we're able to pull off the API I propose.

[aldeed]

I should mention that the plan doesn't cover finds yet. I think this could be implemented without find support initially, and would still be very useful. But feel free to contribute your thoughts on additional "find" options that would support joins.

[mquandalle]

Thanks for the quick answer and the RELATIONSHIPS.md file :-)

Colors = new Meteor.Collection2('colors', {
  schema: {
    name: {
      type: String
    },
    cars: {
      type: [Cars],
      optional: true
    }
  }
});

I like this API for the relationships, the type: [Cars]. I was actually thinking of the same thing (among other) while writing my previous message. I just wonder if we can really handle all cases with this API.

One other remaining question is how much compatible those relationships are with the other options. For instance you used optionnal, some other options may makes sense as well:

• ? label
• ✓ optional
• ✗ min
• ✗ max
• ✓ minCount
• ✓ maxCount
• ? allowedValues (based on the entire related doc?)
• ? valueIsAllowed
• ✗ decimal
• ✗ regEx
• ✗ unique
• ? autoValue
• ? denyInsert
• ? denyUpdate

Then there is the transaction implementation in order to avoid inconsistencies in the database. Maybe this is something that should be considered as separate project that is nearer to minimongo than collection. And maybe the logic has already been implemented in projects like Mongoose.

About find, I agree that relations could be implemented without it first. Then find simply have to joins related docs as nested objects in the field dedicated to the relation. Find could also have a bool option to only retrieve the _id, or to automatically subquery related docs as well.

[aldeed]

• label would still be useful. For example, you might have a "Vehicle color" label that autoform would use along with a select control.
• allowedValues and valueIsAllowed would test against just the foreign _ids that are actually stored, so you could limit which foreign documents can be linked. The related doc object would be passed through and validated against the foreign collection/schema.
• denyInsert and denyUpdate would probably work the same without any modification, but I'd have to give that some thought.
• I think autoValue could be useful to provide a default foreign _id. For example, you could have them selecting a car color, but then use autoValue to specify a default color (an _id from the Colors collection) if they don't pick one.

The idea of a separate "transactions" package is appealing. Would be great if someone suddenly created it or found something existing. :) I'll have to do some searching.

[aldeed]

Might be able to borrow some from this outdated package for transaction support. MongoDB can't possibly support 100% safe transactions, so any solution will have slight risk of data corruption.

[mquandalle]

Minimongo now works on both the server and the client, so maybe minimongo (or a wrapper around) could implement the transaction mechanism (without communication with the "real" mongo database) and then if everything went fine, sync with mongo?

But it's a huge topic on it's own...

[mquandalle]

Or there is RethinkDB ;-)

[craig-l]

This would be so great to have..

✓ label
✗ optional (Use minCount instead... minCount:0 means its optional)
✗ min
✗ max
✓ minCount
✓ maxCount
✓ allowedValues
✓ valueIsAllowed
✗ decimal
✗ regEx
✗ unique
✓ autoValue
? denyInsert
? denyUpdate

Hopefully this makes sense (I'm relatively new to this stuff)..
allowedValues... I think any implementation would be too simplified to be useful but you could do allowedValues = [['key1',''value'],['key2','value']].. with key1 and 2 being fields of the linked object.

valueIsAllowed... Pass thru doc and linkeddoc so we can do complex queries..
e.x. manufacturer x makes only red and blue color cars... If I'm adding a car from manufacturer x I should only be able to specify red and blue as the color.... So to do that I could query manufacturer x's linked colors and check if we have a match.....

The next evolution of this is improving on what https://github.com/erundook/meteor-publish-with-relations is doing... Take a look at https://gist.github.com/erundook/5012259 to see what I mean because we don't want to have to define relationships all over again when we're publishing... they're defining so many nested relationships there... If we have the relationships centrally here on each object the publishWithRelations can happen semi-automatically (just specify which collections should be included in the publish, but the relationships are already known)

Right now I'm hacking relationship key syncing using collection-hooks.

I wish I could give more specific feedback but this is the best I can do... as I said I'm new to this.

[aldeed]

@levycraig, regarding allowedValues, I think it would be easier to use valueIsAllowed for complex checking based on non-ID keys in the related object. Alternatively, allowedValues could be updated to optionally accept a function that returns the array of allowed values. This would be useful (different from valueIsAllowed) because it could be used by autoform to make automatic selects. That way you could do a complex lookup against the related collection and return an array of the matching IDs.

valueIsAllowed could merge in all linked docs, but even if it doesn't, you could do your own sub-queries within the valueIsAllowed function.

Regarding publish-with-relations, it seems like that would be mostly related to the finds/joins portion of this development, which would be phase 2. It's possible that collection2 pkg could use the publish-with-relations pkg to auto-publish related docs to support joined finds, but something similar but more custom might be necessary. Not sure if @erundook has any thoughts.

[craig-l]

That's great.. Giving the option to pass a function into allowedValues would be an excellent way to go IMHO... killing a bunch of birds with one stone.

I can think of so many uses for that when it comes to autoform.. We could make a 'manufacturer' select which would automatically update the 'model' select.. and then a 'color' select.. all based on the allowed values... (no models will be returned/allowed because manufacturer has not been set).

Or a real life example for me... I want to add a 'sales order' and select the 'customer'.. but customers need to have a status of 'active'.. All I need to do is make sure that the 'customer' field's allowedValues only returns an array of active customers.

Just not sure of all the implications / what is required next for making the selects (like where does the data come from on the client.. because I'm sure you would want to allow us in autoform to decide which field/fields show up in the select)... Maybe the autoform selects can come in phase 2 along with an auto-publish functionality.

[aldeed]

It's important to keep separate the concerns of each package and each schema purpose (validation vs. form generation). The options for autoform select elements can already be set to whatever you like, so I don't think it's critical to add a bunch of code trying to make allowedValues work for autogeneration. That said, it's obviously nice to avoid writing the helpers for all of the options you need, so if there's an easy way to specify which foreign property should be used as the option label, then it might be worth doing. Will have to think about the best way to do that. Maybe just another attribute on the afFieldInput helper (options="allowed" optionLabel="firstName"), which by default looks for "name" property.

[matteodem]

To join your conversation, how easy-check solved the problem of referencing collections is by having an additional property called "references".

See an example here: https://github.com/matteodem/meteor-easy-check#possible-configurations-to-a-field

This allows the type to further be a Javascript Data Type while also being a reference to a certain field in the defined collection you want to reference to.

[aldeed]

Thanks, @matteodem. Do you have an example of when it would be useful to have a data type and a reference? In our case, if we require that the type be a Collection2 (or array of Collection2s), then we can determine the type automatically by looking at the schema for the referenced collection.

Furthermore, if we only allow referencing the _id field, then the type will always be either String or an instance of Meteor.Collection.ObjectID, and again we can determine which it is by looking at the referenced collection's schema, using String as default.

So far nobody has come up with a use case for referencing a field other than _id. If there isn't a use case, then we'll skip the field option.

[matteodem]

Now that you ask me I can't really think of a specific case. It could happen in very specific use cases I guess, but as long as there's no concrete example by anyone it'll be the best to reference the _id of the collection.

It'd be also a simpler solution, which I really like.

[aldeed]

Whenever I think of a weird use case, I'm able to rewrite the model to use ids. I'm nervous I might be forgetting something obvious, but even if we go down the simple road of using type, it would be possible to add support for non-id fields on top of that if the need arises.

I might try to do some prototyping of this in a separate branch so that we can try it out. Unless anyone else is interested...

[matteodem]

How's the progress on this? I'll gladly help you on this if you want me to.

[aldeed]

I'd be happy to have the help. I wouldn't be able to start development for another week or two, so if you want to fork and see what you can come up with sooner, that would be good.

Here's a summary of my plan:

• Use the unwrap branch as the base (i.e., Meteor.Collection rather than Meteor.Collection2)
• Use the design in RELATIONSHIPS.md (look for a Meteor.Collection or array of them as the type value; set up relationships tracking based on that)
• Get insert/update/remove working first. There are two pieces to this: validation of related-schema objects and cascading insert/update/remove with rollbacks.
  • For validation, my thought is to update the simple-schema package with something like: if type is an object instance, see if it has a simpleSchema() method, in which case do sub-validation and merge the results.
  • For cascading and rollbacks, I'm assuming you can figure something out based on your work with easy-check. One thing to note is that we should support cascading to schema-less collections, too. If schema-less (i.e., has no simpleSchema() method), then do the insert/update/remove without validation.
• Second, I'd like to try to tackle find joins in an elegant way, too, perhaps something based on the "publish-with-relations" package, such that it would automagically work in publish functions, too. (Since we have the schema available, many of the options available in "publish-with-relations" are already known for us, meaning that the API can be much simpler. I think we'd only need a listing of field names to join with filter, limit, and sort for each.)

That's a lot, so it's probably best to get a start on the first parts and then review and re-evaluate to make sure we're on the right track.

[aldeed]

I created Meteor-Community-Packages/meteor-simple-schema#46 for the simple-schema piece.

[matteodem]

Okay, I'll have a look at it tomorrow or on Friday, thanks!

[mquandalle]

I know that insert/update are not the first step, but still here is the current spec:

// Assuming:
// car: { _id: "4rjv989j4r0v", manufacturer: "h923nrfp9834", color: "9n3ef890n34" }
// color: { _id: "9n3ef890n34", name: "Blue" }

// This:
Cars.update(id, {$set: {color.name: "Red"}});

// Results in this:
// car: { _id: "4rjv989j4r0v", manufacturer: "h923nrfp9834", color: "9n3ef890n34" }
// color: { _id: "9n3ef890n34", name: "Red" }

I'm not sure it's a good idea to modify the name attribute of document 9n3ef890n34. Other cars may be linked to this color as well and I want to keep them "Red". In this case I think that the expected behavior is to link to the color "Blue" if this color already exists, or to create a new color with a new id and linking this new "Blue" with the car.

---

Not related, I have found a package that pretends to implement collections relations, I guess this is relevant: madmaniak/meteor-model

[aldeed]

That's a really good point @mquandalle. Unfortunately, I could see someone wanting it to actually update that document, too, in the case where the linked document is not so much a category list. For example, when a single autoform allows you to update your profile, including your address, but address is stored in a separate collection linked to profile.

Similarly, along with an insert, we might sometimes want to link to the existing "Red" document and other times want to create a new "Red" document.

So maybe insert/update need to accept an option that allows changing this behavior?

[aldeed]

Or maybe just a cascade: true option for the schema field definition.

[testbird]

The meteor-model looks interesting, as it draws from an already widely used model. [maybe it could even be used with @set_collection2() ]

Concerning the RELATIONSHIPS.md, I could imagine it may be good to always use arrays to store relations, even if the schema defines a one2one relation (restriction), as that keeps changing it trivial.

Maybe, explicitly defining the type of relation could also make it clearly defineable how the related documents are edited by default. hasOne, definesOne, linksOne containsOne?

Concerning autoform, maybe it could be most flexible to have forms defined per collection/schema (quickform and custom forms), and let them include forms from related collections/schemas as subforms.

Another oppoach concerning the db format (and a basis for transactions and rollback) may be to store the links between documents as atomical documents in a separate collection (also useful to track the status of usynchronous updating of the actually related documents). http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits
This would also allow to give the relations their own names and properties (having a relations collection .js definition and a links collection).

[testbird]

So maybe insert/update need to accept an option that allows changing this behavior?

Hm, isn't it suffiecient to either use Cars.update() or Colors.update(), depending on what one want's to do?

along with an insert, we might sometimes want to link to the existing "Red" document and other times want to create a new "Red" document.

The template may render the wanted one or both? Manually, or whether this document definesOne other child document or just linksOne?

[testbird]

To make it simple, it could be a convention that editing a relation only adds, removes or modifies links, never the referenced document (but may create new link targets).

Editing a referenced document would require opening that document for editing.

For example, a form may render a subform that opens and offers to edit the linked documents.

[testbird]

Opened issue #54 to compare design practice ideas.

[matteodem]

Is this now still open for discussion, I didn't have time last week but I'd start soonish if there's a definite decision

[meticulo3366]

Hi, any progress made or examples on how to use relationships? I have used it in Mongoose and SQL databases. Looks like a natural course of development for collection2 and autoform...

https://github.com/meticulo3366/meteor-collection2/blob/master/RELATIONSHIPS.md

This also caught my attention specifically in the README.md file below,
https://github.com/aldeed/meteor-collection2#why-use-collection2

The aldeed:autoform package can take your collection's schema and automatically create HTML5 forms based on it. AutoForm provides automatic database operations, method calls, validation, and user interface reactivity. You have to write very little markup and no event handling. Refer to the AutoForm documentation for more information.

is it possible to implement the above? sounds like rails in ruby or cakephp or sails js...

[rdewolff]

Hi all!
Just went through this long and very interesting topic.
Having relationships implemented would be an amazing plus!
Looking forward to have it!
Cheers!

[benstr]

@meticulo3366 & @rdewolff anything implemented would essentially be the same as the example above. reywood:publish-composite (and a few others like it) have become somewhat of the standard from what I have seen. @aldeed has done a great job supporting his packages, the effort has to be fairly large and time consuming... Just saying, it is probably best for us all to get acquainted with reywood:publish-composite, instead of waiting for collection2. Especially with all the DB changes coming to Meteor soon.

[metalik]

thanks @benstr you saved my day~~!!!

[benstr]

@metalik 👍

[AlexFrazer]

+1, very support. For an "initial draft", I think it could be cool to just make it short hand

classes: {
  type: ForeignKey(Classes),
  autoform: {
    foreignKeyLabel: 'name'
  }
}

which could basically just be

classes: {
  type: String,
  allowedValues: function() {
    return Classes.find().map(doc) {
      return doc._id;
    }
  },
  autoform: {
    options: function() {
      return Classes.find().map(doc) {
        return { label: doc.name, value: doc._id }
      }
    }
  }
}

[dandv]

Looks like @jagi's Astronomy already implements relationships.

[lukemt]

DB changes coming to Meteor soon.

@benstr
Let me know a little bit more about this.

[brylie]

The Astronomy relationship syntax seems pretty simple.

relations: {
    addresses: {
      type: 'many',
      class: 'Address',
      local: '_id',
      foreign: 'memberId'
    }
}

However, the Astronomy packages, in general, do not seem well documented.

[petermikitsh]

I think a sugared syntax (essentially what @AlexFrazer suggested) would be awesome.

[omeid]

Having this feature would be awesome and the planning seems to be going great.

However, for the time being, what is the "right" way to handle this? How do you guys go about it? I would appreciate some sort of guide or write up on this topic.

[omeid]

And +1 for the @AlexFrazer suggestion. It is a very "natural" way to go and can be improved behind the scene overtime.

[carlosbaraza]

Any news on this? I would appreciate to receive some guidance about how to achieve something similar without this feature, like @omeid suggested.

[brylie]

@carlosbaraza check out the collection helpers package. It can be used to design methods to retrieve related documents.

E.g. you can store the remote document ID in a remoteDocumentId field and write a remoteDocument helper to traverse the collection based on the remoteDocumentId field.

[carlosbaraza]

Hi @brylie,

Thank you for your answer. It is a very interesting package.

[0o-de-lally]

@aldeed Haven't seen your comment here in a while. Can you update us on your thoughts for this. Thanks in advance, and keep up the good work.

[couzic]

For newcomers mostly

After reading this long thread and trying it out for myself, I settled on using both reywood:publish-composite and dburles:collection-helpers.

For example, imagine you have a list of tasks, and you want to display their text AND their owner's name, stored in another document. Also, you want your client to update whenever an owner's name changes. In other words, you want an "atomic" publication consisting of both the task and its owner. This is where publish-composite comes into play :

Meteor.publishComposite('tasks', {    
    find: function () { // Return a cursor for the list of tasks
        return Tasks.find();
    },
    children: [
        {
            find: function (task) { // Return a cursor for the owner of each specific task
                return Meteor.users.find(task.ownerId, {fields: {username: 1}});
            }
        }
    ]
});

So now, we have all the data published to the client. And every time an owner name changes, any piece of UI watching for changes to the list of tasks will be reactively updated !

Now, to conveniently get the owner from a specific task, we use use collection-helpers :

Tasks = new Mongo.Collection('tasks');
Tasks.helpers({
    findOwner() {
        return Meteor.users.findOne(this.ownerId)
    }
});

Finally, just get the owner's name :

Tasks.findOne().findOwner().username

If you want to see it in action, checkout this small project

[carlosbaraza]

Hi @micouz,

Thank you for the explanation, that is quite useful. I was also using dburles:collection-helpers, keeping the id/ids of the nested collection objects in the main collection objects.

Cheers,
Carlos..

[benstr]

Nice @micouz ! It would be useful on long issues like this one for GitHub to allow for sticky comments.

[BodhiHu]

awesome 💯

[takahser]

I have an issue regarding collection2 with relationships and autoform.
I try to implement an 1:n relationship, where each object has exactly 1 objectType, while to each objectType multiple objects can be referred to.

My schema looks as follows:

// register collections
Objects = new Mongo.Collection('objects');
ObjectTypes = new Mongo.Collection('objectTypes');

// define schema
var Schemas = {};

Schemas.ObjectType = new SimpleSchema({ // object type schema
    name: {
      type: String
    }
});

Schemas.Object = new SimpleSchema({ // object schema
    type: {
        type: ObjectTypes.Schema,
        optional: true
    },
    title: {
        type: String
    }
});

// attach schemas
ObjectTypes.attachSchema(Schemas.ObjectType);
Objects.attachSchema(Schemas.Object);

My autoform looks like this:

{{> quickForm collection="Objects" id="insertTestForm" type="insert"}} 

I actually would expect a select option field for my type attribute, however, a text input appears. Anyone knows why?

According to the documentation [1], it should be a select option field:

If you use a field that has a type that is a Mongo.Collection instance, autoform will automatically provide select options based on _id and name fields from the related Mongo.Collection. You may override with your own options to use a field other than name or to show a limited subset of all documents. You can also use allowedValues to limit which _ids should be shown in the options list.

[1] https://github.com/aldeed/meteor-collection2/blob/master/RELATIONSHIPS.md#user-content-autoform

[luixal]

Yep, an update on this would be great! :)

[alvarolorentedev]

@takahser I think that what is on Relationship.md is just a vision and it doesn't reflect the current implementation on his 100%. But i agree that displaying a dropdown in quickform when referencing another will be a really powerful thing.

I agree also with what is defined in the .md file it should be a reference to the collection and not to the schema. Should look like this:

ThisCollection = new Mongo.Collection('thisCollection');
OtherCollection = new Mongo.Collection('otherCollection');

OtherCollectionSchema = new SimpleSchema({ // object type schema
    name: {
      type: String
    }
});

ThisCollectionSchema = new SimpleSchema({
    type: {
        type: OtherCollection,
        optional: true
    },
    title: {
        type: String
    }
});

OtherCollection.attachSchema(OtherCollectionShema);
ThisCollectionSchema.attachSchema(ThisCollectionSchema);

[luixal]

@kanekotic @takahser that's what I'm doing right now using one of the selectize autoform packages around.

Also selectize allow you to easily make that selector multiple (really powerful when you have a document related to several docs in other collection stored as an array of IDs) :)

[Mapuia]

Do we have any update on relationships like one-to-many, many-to-many etc. defining with has_one, has_many, belongs_to etc..

[sjkdev21]

I was also having trouble with this issue, and needed a quick and easy way to add a reference (or references) to documents in another collection. I quickly put together a package called "autoform-relations" to help with this. It uses meteorhacks:searchsource to provide a reactive search against specified fields in the other collection and allows you to add or remove the desired documents. What actually get's stored is an [String] of document ids.

I put it together quickly but if people find it useful I'm happy to add more to it https://github.com/oohaysmlm/autoform-relations

I use publish-composite to create publications after I've linked the documents up with this custom autoform input.

[thebarty]

Hi guys,

I think those relations should be defined in SimpleSchema/Collection2.

I have spend some time on a write-up for a feature to denormalize those relations and put it into a readme. This is a proposal and no finished package... it might serve as an inspiration though https://github.com/thebarty/meteor-denormalization

[Lukasvo]

You're going to want to check out Grapher