An experimental, declarative synchronizer for iTwin connectors that aims to modify the existing types in the iTwin library as little as possible. For use as a drop-in replacement of the synchronizer in iTwin's connector framework.
- Motivation π
- Getting started π±
- More iModel things π
- Growing taller π²
- Extending
fir
𧩠- Docs π
Like the element properties you already know from iTwin.
const linkPartition: Element<InformationPartitionElementProps> = {
classFullName: LinkPartition.classFullName,
code: Code.createEmpty(),
model: 'repository',
parent: 'root subject',
meta: partitionMeta,
description: 'models my links',
to: toElement,
};
The iTwin connectors team already maintains two connector libraries.
Why do we need a third?
I've been rather vocal about the usability of the API of the synchronizer that ships with @itwin/connector-framework
. This is an internship project and I have no expectation that it be maintained. I had an idea for how I wanted a synchronizer to work and I also wanted to use it, and that required writing it.
fir
is only a synchronizer. Its synchronization module is just under 1,000 lines of code including whole paragraphs of JSDoc and ample whitespace. fir
makes no attempt to be a complete solution to connectors like pcf
, and it doesn't help you run your connector like connector-framework
. It's designed to be dropped into the connector framework, and doesn't replace it.
- Type safety. The iTwin API represents elements as bags of properties that are just plain object types in TypeScript. When inserting elements into an iModel, all of these types are narrowed to
ElementProps
. These objects already have relationships to each other through ECInstanceIds, which are simply the IDs of rows in an iModel's underlying SQLite database. It seems natural then to use these objects as an intermediate representation of an iModel and hang strings between them to relate them instead of IDs. This is exactly whatfir
does. By expanding iTwin's existing types, you'll never mistake the type of a property. Your connector won't compile. This is in contrast topcf
's approach, which trusts that the author ofmodifyProps
won't make a mistake when escaping TypeScript's checks withany
. - Automatic dependencies.
fir
manages dependencies for you. If you forget to give an elementA
to thesync
method, that's okay! If an elementB
is synced, all of its dependencies will be inserted into the iModel. IfB
is a child ofA
, for example,A
and all of its dependencies will be inserted. These are the strings I mentioned above, but more formally they're relationships, called navigation properties in BIS. Scope relationships, repository relationships, parent relationships, model relationships. - Models. Unlike
pcf
andconnector-framework
's synchronizer,fir
places no restrictions on models. Models can contain models arbitrarily deep. The current synchronizer does not operate on models and relies on the connector author to manage them. Inpcf
you can only have models that model partition elements that are children of the root subject. - Extendibility. If an object type has a relationship property that I forgot to handle,
fir
will give control to the connector author to map the element to its "props" type in iTwin.
- For ease of use, like
pcf
,fir
does place constraints on the shape of your iModel. There are four.- Think of the
ElementProps
type as having anexternalSourceAspect
property instead of theExternalSourceAspectProps
type having anelement
property. This allowsfir
to walk the intermediate iModel tree and find and insert all of the dependencies of thesync
argument. - Circular dependencies are impossible to write declaratively. If any of the strings you hang between your object types together form a circle, you can't represent that iModel with
fir
. This becomes a problem when (1) is considered. Usually all of the dependencies point up the tree towards the root; a parent must be inserted before its child so the child knows its ID. But now elements point towards their external source aspects. Consider an elementA
which points to its external source aspect, which points to an external source, which points to a repository. If this repository is contained in a link model, and its modeled element isA
, that's a cycle. It may be necessary to design around this constraint by having separate sources that are not logically separate. You may also omit the external source element if the element is entirely programmatically generated. For example, in the diagram of the cycle below, it may make sense for the link partition to not belong to an external source. - Scope paths must terminate at the root subject. If an element
A
is scoped to an elementB
through its external source aspect, thenB
is a dependency ofA
, and we must knowB
's ID before we can insertA
. Because every element infir
has an external source aspect, we need to locateB
before we can insert it, which requires knowing its scope. Unless we eventually resolve an element's scope to the root subject with known IDIModel.rootSubjectId
,fir
will never terminate as it tries to locate each element in the chain. - Elements must have exactly one external source aspect. This is not a technical limitation. I just haven't implemented it yet.
- Think of the
npm install --save fir-for-connectors
Now install fir
's peer dependency.
npm install @itwin/core-common
You'll only need @itwin/core-common
because fir
makes use of those types in its own types.
import type { Element, Meta, Model } from 'fir-for-connectors';
import { Sync, toElement, toModel } from 'fir-for-connectors';
Let's say we want to add a link to our iModel. We know that the corresponding BIS element is a BisCore:UrlLink
. We search the iTwin API for url link and see UrlLinkProps
. UrlLinkProps
extends ElementProps
, so we can use it with fir
. Generally, anything with a props at the end is fair game. We're adding an element, so we define an object of type Element
and feed it the props we found.
const nationalGeographic: Element<UrlLinkProps> = {
classFullName: UrlLink.classFullName,
code: Code.createEmpty(),
model: linkModel,
meta,
description: 'the homepage of national geographic',
url: 'https://nationalgeographic.com',
to: toElement,
};
This type should look familiar if you're used the iTwin API before. It's nearly identical to UrlLinkProps
. There are a couple important differences.
The model
property isn't an ID, an Id64String
. It's another object.
const linkModel: Model<ModelProps> = {
classFullName: LinkModel.classFullName,
parentModel: 'repository',
modeledElement: linkPartition,
to: toModel,
};
linkPartition
isn't shown, it's just an Element<InformationPartitionElementProps>
. It's okay if the props type doesn't match up with BIS element you want. Just use the closest one, the youngest ancestor props type of the BIS element. You'll see a LinkPartition
if you're searching the iTwin API. It doesn't end in props so we can't use that directly. (There are ways to use the iTwin class types with fir
, we'll talk about that later.)
Let's go back to the URL link.
There's a strange property called to
. This tells fir
how to convert its weird node type to ElementProps
. It's boilerplate most of the time, but it also makes fir
extendible and allows us to become the synchronizer if you need more power. We'll come back to this.
Try deleting the to
property. If you're using an editor with a language server, it will show a bunch of red squiggles and your code won't compile. This is TypeScript in action telling you the to
property is required. Same deal with code
, a property of UrlLinkProps
. Try deleting that too and see what happens.
If you hover over the squiggle, you'll see a verbose and worrying error message from your friendly compiler.
*a few lines of noise*
Property 'to' is missing in type '*noise*' but required in type 'ElementPatch'. ts(2322)
nodes.ts(53, 5): 'to' is declared here.
That's okay, fir
does some weird type stuff to make it work that probably could be done better if I knew how to write TypeScript. All we need is the very last line. It says the property to
is missing!
The other property that's not in iTwin is meta
. This is what fir
calls external source aspects, another "thing" in your iModel, similar to an element, where the metadata of your elements is stored, like their versions and checksums. fir
uses this information to see when an element has changed.
const meta: Meta = {
classFullName: ExternalSourceAspect.classFullName,
scope: linkPartition,
anchor: 'national geographic url',
kind: 'json',
version: '1.0.0',
};
If you ever want to refer to the root subject, use 'root subject'
. If you want to refer to the repository model, use 'repository model'
.
The anchor
property is a unique ID for your element, so fir
knows how to find it in the iModel. It's the same thing as identifier
in BisCore:ExternalSourceAspect
.
URLs don't usually have parent elements, but if your want to give your element a parent you can do that with the parent
property. We give it either an Element
or an { element: Element, relationship: string }
. Use the latter form if you want to specify the type of parent-child relationship. If you use the first kind fir
will try to guess, but it will probably use BisCore:ElementOwnsChildElements
.
Okay, we're almost done! All that's left is to tell fir
to sync our element. To do that, we need a synchronizer. imodel
is your IModelDb
.
const fir = new Sync(imodel);
fir.sync(nationalGeographic);
That's it! If you change that version number because you made a patch to your iModel, say to 1.0.1
, fir
will update the element. Otherwise it will skip it.
There's one more useful method you'll need to know. If you
const id: Id64String = fir.put(nationalGeographic);
you'll get the ID of the link in the iModel, the ECInstanceId. You can feed it to other functions in the iTwin API that do useful things, like define relationships. put
works like the shell program touch
. It puts an element in the iModel if it doesn't exist and returns its ID. It will never update the element. Use sync
for that.
Now that we've got the synchronization all done, we need to ensure that our source data remains the 'single source of truth' for our iModel. This means that each we can define a bijection between the source objects and the iModel objects. For example, if we change the modeled element of a model in the source, we expect the model to be moved in the iModel, and not copied.
fir
will not do this for you unless you tell it to.
fir.trim('root subject');
The trim
method takes a subtree of the iModel, a branch, and deletes all of the elements and models that weren't seen during the lifetime of the Sync
object and whose children were not seen. The root subject is a good place to call this because most of the iModel descends from it unless you have elements floating in the cytoplasm of the repository model.
Make sure to clean up those too, like repositories and external sources.
It may take multiple passes to remove untouched elements from the iModel depending on how the iModel is traversed. Geometry can prevent definition elements from getting cleaned up during a first pass for reasons I don't entirely understand.
Absolutely! Take a look at the integration folder, which has test-connector.ts
. It's the same test connector in connector-framework
but it's written in fir
. Currently it's hard-coded in version 1.0.0
so fir
won't actually update the elements.
fir
supports these iModel things.
- Elements with the
Element
type. - Models with the
Model
type. - Aspects with the
Aspect
type. UseMeta
for external source aspects. Note theaspects
property onElement
. The caveat is that aspects cannot have navigation properties until the iTwin API allows you to get their ID. - Link-table relationships with the
Relationship
type. - Navigation properties by extending or escaping the library.
fir
comes with the common ones, like parent-child relationships and element-model relationships.
These take a little bit of care. They have an anchor
property for provenance. Let's look at an example.
const ship: Relationship = {
classFullName: 'bis:ElementRefersToDocuments',
source: circusTent,
target: drawingOfCircusTent,
anchor: 'circus tent to drawing',
};
circusTent
and drawingOfCircusTent
are elements. Feeding a relationship to put
will insert the relationship into the iModel. If you change the relationship's class, source, or target and call put
again the relationship will move in the iModel. put
will only update the relationship if this triple changes, even if you add additional properties to the relationship (isn't it odd that BIS supports properties on link-table relationships?) because put
only inserts objects if they don't already exist. In our case, the relationship already exists because its identifier in the source (anchor
) hasn't changed, but the relationship in the iModel no longer matches. It would be confusing for put
to give you the ECInstanceId
of a relationship that isn't represented by the source Relationship
, and so we consider the relationship "new".
If you want to update a relationship's properties, use sync
. sync
will also move the relationship if necessary.
Link-table relationships in fir
don't support a to
property. This is because the property is boilerplate. It makes sense for elements because they frequently have navigation properties, but navigation properties for relationships seem like an unusual use case. If this turns out to be a design mistake, the fix is easy. Right now any additional properties are passed directly from Relationship
to the backend. You can always escape the library with put
to add navigation properties to relationships.
Syncing an element requires specifying an awful lot of properties that probably seem redundant. Why should I have to define the BIS class of the element I want to insert? Or the code? Doesn't iTwin know how to make these things for me?
The 'props' types that we've been using offer a thin wrapper around the underlying database, by design. For a better experience, the iTwin authors made class types that do a lot of this work for you. They usually have a create
method or a constructor function. Let's take a look at an example from the test connector in this repository.
const category = SpatialCategory.create(
fir.imodel, fir.put(definitionModel), 'TestConnector'
);
const props: Element<CategoryProps> = {
...category.toJSON(),
model: definitionModel,
parent: undefined,
meta: meta('Category', '1.0.0', repository, source),
description: "I don't know what this root category is for yet.",
rank: Rank.Application,
to: toElement,
};
When we touch the iTwin APIs we have to make use of put
, because the iTwin APIs talk in IDs. Notice we don't specify the class name or the code. Instead, after we construct the class type, we use toJSON
to convert the class type into CategoryProps
, then the spread ...
operator to add it to fir
's element type. If you stop there you have an object of type CategoryProps
. We have to overwrite the stuff fir
needs to know about, like the model that contains the element. Then we can add properties the class type doesn't define, like the category description.
The parent property looks weird, because it's undefined
. At runtime, TypeScript can't know that SpatialCategory
doesn't actually use that parent property. All it knows from the type of toJSON
is that it could use it and that its type is Id64String
. We have to tell TypeScript that this property actually has the type fir
expects, which is an optional Element
. We explicitly write undefined
, because this optional property is (from my understanding) equivalent to Element | undefined
. This is structural typing after all, and a missing property is a different structure.
Let's say we want to add a Bis:ExternalSourceAttachment
to our iModel. I don't know what this is, but it has a navigation property so we can't yet use it with fir
. Here's what we need to do.
type ExternalSourceAttachment<P extends ExternalSourceAttachmentProps = ExternalSourceAttachmentProps> =
Omit<ElementNode<P>, keyof ExternalSourceAttachmentPatch>
& ExternalSourceAttachmentPatch;
type ExternalSourceAttachmentPatch = {
attaches?: Source,
to: To<ExternalSourceAttachment<ExternalSourceAttachmentProps>, ExternalSourceAttachmentProps>
};
function toExternalSourceAttachment(sync: Sync, attachment: ExternalSourceAttachment): ExternalSourceAttachmentProps
{
return {
...toElement(sync, attachment),
attaches: attachment.attaches ? {
id: sync.put(attachment.attaches),
relClassName: ExternalSourceAttachmentAttachesSource.classFullName
} : undefined
};
}
The ElementNode
type nestled in there is fir
's element type. The only difference between it and Element
is that Element
can also be 'root subject'
, and we can't use Omit
on a union type because it doesn't distribute across the union.
We define our own intermediate type ExternalSourceAttachment
. Be careful that you don't also have a type of the same name from @itwin/core-backend
. I like to use qualified imports for the iTwin libraries so I don't confuse myself. This type looks horrific, but all we're doing is feeding the 'props' type argument P to fir
's element type, which will construct an element type that fir
knows how to use. Then we apply our own patch to the result, stripping off the old attaches
property and giving it a different type, an external source, called Source
in fir
for easy access.
Finally, we have to tell fir
how to map this new intermediate type to its 'props' type in the iTwin library. This is done with a to
function as I mentioned earlier.
First, we call toElement
, which you'll remember from all the elements we made above. This turns fir
's element type Element
into ElementProps
. We dump it into our ExternalSourceAttachmentProps
and add the attaches
property to complete the type. We make use of put
to get the ID of the external source this element refers to.
It turns out we got lucky with this example. When we call toElement
we're trying to assign our ExternalSourceAttachment
type to an Element
type. This would be a beautiful case of type narrowing if it weren't for the to
properties on the two types. In TypeScript, if you assign a function f
to another function g
by writing g = f
, the function f
must have at most as large a domain as g
, because functions that have type typeof g
give no indication that they do anything with the excess input, like our attaches
property. They may even explode. In our case toElement
will happily dump everything it receives into the ElementProps
.
If the BIS specifications said that the attaches
relationship is mandatory, we'd have a problem. Element
's to
type doesn't allow that property. We can use as unknown as Element
to tell TypeScript that we're sure the to
function will never be invoked without an attachment
property.
The good news is that because of the way props types are implemented, they're always optional. The same type is used as the parameter to insertElement
and updateElement
, and properties that are undefined during an update will be cleared. This ensures that our nodes will always be narrowable to Element
in an extraordinarily hacky way. However, if your navigation property is mandatory, like bis:SubCategory
's parent, and you forget to specify the property on an insertion, the backend will buck you.
That's a lot of work and boilerplate for a new navigation property.
Yeah, it is. There are two solutions.
- Rely on the iTwin library to define referencing relationships. I wrote the test connector without having to define a new intermediate element type. Most BIS classes don't define new navigation properties, and if they do hopefully they have a
create
that does all of that for you. - Use something like
Element<ExternalSourceAttachmentProps>
withtoElement
; remember that only theto
types prevent narrowing. Then just useput
for theattaches
navigation property. Because there's no intermediatefir
type, any additional properties that you give to your element will be handed to the iTwin library.
Dude it's your library. The whole point of
fir
's 'tree' of element types is that each one is a supertype of its parent. Theto
function is bad design because it prevents narrowing when the supertypes aren't directly assignable to their parent, in which case they're not supertypes but overlapping types.
I'm working on it. In the mean time strictFunctionTypes
is the compiler option that's causing this error. There's no way to say to the compiler, hey, I know this thing isn't a supertype, but I pinky swear that the other type we're binding it to isn't going to invoke its to
function without the value we're hiding from its domain.
- urgent! Need to figure out how to design the node types to allow the
to
function to properly narrow; otherwise, syncing is going to be difficult with elements with mandatory properties - Test the published package in
connector-framework
- Support more than one external source aspect
- Are there any class types that insert other elements into the iModel?
fir
won't know - Add node types for physical objects to better support the category navigation property
- Sync element aspects
- What about syncing
RelationshipProps
? Link table relationships should never cause cycles - Document the
trim
method - Trim untethered external sources and repositories. Can we use
trim
? - Trim a model