Nested Anchor support and extensions to Data Manager OMAS

[mandy-chessell]

Description

This PR provides an extension to Data Manager OMAS and Database Integrator OMIS that allows the caller to build a schema of tables and columns independently from the database asset(s) and then attach them once complete. This reduces the workload on the open metadata ecosystem because LatestChange classifications are not maintained on the database assets while the tables and columns are being constructed. The original methods are still available so there is no impact on existing callers/connectors.

Database schema types

Database tables and columns are specialized types of Schema Attributes in open metadata. They are connected to a database asset via a RelationalDBSchemaType which is a specialized ComplexSchemaType. Data Manager OMAS has, in the past, managed the RelationalDBSchemaType internally in its createTableXXX and createViewXXXmethods. There are now methods to create the RelationalDBSchemaType explicitly and attach tables and views to it (no change is needed to database columns since they are attached to tables/views). Once the tables/views and their nested columns are created, there is another method to attach the RelationalDBSchemaType to either the database asset or the database schema asset.

For example this is the original code

String databaseSchemaGUID = this.getDatabaseSchema(client, databaseManagerGUID, databaseGUID, userId);

DatabaseTableProperties properties = new DatabaseTableProperties();

properties.setQualifiedName(databaseTableName);
properties.setDisplayName(databaseTableDisplayName);
properties.setDescription(databaseTableDescription);

String databaseTableGUID = client.createDatabaseTable(userId, databaseManagerGUID, databaseManagerName, databaseSchemaGUID, properties);

String databaseColumnGUID = this.createDatabaseColumn(client, databaseManagerGUID, databaseTableGUID, userId);

and this is the new alternative

databaseSchemaGUID = this.getDatabaseSchema(client, databaseManagerGUID, databaseGUID, userId);

String databaseSchemaTypeGUID = client.createDatabaseSchemaType(userId, databaseManagerGUID, databaseManagerName, "SchemaOf:" + databaseSchemaName);

DatabaseTableProperties properties = new DatabaseTableProperties();

properties.setQualifiedName(databaseTableName);
properties.setDisplayName(databaseTableDisplayName);
properties.setDescription(databaseTableDescription);

String databaseTableGUID = client.createDatabaseTableForSchemaType(userId, databaseManagerGUID, databaseManagerName, databaseSchemaTypeGUID, properties);

databaseColumnGUID = this.createDatabaseColumn(client, databaseManagerGUID, databaseTableGUID, userId);

client.attachSchemaTypeToDatabaseAsset(userId, databaseManagerGUID, databaseManagerName, databaseSchemaGUID, databaseSchemaTypeGUID);

Nested anchor support

While the schema type is being constructed, all of the tables and columns are anchored on the RelationalDBSchemaType so that they can be deleted together. These anchors are established as part of the createEntity method.

When the RelationalDBSchemaType is attached to a database asset, its anchor is set to the GUID of the database asset. If the anchor GUID of all of the tables, views and columns where then updated to the database asset, much of the performance savings by building the schema separately would be lost. So the generic handlers have been updated so that it is able to honour nested anchors. This means that if the database asset is deleted, the attached schema type and its tables/views/columns are also deleted - as if they where all part of the same anchor.

I have also prevented LatestChange from being updated when an Anchors classification changes.

Template support

The ability to create elements using templates has been enhanced to support nested anchors. If the template uses nested anchors, these are copied for the new bean. However the copies are given the new top-level bean as the anchor. Effectively the nesting is not preserved in the new bean.

How Has This Been Tested?

New FVT tests have been added to the Data Manager FVT to exercise this function

Any additional notes for reviewers?

[mandy-chessell]

This PR also inculdes corrections to the diagrams used in the automated curation open metadata lab