diff --git a/site/docs/content-packs/cim-content-pack/overview.md b/site/docs/content-packs/cim-content-pack/overview.md index e9fc533250..8bda66eaa0 100644 --- a/site/docs/content-packs/cim-content-pack/overview.md +++ b/site/docs/content-packs/cim-content-pack/overview.md @@ -1,3 +1,8 @@ +--- +hide: +- toc +--- + diff --git a/site/docs/content-packs/simple-content-pack/overview.md b/site/docs/content-packs/simple-content-pack/overview.md index 8af7b7e42a..80fda8c61f 100644 --- a/site/docs/content-packs/simple-content-pack/overview.md +++ b/site/docs/content-packs/simple-content-pack/overview.md @@ -1,3 +1,8 @@ +--- +hide: +- toc +--- + diff --git a/site/docs/content-packs/types-content-pack/overview.md b/site/docs/content-packs/types-content-pack/overview.md index 2952f92bca..ffd6ffcd2d 100644 --- a/site/docs/content-packs/types-content-pack/overview.md +++ b/site/docs/content-packs/types-content-pack/overview.md @@ -1,3 +1,8 @@ +--- +hide: +- toc +--- + diff --git a/site/docs/guides/planning/four-layers-of-choice.svg b/site/docs/guides/planning/four-layers-of-choice.svg index 227e597473..981a9d0538 100644 --- a/site/docs/guides/planning/four-layers-of-choice.svg +++ b/site/docs/guides/planning/four-layers-of-choice.svg @@ -1,4 +1,4 @@ -
Content Packs
Connectors
Configuration
Runtime Libraries
What you want to do
(use cases)
 \ No newline at end of file +
Content Packs
Connectors
Configuration
Runtime Libraries
What you want to do
(use cases)
 \ No newline at end of file diff --git a/site/docs/guides/planning/index.md b/site/docs/guides/planning/index.md index 9e9dd73e69..f5747c56d8 100644 --- a/site/docs/guides/planning/index.md +++ b/site/docs/guides/planning/index.md @@ -3,10 +3,10 @@ # Planning Guide -Egeria is highly modular, allowing you to choose which parts to use as supplied, and which to remove, extend, or replace with something customized for your organization. +Egeria is highly modular, allowing you to choose which components to use "as supplied", and which to remove, extend, or replace with something customized for your organization. Typically, Egeria's function is broader than you need, at least initially, so often the challenge is to know which pieces you need to do the things you want to do, and what you can safely ignore. -There are four types of "content" from Egeria that you need to consider: Content Packs, Connectors, Configuration and the Runtime Libraries. +There are four types of "component" from Egeria that you need to consider: Content Packs, Connectors, Configuration and the Runtime Libraries. ![Four layers of content](four-layers-of-choice.svg) @@ -15,10 +15,20 @@ There are four types of "content" from Egeria that you need to consider: Content * The *configuration* controls the activation of Egeria's services in the runtime. Related services are grouped together into what are called [OMAG Servers](/concepts/omag-server) so they can be activated and deactivated together. Egeria's services are used to build out your knowledge graph. They can be called from the connectors, or other tools and technologies using Egeria's clients, or directly via the REST APIs. Managing configuration is described in the [Administration Guide](/guides/admin). * The *runtime libraries* provide the services to manage the connectors and the knowledge graph. They are packaged into a choice of two runtimes. The [OMAG Server Platform](/concepts/omag-server-platform) is the typical runtime to use, particularly initially, since it can run all functions using multiple OMAG Servers and its configuration can be dynamically changed making it easy to experiment with different settings. The [OMAG Server Runtime](/concepts/omag-server-runtime) is a specialist runtime for a cloud-native deployment that only runs a single OMAG Server using a fixed configuration. The specific software services included in either runtime can be both removed or added at build time to allow you to fully customize Egeria's runtimes. -The *Planning Guide* provides information to help you plan the deployment of Egeria in your organization. As a general rule of thumb, we suggest using the content as supplied and only customize it as you get more confident. To this end, when you build the main egeria repository ([egeria.git]()), it creates a distribution folder containing content packs, connectors, sample configuration and the runtime libraries along with `README.md` files describing how to use this content. The Egeria project also includes additional content, such as connectors, in other repositories that can be built and added to this distribution. The distribution also includes a Docker script file so that, once you have all of the content you need assembled, it can be turned into a docker container for deployment. Alternatively it can be run directly on a local machine running Linux or MacOS (bare metal). +The *Planning Guide* provides information to help you plan the deployment of Egeria in your organization. +## Egeria's distribution -The *Planning Guide* is divided into two parts as follows: +As a general rule of thumb, we suggest using Egeria's components as supplied and only customize them as you get more confident. To this end, when you [build the main egeria repository](/education/tutorials/building-egeria-tutorial/overview) (called [egeria.git](https://github.com/odpi/egeria)), it creates a distribution folder containing content packs, connectors, sample configuration and the runtime libraries along with `README.md` files describing how to use this content. + +--8<-- "snippets/inside-egerias-distribution.md" + +The Egeria project also includes additional content, such as connectors, in other repositories that can be built and added to this distribution. The distribution also includes a Docker script file so that, once you have all of the content you need assembled, it can be turned into a docker container for deployment. Alternatively it can be run directly on a local machine running Linux or MacOS (bare metal). + + +## Designing your own deployment + +The rest of this *Planning Guide* assumes you are familiar with the basic concepts of Egeria and have some knowledge of the contents of the Egeria's distribution. It aims to guide you through the process if customizing your Egeria deployment. It is divided into two parts as follows: * *Planning Deployment* focuses on the setting up of the Egeria software and its supporting technology @@ -26,11 +36,13 @@ The *Planning Guide* is divided into two parts as follows: * [Designing your security model](/guides/planning/security/overview) describes how to build a threat model for Egeria. * [Planning User Interfaces](/guides/planning/user-interfaces) describes how to plan for the two user interfaces that Egeria supplies. -* *Preparing Metadata Ecosystem* describes the types of metadata that should be populated in your open metadata ecosystem to allow it to operate effectively. +* *Preparing Metadata Ecosystem* describes the types of metadata that should be populated in your open metadata ecosystem to allow it to operate effectively. This metadata can be added directly into your metadata repository, or added to your own content packs to load across multiple Egeria deployments. * [Metadata Valid Values](/guides/planning/valid-values/overview) to control the values used in open metadata elements. * [Multiple Language Translations](/guides/planning/translations/overview) to support multiple languages in the metadata elements. * [Onboarding Organization](/guides/planning/organization/overview) to define the people, their roles and userIds and how they are organized into teams. This is used to control access to metadata and associated resources, route stewardship actions to the appropriate individuals/teams and synchronize organizational information between systems. * [Governance Program](/guides/planning/governance-program/overview) to coordinate the various governance activities in the organization. - + +The connectors supplied by Egeria + --8<-- "snippets/abbr.md" diff --git a/site/docs/guides/planning/planning-guide.drawio b/site/docs/guides/planning/planning-guide.drawio index efa1eba062..1ba48a8d0a 100644 --- a/site/docs/guides/planning/planning-guide.drawio +++ b/site/docs/guides/planning/planning-guide.drawio @@ -1,20 +1,20 @@ - + - + - + - + - + @@ -22,4 +22,105 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/docs/guides/planning/sample-configs.svg b/site/docs/guides/planning/sample-configs.svg new file mode 100644 index 0000000000..d85b8919d0 --- /dev/null +++ b/site/docs/guides/planning/sample-configs.svg @@ -0,0 +1,4 @@ + + + +
OMAG Server Platform
    simple-metadata-store
active-metadata-store
view-server
integration-daemon
engine-host
optional
cohort link
XTDB Metadata
Repository
In Memory Repository
REST API
Apache Kafka Event Broker
REST API
External program or User Interface
Open Metadata and
Governance Requests
REST API
REST API
Operational
Requests
 \ No newline at end of file diff --git a/site/docs/types/6/0620-Resource-Profiling.svg b/site/docs/types/6/0620-Resource-Profiling.svg index b62e5fabc9..6e1311a26c 100644 --- a/site/docs/types/6/0620-Resource-Profiling.svg +++ b/site/docs/types/6/0620-Resource-Profiling.svg @@ -1,4 +1,4 @@ -
0620 - Resource Profiling
«entity»
ResourceProfileAnnotation
length : int
inferredDataType : string
inferredFormat : string
inferredLength : int
inferredPrecision : int
inferredScale : int
profileProperties : map<string, string>
profileFlags : map<string, boolean>
profileCounts : map<string, long>
valueList : array<string>
valueCount : map<string, int>
valueRangeFrom : string
valueRangeTo : string
averageValue : string
«entity»
DataFieldAnnotation
«entity»
ResourceProfileLogAnnotation
«entity»
Asset
*
resourceProfileData
*
resourceProfileAnnotations
«relationship»
ResourceProfileData
«entity»
FingerprintAnnotation
fingerprint : string
fingerprintAlgorithm : string
hash : long
hashAlgorithm : string
 \ No newline at end of file +
0620 - Resource Profiling
«entity»
ResourceProfileAnnotation
profilePropertyNames : array<string>
length : int
inferredDataType : string
inferredFormat : string
inferredLength : int
inferredPrecision : int
inferredScale : int
profileStartDate : date
profileEndDate : date
profileProperties : map<string, string>
profileFlags : map<string, boolean>
profileDates : map<string, date>
profileCounts : map<string, long>
profileDoubles : map<string, double>
valueList : array<string>
valueCount : map<string, int>
valueRangeFrom : string
valueRangeTo : string
averageValue : string
«entity»
DataFieldAnnotation
«entity»
ResourceProfileLogAnnotation
«entity»
Asset
*
resourceProfileData
*
resourceProfileAnnotations
«relationship»
ResourceProfileData
«entity»
FingerprintAnnotation
fingerprint : string
fingerprintAlgorithm : string
hash : long
hashAlgorithm : string
 \ No newline at end of file diff --git a/site/docs/types/6/area-6-discovery.drawio b/site/docs/types/6/area-6-discovery.drawio index d78fb06828..cb5b508e78 100644 --- a/site/docs/types/6/area-6-discovery.drawio +++ b/site/docs/types/6/area-6-discovery.drawio @@ -1,6 +1,6 @@ - + - + @@ -396,7 +396,7 @@ - + @@ -660,7 +660,7 @@ - + @@ -773,16 +773,16 @@ - + - + - + - + @@ -839,7 +839,7 @@ - + @@ -853,68 +853,68 @@ - + - - + + - + - - + + - + - + - + - + - + - + - + - + - - + + - + - - + + - + - + @@ -1107,7 +1107,7 @@ - + diff --git a/site/snippets/egerias-distribution.md b/site/snippets/egerias-distribution.md new file mode 100644 index 0000000000..4955cf9ea1 --- /dev/null +++ b/site/snippets/egerias-distribution.md @@ -0,0 +1,10 @@ + + + +Egeria's distribution is created as part of the located in the `egeria/open-metadata-distribution/omag-server-platform/build` directory that is created by Egeria's build. There are two versions, they both contain the same content. + +* `distributions` contains the zipped and compressed distribution file for Egeria's platform. This file provides a convenient package to copy Egeria onto a specific machine to run. See [Installing Egeria](https://egeria-project.org/education/tutorials/building-egeria-tutorial/overview/#installing-egeria) for information on how to unpack this file. +* `unpacked` contains an unzipped copy of the distribution file, useful for running Egeria on the machine where it is built. + +???+ education "Inside Egeria's Distribution" + --8<-- "snippets/inside-egerias-distribution-text.md" diff --git a/site/snippets/inside-egerias-distribution-text.md b/site/snippets/inside-egerias-distribution-text.md new file mode 100644 index 0000000000..9fddde89ff --- /dev/null +++ b/site/snippets/inside-egerias-distribution-text.md @@ -0,0 +1,47 @@ + + + + +At the top of the distribution is a Docker script (called `DockerFile`) that converts the distribution into a Docker container for running in a cloud environment. If Egeria is to run in a cloud environment then this script is run after you have customized the content of the distribution. This customization may involve adding connector implementations and, if it is for production use, stripping out the examples, samples and libraries that you do not need. + +The `assembly` directory contains the real content of the distribution. This is where the content packs, connectors, sample configuration and runtime libraries are located. It is organized into three parts: + +* The `platform` directory includes all the files needed to run the platform. +* The `etc` directory has additional files and utilities that can be helpful when running the platform. +* The `opt` directory has samples and other useful content that can be helpful when experimenting with Egeria. + +To run the platform, change to the `platform` directory and follow the instructions in the `README.md`. + +### Content Packs + +The content packs are located in the `opt/content-packs` directory. They include: + +* [Core Content Pack](/content-packs/core-content-pack/overview) that provides metadata to configure and call the connectors included in the distribution. +* [Cloud Information Model](/content-packs/cim-content-pack/overview) that provides a simple commerce [glossary](/practices/common-data-definitions/overview). +* [Coco Pharmaceuticals Sample Metadata](/content-packs/coco-content-pack/overview) that provides metadata that drives the [Coco Pharmaceuticals scenarios](/practices/coco-pharmaceuticals). +* [Simple Catalog Content Pack](/content-packs/simple-content-pack/overview) that provides a simple asset catalog showing how databases, APIs and Events can be catalogued and linked together with a glossary term. +* [Open Metadata Types Content Pack](/content-packs/types-content-pack/overview) that provides the definitions of the [Open Metadata Types](/types). + +### Sample Configurations + +The `opt/sample-configs` directory contains the configuration for five Egeria servers (called [OMAG Servers](/concepts/omag-server)) that all run on a single instances of Egeria's platform (called the [OMAG Server Platform](/concepts/omag-server-platform)). Server configuration defines which of Egeria's services are active, which in turn defines the types of connectors that can run. The sample configurations are sufficient to run most Egeria use cases. Adding additional servers is often needed just to scale the deployment. + +The servers are: + +* **simple-metadata-store** is a [Metadata Access Store](/concepts/metadata-access-store/) that provides REST APIs for retrieving and maintaining open metadata. This server is set up to use a repository that keeps its metadata in memory. It loads the [Simple Catalog Content Pack](https://egeria-project.org/content-packs/simple-content-pack/overview/). This means that each time the server is restarted, it starts with just the content of the [Simple Catalog Content Pack](/content-packs/simple-content-pack/overview) in its repository. + +The `simple-metadata-store` server is not configured to use Apache Kafka and so it does not produce events when metadata is changed. The next set of servers make use of Apache Kafka to both send and receive events. The Apache Kafka broker should be listening at `localhost:9092`. + +* **active-metadata-store** is a [Metadata Access Store](/concepts/metadata-access-store/) that supports both REST APIs for retrieving and maintaining open metadata along with event notifications each time there is change in the metadata. It is storing its metadata in an XTDB key-value repository stored on the local file system under the `platform/data/servers/active-metadata-store` directory. This means that any metadata that you create will still be in the repository when you restart this server. (The repository can be cleared by deleting the `platform/data/servers/active-metadata-store/repository` directory.) This server automatically loads the [Core ] + +* **integration-daemon** is an [Integration Daemon](https://egeria-project.org/concepts/integration-daemon/) that catalogs files stored on the filesystem. It is set up to catalog any file located in `sample-data/data-files` under the `platform` directory. It is also looking for additional configuration added to active-metadata-store under the **Egeria:IntegrationGroup:DefaultIntegrationGroup** [integration group](https://egeria-project.org/concepts/integration-group/). + +* **engine-host** is an [Engine Host](https://egeria-project.org/concepts/engine-host/) that is running the **AssetSurvey** and **AssetGovernance** [governance engines](https://egeria-project.org/concepts/governance-engine/) used to create and manage metadata. The configuration of these governance engines is found in the active-metadata-store. + +The final server provides the services for Egeria's UIs. + +* **view-server** is a [View Server](https://egeria-project.org/concepts/view-server/) that calls the active-metadata-store to send and retrieve metadata from its repository. Its services are designed to support calls from non-Java environments such as python and javascript. Egeria's user interfaces make calls to the view server. + +These service call one another as follows: + +![Sample server interactions](/guides/planning/sample-configs.svg) \ No newline at end of file