Skip to content

Commit

Permalink
Merge branch 'master' into master
Browse files Browse the repository at this point in the history
  • Loading branch information
@cstirdivant
cstirdivant authored Nov 22, 2023
2 parents 67f47c4 + ea8db32 commit 433703b
Show file tree
Hide file tree
Showing 6 changed files with 143 additions and 171 deletions.
12 changes: 6 additions & 6 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

The Digital Buildings project is an open-source, Apache-licensed effort to create a uniform schema and toolset for representing structured information about buildings and building-installed equipment. A version of the Digital Buildings ontology and toolset is currently being used by Google to manage buildings in its portfolio.

The Digital Buildings project originated from the need to manage a very large, heterogeneous building portfolio in a scalable way. The project aims to enable management applications/analyses that are trivially portable between buildings. This goal is achieved through a combination of semantically-expressive abstract modeling, an easy-to-use configuration language, and robust validation tooling. Digital Buildings work has been inspired by [Project Haystack](https://project-haystack.org/tag) and [BrickSchema](https://brickschema.org/) and maintains cross-compatibility and/or convergence as a long-term objective.
The Digital Buildings project originated from the need to manage a very large and heterogeneous building portfolio in a scalable way. The project aims to enable management applications and analyses that are trivially portable between buildings. This goal is achieved through a combination of semantically-expressive abstract modeling, an easy-to-use configuration language, and robust validation tooling. Digital Buildings work has been inspired by [Project Haystack](https://project-haystack.org/tag) and [BrickSchema](https://brickschema.org/) and maintains cross-compatibility and/or convergence as a long-term objective.

In creating the Digital Buildings project, we have considered the following:

Expand All @@ -23,16 +23,16 @@ In creating the Digital Buildings project, we have considered the following:

This project is structured as follows:
* An [ontology](./ontology/README.md) that defines the parameters of the semantic data model ("Terminology box") and tools for building, validating, and associating real equipment with a specific model. It contains the following formats:
* [Yaml format](./ontology/yaml/README.md)
* [YAML format](./ontology/yaml/README.md)
* [RDF/OWL format](./ontology/rdf/README.md)

* A [model instance configuration](./ontology/docs/building_config.md) (a.k.a building configuration file) that contains a mapping between the ontology and the "raw" data. Building configuration files are the "Assertion box."
* A [model instance configuration](./ontology/docs/building_config.md) (a.k.a building configuration file) that contains a mapping between the ontology and the "raw" real-world data. Building configuration files are the "Assertion box."
* Tools that enable the following:
* [**ABEL**](./tools/abel/README.md)**:** facilitates easier building configration construction by converting from a templatized Google Sheet to a [building configuration file](./ontology/docs/building_config.md) (and from a building configuration file back to a Google Sheet).
* [**Explorer**](./tools/explorer/README.md)**:** allows users to explore ontology type fields and compare ontology types to each other.
* [**Instance Validator**](./tools/validators/ontology_validator/README.md) which allows to validate the yaml ontology upon a change or an extension.
* [**Ontology Validator**](./tools/validators/ontology_validator/README.md)**:** validates the ontology upon a change or an extension (currently only for the Yaml format).
* [**RDF/OWL Generator**](./tools/rdf_generator/README.md)**:** generates an RDF version from the yaml ontology files.
* [**Instance Validator**](./tools/validators/ontology_validator/README.md)**:** validates a concrete application (instance) of DBO (i.e., a building configuration file) with optional telemetry validation.
* [**Ontology Validator**](./tools/validators/ontology_validator/README.md)**:** validates the ontology upon a change or an extension (currently supports YAML format only).
* [**RDF/OWL Generator**](./tools/rdf_generator/README.md)**:** generates an RDF version from the YAML ontology files.
* An [Internal Building Representation (IBR)](./ibr/README.md) file format to represent data from different verticals such as spatial or assets.

## Learning Modules
Expand Down
99 changes: 51 additions & 48 deletions tools/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,41 @@

Various tools have been developed to support the use of the Digital Buildings Ontology and Building Configuration files.

The tools are:
The tools and their functions are summarized below:
* [ABEL](./abel/README.md) generates from/to Google spreadsheet/[Building Configuration](../ontology/docs/building_config.md).
* [Explorer](./explorer/README.md) allows users to explorer the ontology types and their associated fields.
* [Instance Validator](./validators/instance_validator/README.md) which allows to validate the yaml ontology upon a change or an extension.
* A sub function of the Instance Validator is to also [validate telemetry messages](./validators/instance_validator/README.md#telemetry-validation)
corresponding to entity blocks in a building configuration file.
* [Ontology Validator](./validators/ontology_validator/README.md) which allows to validate the yaml ontology upon a change or an extension.
* [Instance Validator](./validators/instance_validator/README.md) allows users to validate a concrete instance of the ontology (i.e., a building configuration file).
* A sub-function of the Instance Validator is to also validate telemetry messages for each reporting entity in the building configuration file using [Telemetry Validator](./validators/instance_validator/README.md#telemetry-validation).
* [Ontology Validator](./validators/ontology_validator/README.md) allows users to validate a local version of the YAML ontology upon a change or an extension.

## Digital Buildings Toolkit

The Digital Buildings Toolkit provides a centralized
method for interfacing with all of the tools contained within the Digital
Buildings Repository.
The Digital Buildings Toolkit provides a centralized method for interfacing with the following Digital Buildings tools:
* Instance Validator (with optional Telemetry Validator)
* GUID generator

There are currently two methods for interacting with the Toolkit: the Toolkit Web Application (currently in alpha) and the Toolkit Command Line Interface (CLI).

### Toolkit Web Application

The [web-based toolkit application](dbo-toolkit-app.azurewebsites.net) also exists to provide a user-friendly interface to all of the Digital Buildings tools.
Only the Instance Validator is currently supported, but other tools are planned to be added in the near future.
The [web-based toolkit application](dbo-toolkit-app.azurewebsites.net) exists to provide a user-friendly interface to all of the Digital Buildings tools.

**Note:** Currently, the Toolkit Web Application only supports the Instance Validator. The team plans to support other tools in the near future.

### Install
### Toolkit CLI Installation

To install please follow the instructions below.

#### First create a virtual env
#### Create a Virtual Environment

Create the virtual environment with `venv` followed by the environment name, in this example: `tooling`, in the digitalbuildings repository.
First, create a virtual environment with `venv` followed by the environment name (in this example: `tooling`) in the digitalbuildings repository.

```
python -m venv tooling
```


Activate the virtual environment
#### Activate the Virtual Environment

Mac OS / Linux:
```
Expand All @@ -45,16 +47,19 @@ Windows
```
tooling\Scripts\activate
```
#### Install Packages
Next, you can either use pip or setup (to be deprecated) to install the necessary packages and dependencies.

##### Install Pip
1. Run the following command to ensure that your Python package management tools are up-to-date.

Then you can either use pip or setuptools.

#### Pip
1. Run `python3 -m pip install --upgrade pip` to ensure that your Python package management tools are up-to-date.
```
python3 -m pip install --upgrade pip
```

2. Run `bash pip_install.sh` or `pip_install.bat` (windows) from the following directory digitalbuildings/tools.
2. Run `bash pip_install.sh` (MacOS / Linux) or `pip_install.bat` (Windows) from the following directory: `digitalbuildings/tools`.

#### Docker
##### Install Docker (Optional)

1. Install [Docker Desktop](https://docs.docker.com/desktop/)
2. Run `./tools/docker_run.sh` to build the docker image.
Expand All @@ -63,51 +68,49 @@ Then you can either use pip or setuptools.
$ ./tools/docker_run.sh abel
```

#### Setup (to be deprecated)
##### Run Setup (to be deprecated)

1. Follow setup instructions for the [Instance Validator](./validators/instance_validator).
2. Follow setup instructions for the [GUID Generator](./guid_generator).
3. Run `sudo python setup.py` for this directory.

### Toolkit Workflow
### Toolkit CLI Workflow

Run `python toolkit.py` and provide the following arguments:

1. `-i/--input` The absolute filepath of a building configuration file(.yaml).
1. `--input` or `-i`: The absolute filepath of a building configuration file(.yaml).

2. `-m/--modified-types-filepath` **[Optional]** Validate entity types in the building configuration file against a modified ontology that is not in the main repository. Default is the [Digital Buildings Ontology](https://github.com/google/digitalbuildings/tree/master/ontology/yaml).
2. `--modified-types-filepath` or `-m` **[Optional]**: Validates entity types in the building configuration file against a modified ontology (i.e., a local branch) that is not in the main repository. Default is the [Digital Buildings Ontology](https://github.com/google/digitalbuildings/tree/master/ontology/yaml).

3. `-g/--generate` Generates GUIDs for entities in the building configuration file. Since the instance validator expects the [new building configuration format](https://github.com/google/digitalbuildings/blob/master/ontology/docs/building_config.md#new-format), this option also converts a building configuration from the old format to the new format.
3. `--generate` or `-g`: Generates GUIDs for entities in the building configuration file. Since the instance validator expects the [new building configuration format](https://github.com/google/digitalbuildings/blob/master/ontology/docs/building_config.md#new-format), this option also converts a building configuration from the old format to the new format.

4. `-v/--validate` Runs instance validator to validate the building configuration file.
4. `--validate` or `-v`: Runs instance validator to validate the building configuration file.

5. After a building configuration's entity types are validated, validation must also be run on the telemetry payload using:
5. [Optional] Telemetry Validation: After a building configuration's entity types are validated, validation must also be run on the telemetry payload using the following parameters. **NOTE:** The OAuth credential (`-c`) and subscription (`-s`) are provided by the Google team. Please reach out to your IoT TPM for guidance.

* `-s/--subscription` The fully-qualified path to a Google Cloud Pubsub subscription, e.g. projects/google.com:your-project/subscriptions/your-subscription.
* `--subscription` or `-s`: The fully-qualified path to a Google Cloud Pubsub subscription (e.g., `projects/google.com:your-project/subscriptions/your-subscription`).

* `--credential` or `-c`: An absolute or relative path to an OAuth client credential JSON file.

* `--timeout` or `-t` **[Optional]**: The timeout duration in seconds for the telemetry validation test. The default value is 600 seconds, or 10 minutes. If this time limit is exceeded before the validator receives a test pubsub message for each of the entities configured in the given instance config file, the test will fail with an error and report the entities that were not heard from.

* `--credential` or `-c`: Should be an absolute or relative path to an OAuth client credential JSON file.
7. `--report-directory` or `-d` **[Optional]**: Writes instance validation (instance_validation_report.txt) and telemetry validation (telemetry_validation_report.json) reports to the specified `report-directory`. By default, writes instance validation output to the console and telemetry validation output to the current working directory.

* `-t/--timeout` **[Optional]** The timeout duration in seconds for the telemetry validation test. The default value is 600 seconds, or 10 minutes. If this time limit is exceeded before the validator receives a test pubsub message for each of the entities configured in the given instance config file, the test will fail with an error and report the entities that were not heard from.
8. `--udmi` **[Optional]**: Validates entity metadata as [UDMI](https://github.com/faucetsdn/udmi/). Flag is set to `True` by default; change this parameter to `--udmi=False` when not validating against UDMI.

* `--udmi` **[Optional]** Validates entity metadata as [UDMI](https://github.com/faucetsdn/udmi/). Flag is set to `True` by default and include `--udmi=False` when not validating against udmi.
For example, the following input

`python instance_validator.py -i input.yaml` validates a building config against the udmi standard.

* **NOTE:** The OAuth credential and subscription are provided by the Google team. Please reach out to your IoT TPM for guidance.

6. `-d/--report-directory` To write instance validation (instance_validation_report.txt) and telemetry validation (telemetry_validation_report.json) reports to the report-directory; otherwise writes instance validation to console and telemetry validation to current working directory.
```
python toolkit.py -i //path/to/building/configuration/file.yaml -g -v -s subscription-name -c //path/to/oauth/cred.json -d //path/to/report-directory
```

For example:
`python toolkit.py -i //path/to/file -g -v -s subscription-name -c //path/to/oauth/cred.json -d //path/to/report-directory`
1. Takes in a building configuration file.
2. Generates guids for every entity instance.
3. Re-writes building config in the new format.
4. Validates the building configuration.
5. Validates the telemetry payload.
6. Writes validation results to the report directory as //path/to/report-directory/instance_validation_report.txt and //path/to/report-directory/telemetry_validation_report.json for instance validation and telemetry validation respectfully.
results in these actions:
1. Ingests a building configuration file.
2. Generates guids for every entity instance in the buiding configuration file and re-writes building configuration file in the new format.
4. Validates the building configuration file with the instance validator.
5. Validates the telemetry payloads for each reporting entity in the building configuration file.
6. Writes instance and telemetry validation results to the report directory as `//path/to/report-directory/instance_validation_report.txt` and `//path/to/report-directory/telemetry_validation_report.json`.

**NOTE:** The new building configuration format switches entities being keyed by codes
to being keyed by guids, and Ids are removed. To convert from old format to the
new format, run your building configuration file(.yaml) through the [guid generator](https://github.com/google/digitalbuildings/tree/master/tools/guid_generator).
**NOTE:** The new building configuration format requires that entities are keyed by Version 4 UUIDs (referred to as guids) instead of the code. To convert from old format to the new format, run your building configuration file(.yaml) through the [guid generator](https://github.com/google/digitalbuildings/tree/master/tools/guid_generator).

Any questions or concerns can be emailed to **BOS-GPS@google.com**
Any questions or concerns can be emailed to **BOS-GPS@google.com**.
40 changes: 22 additions & 18 deletions tools/explorer/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
# Ontology Explorer

The Ontology explorer allows a user to ask a few basic question of the ontology:
* What fields are associated with a type name
* What are the types associated with a given set of fields
* Is a field defined within a certain namespace
The Ontology explorer allows a user to ask a few basic questions of the ontology:
* What fields are associated with a type name?
* What are the types associated with a given set of fields?
* Is a field defined within a certain namespace?

## Install

To install please follow the instructions below.

## First create a virtual env
### Create a Virtual Environment

Create the virtual environment with `virtualenv` followed by the environment name, in this example: `tooling`
First, create a virtual environment with `venv` followed by the environment name (in this example: `tooling`) in the digitalbuildings repository.

```
virtualenv tooling
python -m venv tooling
```


Activate the virtual environment
### Activate the Virtual Environment

Mac OS / Linux:
```
Expand All @@ -29,32 +29,36 @@ Windows
```
tooling\Scripts\activate
```
### Install Packages
Next, you can either use pip or setup (to be deprecated) to install the necessary packages and dependencies.

#### Pip
1. Run the following command to ensure that your Python package management tools are up-to-date.

Then you can either use pip or setuptools.


## Pip
```
python3 -m pip install --upgrade pip
```

1. Run `python3 -m pip install --upgrade pip` to ensure that your Python package management tools are up-to-date.
2. Run `python3 -m pip install .` from the following directories:

* digitalbuildings/tools/validators/ontology_validator
* digitalbuildings/tools/validators/instance_validator
* digitalbuildings/tools/explorer


### Setup (to be deprecated)
#### Setup (to be deprecated)
To install the dependencies, please run `python setup.py install` from the following directories, in order:
* digitalbuildings/tools/validators/ontology_validator
* digitalbuildings/tools/validators/instance_validator
* digitalbuildings/tools/explorer

## Usage
There are two main ways in which the ontology explorer can be used:

1. Import explorer_handler from explorer.lib and call Build()
* This returns an OntologyWrapper object which can be used to execute query functions on the Digital Buildings Ontology. You may also pass to Build() a path to an ontology extension.
2. The explorer may also be run as a stand-alone command-line interface.
* Run `python explorer.py` to start the application.
* If you have extended the ontology by adding new types to your local ontology, run the following: `python explorer.py --modified-ontology-types=path/to/modified/ontology/types/folder`
* This returns an OntologyWrapper object which can be used to execute query functions on the Digital Buildings Ontology. You may also pass to Build() a path to an ontology extension.

2. As a stand-alone command-line interface (CLI).
* Run `python explorer.py` to start the application.
* If you have extended the ontology by adding new types to your local ontology, run the following: `python explorer.py --modified-ontology-types=path/to/modified/ontology/types/folder`

Loading

0 comments on commit 433703b

Please sign in to comment.