From 68d7e93b62bf302597fd7c754ade92b18ba17fed Mon Sep 17 00:00:00 2001 From: shambergoldstein <124837286+shambergoldstein@users.noreply.github.com> Date: Mon, 8 Jan 2024 11:21:02 -0800 Subject: [PATCH 1/6] Update README.md --- tools/abel/README.md | 288 +++++++++++++++++++++++++------------------ 1 file changed, 165 insertions(+), 123 deletions(-) diff --git a/tools/abel/README.md b/tools/abel/README.md index bfbf46bf6..463a415db 100644 --- a/tools/abel/README.md +++ b/tools/abel/README.md @@ -1,191 +1,233 @@ -# ABEL +# ABEL User Guide -ABEL is an acronym that stands for **A**utomated **B**uilding **E**ntity **L**oader. It is a tool that assists systems integrators (and other DBO users) in efficiently creating and -modifying [Building Configuration YAML files](../../ontology/docs/building_config.md). +### What is ABEL? + +ABEL, or **A**utomated **B**uilding **E**ntity **L**oader, is a tool that assists systems integrators (and other DBO users) in efficiently creating and modifying [Building Configuration YAML files](../../ontology/docs/building_config.md) through use of a [spreadsheet](../../tools/abel/validators/README.md). + +Building configuration files contain information about equipment, devices, and spaces within a building such as their device type, input and output points (fields) used to control them, and their relationship to one another. Building configuration files are onboarded using [DB API](https://docs.google.com/document/d/13dzEmBAwz5Dtcx35qs0GXcKWi1QR4Bitw92wW6H54VE/edit?tab=t.0#heading=h.10pbm1zd8jm8) in order to create a digital representation of the devices in the cloud and make use of the data coming from their associated fields. Due to the sheer size of building configuration files (a complete building configuration file for a building can contain thousands of devices with tens of thousands of fields), creating or editing one manually in YAML format can be very time consuming and prone to mistakes. ABEL streamlines this process by allowing the systems integrator to create and edit these files in a spreadsheet format to make it quicker and easier to edit multiple entities or fields at once. ABEL has the following key features: -* Convert a Google Sheet (in the ABEL Spreadsheet Template) to a building configuration YAML file - * Run Instance Validator (with optional telemetry validation) on the building configuration file -* Convert a building configuration YAML file to a Google Sheet -## Table of Contents +* Create and edit a Google Sheet (in the ABEL Spreadsheet Template) and export it to a building configuration file for onboarding. + * Run Instance Validator (with optional telemetry validation) on the building configuration file during the conversion process to ensure there are no errors in the file prior to attempting onboarding. +* Convert a building configuration file to a Google Sheet to make edits on. +* Split a building configuration file to separate out a single namespace to make it easier to focus edits and onboarding processes to a single area as needed. + -* [Setup and Installation](#setup-and-installation) +## User Guide Contents + +* [Getting Started](#getting-started) + * [Set Up a Tooling Environment](#set-up-a-tooling-environment) + * [Obtain a GCP OAuth Client Credential](#obtain-a-gcp-oauth-client-credential) * [Using ABEL](#using-abel) - * [Update Workflow](#update-workflow) - * [Initialization Workflow](#initialization-workflow) - * [Split Workflow](#split-workflow) + * [General Use](#general-use) + * [Update your Repository Clone and Activate your Virtual Environment](#update-your-repository-clone-and-activate-your-virtual-environment) + * [ABEL Arguments](#abel-arguments) + * [Updating an Existing Building](#updating-an-existing-building) + * [Create a Spreadsheet then Convert it to a Building Configuration File](#create-a-spreadsheet-then-convert-it-to-a-building-configuration-file) + * [Generate a Building Configuration File from an Existing Spreadsheet](#generate-a-building-configuration-file-from-an-existing-spreadsheet) + * [Initializing a New Building](#initializing-a-new-building) + * [Splitting a Building Configuration File](#splitting-a-building-configuration-file) -## Setup and Installation -Before starting the setup and installation process, please ensure that the dependencies are met: +## Getting Started -1. You are running **Python 3.9** or higher +ABEL is a Python tool that must be run in the command line. Before starting the setup and installation process outlined below, please ensure that the following dependencies are met: -2. you have installed the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) +* You are running [Python](http://go/swc) 3.9 or higher +* You have installed the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) +* You have a local clone of the Digital Buildings repository downloaded on your computer + * A local clone can be created by running the following in the gCloud CLI: + + ``` + git clone https://github.com/google/digitalbuildings.git + ``` -The total setup process should only take about 15 minutes. The process is broken into two parts: +The following steps only need to be done on initial setup. Any steps that must be repeated with use will be outlined in the [Using ABEL](#using-abel) section. -1. Setup a virtual environment using [virtualenv](https://virtualenv.pypa.io/en/latest/) and install all of the -proper dependencies -2. Obtain credentials for [Google Sheets API](https://developers.google.com/sheets/api/reference/rest) for use with -ABEL +### Set Up a Tooling Environment -### Set up a tooling environment +A virtual tooling environment ([venv](https://virtualenv.pypa.io/en/latest/)) must be made to create a separate, isolated instance of the Python runtime for a project, with its own packages. To create this environment, follow the following steps: -If environment is already set up then the following steps can be ignored. +1. In the command line, navigate to the Digital Buildings tooling library cloned on your computer. If the Digital Buildings repository is not in your home directory, replace “~” with the folder it exists within. +``` +cd ~/digitalbuildings/tools/ +``` -1. Clone the Digital Buildings repository +2. Create a virtual environment with an environment name of your choice, in this example the environment is named “tooling.” +``` +python3 -m venv tooling +``` + + +### Obtain a GCP OAuth Client Credential + +Contact your IoT Technical Program Manager and ask for an OAuth client credential.json file for authenticating against Google Sheets API. Use the credential.json file for ABEL's --credential command line argument discussed in the [ABEL Arguments](#abel-arguments) section below. + + +## Using ABEL + +When using ABEL there is a set of commands that should be run each time to ensure that the tool and all its requirements are up to date within your local repo. These instructions can be found in the [General Use](#general-use) section. The actual command to run ABEL will use a different set of arguments depending on the action being performed. Please go to the section for the action you are trying to perform to see instructions on which [arguments](#abel-arguments) to pass in. - ``` - git clone https://github.com/google/digitalbuildings.git - ``` -2. Navigate to the [DigitalBuildings tooling library](../../tools/) +### General Use +This section describes commands that should be run every time ABEL is used to make sure the program and validation are up to date, as well as an overview of all of the arguments possible to pass into ABEL. + + +#### Update your Repository Clone and Activate your Virtual Environment + +The following commands will make sure the Digital Buildings repository as a whole is up to date, that your virtual environment is active, and that you have the most up to date packages installed for ABEL: + +1. In the command line, navigate to the Digital Buildings repository cloned on your computer. If the Digital Buildings repository is not in your home directory, replace “~” with the folder it exists within. ``` - cd digitalbuildings/tools/ + cd ~/digitalbuildings/ ``` -3. Create a virtual environment with an environment name, in this example: `tooling` +2. Pull any updates made to the repository since your last pull. + ``` + git pull + ``` +3. Navigate to your tooling virtual environment. Replace “tooling” with the name you gave your venv. ``` - python3 -m venv tooling + cd /tools/ ``` -4. Activate your virtual environment +4. Activate your virtual environment. - * MacOs/Linux: - - ``` - source tooling/bin/activate - ``` - - * Windows: + * MacOs/Linux: + ``` + source tooling/bin/activate + ``` - ``` - tooling\Scripts\activate - ``` - -5. Depending on your operating system, run either of the global setup scripts to configure dependencies + * Windows: + ``` + tooling\Scripts\activate + ``` - * Linux/MacOs: +5. Configure dependencies/packages. - ``` - ./pip_install.sh - ``` - - * Windows: + * MacOs/Linux: + ``` + ./pip_install.sh + ``` - ``` - pip_install.bat - ``` - -6. Navigate to the [ABEL directory](./) + * Windows: + ``` + pip_install.bat + ``` +6. Navigate to the ABEL directory. ``` -cd abel +cd /abel/ ``` -### Obtain a GCP OAuth client credential +7. Continue to whichever section below matches the action you are trying to perform and follow the instructions provided. -1. Contact your IoT Technical Program Manager and ask for an OAuth client `credential.json` file for authenticating against Google Sheets API. -2. Use the `credential.json` file for ABEL's `--credential` command line argument. +#### ABEL Arguments -## Using ABEL -ABEL has a few pieces of core functionality, they are as follows: -1. Modify a spreadsheet or a building configuration for an existing building to produce an [UPDATE building config](../../ontology/docs/building_config.md#update) - * [Update Workflow](#update-workflow) +Below is a list of all arguments that can be passed into abel. Some of these are required for any ABEL process whereas others are only required for some. The task-based instructions provided in the following three sections will show which arguments you need for your desired action. -2. Create a [Building Config](../../ontology/docs/building_config.md) from an [ABEL spreadsheet](../../tools/abel/validators/README.md) for a new building - * [Initialization Workflow](#initialization-workflow) +| Argument | Requirement | Description | +| ------------- | ------------- | ------------- | +| -c or --credential | Always Required | The absolute or relative path to a [GCP OAuth client credential JSON file](#obtain-a-gcp-oauth-client-credential). An OAuth client credential is required for authentication against Google Sheets service. | +| -s or --spreadsheet_id | Required for Certain Workflows | The ID associated with a Google Sheets spreadsheet. A Google Sheets ID is found embedded into the spreadsheet's URL (e.g., https://docs/google/com/spreadsheets/d//edit#gid=123467). This is required when ABEL is creating a building configuration YAML file from a Google spreadsheet. | +| -b or --building_config | Required for Certain Workflows | The absolute or relative file path to a local building configuration file (YAML file on the system integrator’s computer). Only required for when able is making a spreadsheet from a building configuration file. | +| -p or --subscription | Optional | A fully-qualified path to a Google Cloud Pubsub [subscription](https://pantheon.corp.google.com/cloudpubsub/subscription/list?mods=logs_tg_prod&project=essential-keep-197822&pageState=(%22cpsSubscriptionList%22:(%22p%22:0),%22duration%22:(%22groupValue%22:%22PT1H%22,%22customValue%22:null))&pli=1), e.g., projects/google.com:your-project/subscriptions/your-subscription. This parameter is only required if telemetry validation is desired during the building configuration generation. | +| -o or --timeout | Optional | Custom timeout duration in seconds for the telemetry validation test. The default value is 600 seconds, or 10 minutes. If this time limit is exceeded before the validator receives a test pubsub message for each of the entities configured in the given building configuration file, the test will fail with an error and report the entities that were not heard from. | +| -m or --modified-types-filepath | Optional | A fully-qualified path to a modified ontology that is not in the Digital Buildings repository. The Ontology Validator will surface validation results from the modified ontology, and building configuration files will be validated against the modified ontology. This should be used to validate against ontology edits/additions the system’s integrator has made that have not had approved pull requests. Note that the building configuration must pass validation against the ontology in the Digital Buildings repository before the file can be considered ready to onboard. | +| -d or --output-dir | Optional | An absolute or relative file path to a directory which ABEL can write files to. ABEL must have write access to this directory or else an error will be raised. If not provided, ABEL will write files to the current directory by default. | -3. Split a building configuration file on a namespace - * [Split Workflow](#split-workflow) -### Command-line arguments for ABEL -1. `-c` or `--credential` **Required**: The absolute or relative path to a GCP OAuth client credential file. An OAuth client credential is required for authentication against Google Sheets service. +### Updating an Existing Building -2. `-s` or `--spreadsheet_id`: The ID associated with a Google Sheets spreadsheet - * [ABEL Spreadsheet Template](https://docs.google.com/spreadsheets/d/1b6IRimNS1dAtPjkNN-fk4TirnLzOiDyyUmOKP_MhMM0/copy#gid=980240783) - * A Google Sheets ID is found embedded into the spreadsheet's URL (e.g., `https://docs/google/com/spreadsheets/d//edit#gid=123467`). Required when ABEL is creating a building configuration from a Google spreadsheet. +Use the directions in this section when you would like to update or add entities for a building that has already been onboarded itself. Make sure you have completed the steps in the [General Use](#general-use) section before continuing to any of the steps below. -3. `-b` or `--building_config`: The absolute or relative path to a local building configuration file. Only required for the `Building Configuration -> Spreadsheet` workflow. - * [Building Configuration Docs](../../ontology/docs/building_config.md) +During the conversion of the ABEL spreadsheet to a building configuration YAML file, if your spreadsheet does not pass validation then ABEL will fast-fail and a validation report will be created in the current or specified directory with the name “spreadsheet_validation_.log”. If the spreadsheet does pass validation then the resulting building configuration file and instance validation report will be written to the current or specified directory with the following names accordingly: “bc_export_.yaml” and +“instance_validation_.log”. -4. `-p` or `--subscription` **[Optional]**: A fully-qualified path to a Google Cloud Pubsub subscription, e.g., `projects/google.com:your-project/subscriptions/your-subscription`. This parameter is only required for telemetry validation. -5. `-o` or `--timeout` **[Optional]**: Custom timeout duration in seconds for the telemetry validation test. The default value is 600 seconds, or 10 minutes. If this time limit is exceeded before the validator receives a test pubsub message for each of the entities configured in the given building configuration file, the test will fail with an error and report the entities that were not heard from. +#### Create a Spreadsheet then Convert it to a Building Configuration File -6. `-m` or `--modified-types-filepath` **[Optional]**: A fully-qualified path to a modified ontology that is not in the [DigitalBuildings repository](../..). The [Ontology Validator](../validators/ontology_validator) will surface validation results from the modified ontology, and building configuration files will be validated against the modified ontology. +Use the commands below if you do not have an ABEL spreadsheet already created for the entities you would like to onboard. This process will generate a spreadsheet template for you to fill out, then allow you to convert that spreadsheet into a building configuration YAML file once completed. -7. `-d` or `--output-dir`: An absolute or relative file path to a directory which ABEL can write files to. ABEL must have write access to this directory or else an error will be raised. +Because the building already exists in the cloud, this process requires an initial building configuration file to be passed in for editing in spreadsheet format. Export the configuration file for the building or specific for entities using [DB API](https://docs.google.com/document/d/13dzEmBAwz5Dtcx35qs0GXcKWi1QR4Bitw92wW6H54VE/edit?tab=t.0#heading=h.10pbm1zd8jm8) prior to following the steps below. -### The ABEL Spreadsheet -The ABEL spreadsheet serves as a user-friendly interface for ABEL and is what allows a user to make changes to machine-readable documents like [Building Config](../../ontology/docs/building_config.md). +1. Run the ABEL command with the arguments provided. Note that the arguments in brackets are optional. See the [ABEL Arguments](#abel-arguments) section for details. +``` +python3 abel.py -c -b +[-p ] [-o <300>] [-m ] +``` -Please see the [ABEL Spreadsheet docs](../../tools/abel/validators/README.md) for detailed instructions on how to create your own spreadsheet. +2. Select option 1, modify a spreadsheet/building config for an existing building, by entering a “1” into the command prompt. This indicates that you are working with an already onboarded building. -### Update Workflow -If you would like to create a building configuration to update an already onboarded building, there are two options: +3. Select option 1, edit or update an existing building config, by entering a “1” into the command prompt.This indicates that you do not have an existing spreadsheet and would like to create one off of an initial building configuration file. Note that the building configuration file’s metadata is set to update or initialize for this step to be successful. -Generate an ABEL spreadsheet from an exported building configuration. - 1. In `digitalbuildings/tools/abel` run ABEL with the command: +4. Make any edits or additions to the building entities using the ABEL spreadsheet generated. - ``` - python3 abel.py -c /path/to/credential.json -b absolute/path/to/building/config - ``` -To generate a building config from an already-updated spreadsheet. - 1. In `digitalbuildings/tools/abel` run ABEL with the command: +5. Return to the command prompt when finished and press any key to perform validation and generate a new building configuration YAML file. - ``` - python3 abel.py -s -c -d - ``` - 3. If your spreadsheet does not pass the validation criteria found in the - [spreadsheet docs](../../tools/abel/validators/README.md) then ABEL will fast-fail and a validation report will be created in the current directory with the name `spreadsheet_validation_.log` +#### Generate a Building Configuration File from an Existing Spreadsheet - 4. The resulting building configuration file and instance validation report will be written to the current directory with the following names: - * `bc_export_.yaml` - * `instance_validation_.log` +Use the commands below if you have already created and edited an ABEL spreadsheet for the entities you would like to onboard. This process will convert your edited spreadsheet into a building configuration YAML file. -### Initialization Workflow -If you would like create a building configuration file under the initialization operation +1. Run the ABEL command with the arguments provided. Note that the arguments in brackets are optional. See the [ABEL Arguments](#abel-arguments) section for details. +``` +python3 abel.py -s -c +[-p ] [-o <300>] [-m ] +[-d ] +``` -The process for using an ABEL spreadsheet to generate a new building configuration is as -follows: +2. Select option 1, modify a spreadsheet/building config for an existing building, by entering a “1” into the command prompt. This indicates that you are working with an already onboarded building. -1. Create a spreadsheet for ABEL from [ABEL Spreadsheet template](https://docs.google.com/spreadsheets/d/1nlFVwVvmumBSIAAAv7xq1-xuqGy0k1ONyrG2rMqrgdE/copy#gid=980240783) -2. Populate your spreadsheet. A well-defined guide on how to populate your - spreadsheet can be found in the [spreadsheet docs](../../tools/abel/validators/README.md) +3. Select option 2, create building config from an updated spreadsheet, by entering a “2” into the command prompt. This indicates that you have already created and edited an ABEL spreadsheet and would like to convert it to a building configuration file. Selecting this option will initialize validation and generate a building configuration YAML file. -3. In `digitalbuildings/tools/abel` run ABEL with the command: - ``` - python3 abel.py -s -c -d - ``` -4. Choose option 2: Create a spreadsheet for a new building +### Initializing a New Building -5. If your spreadsheet does not pass the validation criteria found in the [spreadsheet docs](../../tools/abel/validators/README.md) then ABEL will fast-fail and a validation report will be created in the directory specified with the `-d` arguments with the name `spreadsheet_validation_.log` +Use the directions in this section when you would like to add entities for a building that has not yet been onboarded itself. Make sure you have completed the steps in the [General Use](#general-use) section before continuing to any of the steps below. -6. The resulting building configuration file and instance validation report will be written to the same directory with the following names: - * `bc_export_.yaml` - * `instance_validation_.log` +During the conversion of the ABEL spreadsheet to a building configuration YAML file, if your spreadsheet does not pass validation then ABEL will fast-fail and a validation report will be created in the current or specified directory with the name “spreadsheet_validation_.log”. If the spreadsheet does pass validation then the resulting building configuration file and instance validation report will be written to the current or specified directory with the following names accordingly: “bc_export_.yaml” and +“instance_validation_.log”. -### Split Workflow -To split a building configuration file on a specific DBO namespace: +Note that because the building does not exist in the cloud, there is no option to generate a spreadsheet off of an initial building configuration file as in the [Create a Spreadsheet then Convert it to a Building Configuration File](#create-a-spreadsheet-then-convert-it-to-a-building-configuration-file) section for existing buildings. The spreadsheet template must be manually copied instead as shown in step one below. -1. Run abel with a json credential and a building configuration: - ``` - python3 abel.py -c -b - ``` -2. Select option 3: `Split a building config` - -4. ABEL will generate a new building configuration split on the desired namespace along with any dependencies (e.g., a building configuration containing only entities in the DBO `METERS` namespace). - * Provide the `-d` argument if you would like ABEL to write to a specific directory. +1. Create a spreadsheet for ABEL from the [ABEL Spreadsheet template](https://docs.google.com/spreadsheets/d/1b6IRimNS1dAtPjkNN-fk4TirnLzOiDyyUmOKP_MhMM0/copy#gid=980240783) by following the link and clicking “Make a copy”. + +2. Populate your spreadsheet. A well-defined guide on how to populate your spreadsheet can be found in the [spreadsheet docs](https://github.com/google/digitalbuildings/blob/master/tools/abel/validators/README.md). + +3. Run the ABEL command with the arguments provided. Note that the arguments in brackets are optional. See the [ABEL Arguments](#abel-arguments) section for details. +``` +python3 abel.py -s -c +[-p ] [-o <300>] [-m ] +[-d ] +``` + +4. Choose option 2, create a spreadsheet for a new building, by entering a “2” into the command prompt. This indicates that you are working with a new building and would like to convert a newly created spreadsheet into a building configuration file. Selecting this option will initialize validation and generate a building configuration YAML file. + + +### Splitting a Building Configuration File + +Use the directions in this section when you would like to split a single namespace’s entities out of a master building configuration file. Note that this process will split the namespace into a new building configuration file, not an ABEL spreadsheet. If you would like to make edits in spreadsheet format for the namespace, first split the namespace using the steps below then follow the steps in the [Create a Spreadsheet then Convert it to a Building Configuration File](#create-a-spreadsheet-then-convert-it-to-a-building-configuration-file) section to make the conversion. Make sure you have completed the steps in the [General Use](#general-use) section before continuing to any of the steps below. + +The resulting building configuration file will be written to the current or specified directory with the name, “bc_export_.yaml”. + +1. Run the ABEL command with the arguments provided. Note that the arguments in brackets are optional. See the [ABEL Arguments](#abel-arguments) section for details. +``` +python3 abel.py -b -c +[-p ] [-o <300>] [-m ] +[-d ] +``` + +2. Select option 3, split a building config, by entering a “3” into the command prompt. + +3. Type the namespace you wish to split out into the command prompt. Note that ABEL can only split out one namespace at a time. ABEL will generate a new building configuration split on the desired namespace along with any dependencies (e.g., a building configuration containing only entities in the DBO METERS namespace). From 850935906521f1a3770aa8516c83be742828fc69 Mon Sep 17 00:00:00 2001 From: shambergoldstein <124837286+shambergoldstein@users.noreply.github.com> Date: Thu, 11 Jan 2024 12:29:30 -0800 Subject: [PATCH 2/6] Update README.md v2 changes based on pr feedback --- tools/abel/README.md | 56 +++++++++++++++++--------------------------- 1 file changed, 21 insertions(+), 35 deletions(-) diff --git a/tools/abel/README.md b/tools/abel/README.md index 463a415db..28739d07d 100644 --- a/tools/abel/README.md +++ b/tools/abel/README.md @@ -4,14 +4,13 @@ ABEL, or **A**utomated **B**uilding **E**ntity **L**oader, is a tool that assists systems integrators (and other DBO users) in efficiently creating and modifying [Building Configuration YAML files](../../ontology/docs/building_config.md) through use of a [spreadsheet](../../tools/abel/validators/README.md). -Building configuration files contain information about equipment, devices, and spaces within a building such as their device type, input and output points (fields) used to control them, and their relationship to one another. Building configuration files are onboarded using [DB API](https://docs.google.com/document/d/13dzEmBAwz5Dtcx35qs0GXcKWi1QR4Bitw92wW6H54VE/edit?tab=t.0#heading=h.10pbm1zd8jm8) in order to create a digital representation of the devices in the cloud and make use of the data coming from their associated fields. Due to the sheer size of building configuration files (a complete building configuration file for a building can contain thousands of devices with tens of thousands of fields), creating or editing one manually in YAML format can be very time consuming and prone to mistakes. ABEL streamlines this process by allowing the systems integrator to create and edit these files in a spreadsheet format to make it quicker and easier to edit multiple entities or fields at once. +Building configuration files contain information about equipment, devices, and spaces within a building such as their device type, input and output points (fields) used to control them, and their relationship to one another. Building configuration files are onboarded in order to create a digital representation of the devices in the cloud and make use of the data coming from their associated fields. Due to the sheer size of building configuration files (a complete building configuration file for a building can contain thousands of devices with tens of thousands of fields), creating or editing one manually in YAML format can be very time consuming and prone to mistakes. ABEL streamlines this process by allowing the systems integrator to create and edit these files in a spreadsheet format to make it quicker and easier to edit multiple entities or fields at once. ABEL has the following key features: * Create and edit a Google Sheet (in the ABEL Spreadsheet Template) and export it to a building configuration file for onboarding. * Run Instance Validator (with optional telemetry validation) on the building configuration file during the conversion process to ensure there are no errors in the file prior to attempting onboarding. * Convert a building configuration file to a Google Sheet to make edits on. -* Split a building configuration file to separate out a single namespace to make it easier to focus edits and onboarding processes to a single area as needed. ## User Guide Contents @@ -27,17 +26,15 @@ ABEL has the following key features: * [Create a Spreadsheet then Convert it to a Building Configuration File](#create-a-spreadsheet-then-convert-it-to-a-building-configuration-file) * [Generate a Building Configuration File from an Existing Spreadsheet](#generate-a-building-configuration-file-from-an-existing-spreadsheet) * [Initializing a New Building](#initializing-a-new-building) - * [Splitting a Building Configuration File](#splitting-a-building-configuration-file) ## Getting Started ABEL is a Python tool that must be run in the command line. Before starting the setup and installation process outlined below, please ensure that the following dependencies are met: -* You are running [Python](http://go/swc) 3.9 or higher -* You have installed the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) +* You are running Python 3.9 or higher * You have a local clone of the Digital Buildings repository downloaded on your computer - * A local clone can be created by running the following in the gCloud CLI: + * A local clone can be created by running the following in your CLI: ``` git clone https://github.com/google/digitalbuildings.git @@ -48,7 +45,7 @@ The following steps only need to be done on initial setup. Any steps that must b ### Set Up a Tooling Environment -A virtual tooling environment ([venv](https://virtualenv.pypa.io/en/latest/)) must be made to create a separate, isolated instance of the Python runtime for a project, with its own packages. To create this environment, follow the following steps: +A virtual tooling environment ([venv](https://docs.python.org/3/library/venv.html)) must be made to create a separate, isolated instance of the Python runtime for a project, with its own packages. To create this environment, follow the following steps: 1. In the command line, navigate to the Digital Buildings tooling library cloned on your computer. If the Digital Buildings repository is not in your home directory, replace “~” with the folder it exists within. ``` @@ -90,12 +87,12 @@ The following commands will make sure the Digital Buildings repository as a whol git pull ``` -3. Navigate to your tooling virtual environment. Replace “tooling” with the name you gave your venv. +3. Navigate to the tools folder where your virtual environment was created. ``` cd /tools/ ``` -4. Activate your virtual environment. +4. Activate your virtual environment. Replace “tooling” with the name you gave your venv. * MacOs/Linux: ``` @@ -120,9 +117,16 @@ The following commands will make sure the Digital Buildings repository as a whol ``` 6. Navigate to the ABEL directory. -``` -cd /abel/ -``` + + * MacOs/Linux: + ``` + cd abel + ``` + + * Windows: + ``` + cd /abel/ + ``` 7. Continue to whichever section below matches the action you are trying to perform and follow the instructions provided. @@ -163,10 +167,10 @@ python3 abel.py -c -b ] ``` -2. Select option 1, modify a spreadsheet/building config for an existing building, by entering a “1” into the command prompt. This indicates that you are working with an already onboarded building. +2. Select ```option 1```, modify a spreadsheet/building config for an existing building, by entering a “1” into the command prompt. This indicates that you are working with an already onboarded building. -3. Select option 1, edit or update an existing building config, by entering a “1” into the command prompt.This indicates that you do not have an existing spreadsheet and would like to create one off of an initial building configuration file. Note that the building configuration file’s metadata is set to update or initialize for this step to be successful. +3. Select ```option 1```, edit or update an existing building config, by entering a “1” into the command prompt.This indicates that you do not have an existing spreadsheet and would like to create one off of an initial building configuration file. Note that the building configuration file’s metadata is set to update or initialize for this step to be successful. 4. Make any edits or additions to the building entities using the ABEL spreadsheet generated. @@ -186,10 +190,10 @@ python3 abel.py -s -c [-d ] ``` -2. Select option 1, modify a spreadsheet/building config for an existing building, by entering a “1” into the command prompt. This indicates that you are working with an already onboarded building. +2. Select ```option 1```, modify a spreadsheet/building config for an existing building, by entering a “1” into the command prompt. This indicates that you are working with an already onboarded building. -3. Select option 2, create building config from an updated spreadsheet, by entering a “2” into the command prompt. This indicates that you have already created and edited an ABEL spreadsheet and would like to convert it to a building configuration file. Selecting this option will initialize validation and generate a building configuration YAML file. +3. Select ```option 2```, create building config from an updated spreadsheet, by entering a “2” into the command prompt. This indicates that you have already created and edited an ABEL spreadsheet and would like to convert it to a building configuration file. Selecting this option will initialize validation and generate a building configuration YAML file. ### Initializing a New Building @@ -212,22 +216,4 @@ python3 abel.py -s -c [-d ] ``` -4. Choose option 2, create a spreadsheet for a new building, by entering a “2” into the command prompt. This indicates that you are working with a new building and would like to convert a newly created spreadsheet into a building configuration file. Selecting this option will initialize validation and generate a building configuration YAML file. - - -### Splitting a Building Configuration File - -Use the directions in this section when you would like to split a single namespace’s entities out of a master building configuration file. Note that this process will split the namespace into a new building configuration file, not an ABEL spreadsheet. If you would like to make edits in spreadsheet format for the namespace, first split the namespace using the steps below then follow the steps in the [Create a Spreadsheet then Convert it to a Building Configuration File](#create-a-spreadsheet-then-convert-it-to-a-building-configuration-file) section to make the conversion. Make sure you have completed the steps in the [General Use](#general-use) section before continuing to any of the steps below. - -The resulting building configuration file will be written to the current or specified directory with the name, “bc_export_.yaml”. - -1. Run the ABEL command with the arguments provided. Note that the arguments in brackets are optional. See the [ABEL Arguments](#abel-arguments) section for details. -``` -python3 abel.py -b -c -[-p ] [-o <300>] [-m ] -[-d ] -``` - -2. Select option 3, split a building config, by entering a “3” into the command prompt. - -3. Type the namespace you wish to split out into the command prompt. Note that ABEL can only split out one namespace at a time. ABEL will generate a new building configuration split on the desired namespace along with any dependencies (e.g., a building configuration containing only entities in the DBO METERS namespace). +4. Choose ```option 2```, create a spreadsheet for a new building, by entering a “2” into the command prompt. This indicates that you are working with a new building and would like to convert a newly created spreadsheet into a building configuration file. Selecting this option will initialize validation and generate a building configuration YAML file. From 59f880617f666a875f2c3d9d7b79a275f087b003 Mon Sep 17 00:00:00 2001 From: shambergoldstein <124837286+shambergoldstein@users.noreply.github.com> Date: Thu, 11 Jan 2024 12:48:55 -0800 Subject: [PATCH 3/6] Update README.md --- tools/abel/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/abel/README.md b/tools/abel/README.md index 28739d07d..f00e4965b 100644 --- a/tools/abel/README.md +++ b/tools/abel/README.md @@ -158,7 +158,7 @@ During the conversion of the ABEL spreadsheet to a building configuration YAML f Use the commands below if you do not have an ABEL spreadsheet already created for the entities you would like to onboard. This process will generate a spreadsheet template for you to fill out, then allow you to convert that spreadsheet into a building configuration YAML file once completed. -Because the building already exists in the cloud, this process requires an initial building configuration file to be passed in for editing in spreadsheet format. Export the configuration file for the building or specific for entities using [DB API](https://docs.google.com/document/d/13dzEmBAwz5Dtcx35qs0GXcKWi1QR4Bitw92wW6H54VE/edit?tab=t.0#heading=h.10pbm1zd8jm8) prior to following the steps below. +Because the building already exists in the cloud, this process requires an initial building configuration file to be passed in for editing in spreadsheet format. Export the configuration file for the building or specific for entities prior to following the steps below. 1. Run the ABEL command with the arguments provided. Note that the arguments in brackets are optional. See the [ABEL Arguments](#abel-arguments) section for details. ``` From 6caea074b36777f4212476a4e5a7ed065c4a5ca4 Mon Sep 17 00:00:00 2001 From: shambergoldstein <124837286+shambergoldstein@users.noreply.github.com> Date: Thu, 11 Jan 2024 12:50:06 -0800 Subject: [PATCH 4/6] Update README.md --- tools/abel/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/abel/README.md b/tools/abel/README.md index f00e4965b..2c63e02c5 100644 --- a/tools/abel/README.md +++ b/tools/abel/README.md @@ -140,7 +140,7 @@ Below is a list of all arguments that can be passed into abel. Some of these are | -c or --credential | Always Required | The absolute or relative path to a [GCP OAuth client credential JSON file](#obtain-a-gcp-oauth-client-credential). An OAuth client credential is required for authentication against Google Sheets service. | | -s or --spreadsheet_id | Required for Certain Workflows | The ID associated with a Google Sheets spreadsheet. A Google Sheets ID is found embedded into the spreadsheet's URL (e.g., https://docs/google/com/spreadsheets/d//edit#gid=123467). This is required when ABEL is creating a building configuration YAML file from a Google spreadsheet. | | -b or --building_config | Required for Certain Workflows | The absolute or relative file path to a local building configuration file (YAML file on the system integrator’s computer). Only required for when able is making a spreadsheet from a building configuration file. | -| -p or --subscription | Optional | A fully-qualified path to a Google Cloud Pubsub [subscription](https://pantheon.corp.google.com/cloudpubsub/subscription/list?mods=logs_tg_prod&project=essential-keep-197822&pageState=(%22cpsSubscriptionList%22:(%22p%22:0),%22duration%22:(%22groupValue%22:%22PT1H%22,%22customValue%22:null))&pli=1), e.g., projects/google.com:your-project/subscriptions/your-subscription. This parameter is only required if telemetry validation is desired during the building configuration generation. | +| -p or --subscription | Optional | A fully-qualified path to a Google Cloud Pubsub subscription, e.g., projects/google.com:your-project/subscriptions/your-subscription. This parameter is only required if telemetry validation is desired during the building configuration generation. | | -o or --timeout | Optional | Custom timeout duration in seconds for the telemetry validation test. The default value is 600 seconds, or 10 minutes. If this time limit is exceeded before the validator receives a test pubsub message for each of the entities configured in the given building configuration file, the test will fail with an error and report the entities that were not heard from. | | -m or --modified-types-filepath | Optional | A fully-qualified path to a modified ontology that is not in the Digital Buildings repository. The Ontology Validator will surface validation results from the modified ontology, and building configuration files will be validated against the modified ontology. This should be used to validate against ontology edits/additions the system’s integrator has made that have not had approved pull requests. Note that the building configuration must pass validation against the ontology in the Digital Buildings repository before the file can be considered ready to onboard. | | -d or --output-dir | Optional | An absolute or relative file path to a directory which ABEL can write files to. ABEL must have write access to this directory or else an error will be raised. If not provided, ABEL will write files to the current directory by default. | From 5eb313b2320344ecfaabb1987e256b70df11e52f Mon Sep 17 00:00:00 2001 From: shambergoldstein <124837286+shambergoldstein@users.noreply.github.com> Date: Fri, 19 Jan 2024 07:53:43 -0800 Subject: [PATCH 5/6] Small changes --- tools/abel/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/abel/README.md b/tools/abel/README.md index 2c63e02c5..32a17c2ba 100644 --- a/tools/abel/README.md +++ b/tools/abel/README.md @@ -2,7 +2,7 @@ ### What is ABEL? -ABEL, or **A**utomated **B**uilding **E**ntity **L**oader, is a tool that assists systems integrators (and other DBO users) in efficiently creating and modifying [Building Configuration YAML files](../../ontology/docs/building_config.md) through use of a [spreadsheet](../../tools/abel/validators/README.md). +ABEL, or **A**utomated **B**uilding **E**ntity **L**oader, is a tool that assists systems integrators (and other DBO users) in efficiently creating and modifying [Building Configuration YAML files](../../ontology/docs/building_config.md) through use of a [spreadsheet](./validators/README.md). Building configuration files contain information about equipment, devices, and spaces within a building such as their device type, input and output points (fields) used to control them, and their relationship to one another. Building configuration files are onboarded in order to create a digital representation of the devices in the cloud and make use of the data coming from their associated fields. Due to the sheer size of building configuration files (a complete building configuration file for a building can contain thousands of devices with tens of thousands of fields), creating or editing one manually in YAML format can be very time consuming and prone to mistakes. ABEL streamlines this process by allowing the systems integrator to create and edit these files in a spreadsheet format to make it quicker and easier to edit multiple entities or fields at once. @@ -23,7 +23,7 @@ ABEL has the following key features: * [Update your Repository Clone and Activate your Virtual Environment](#update-your-repository-clone-and-activate-your-virtual-environment) * [ABEL Arguments](#abel-arguments) * [Updating an Existing Building](#updating-an-existing-building) - * [Create a Spreadsheet then Convert it to a Building Configuration File](#create-a-spreadsheet-then-convert-it-to-a-building-configuration-file) + * [Create a Spreadsheet and Convert it to a Building Configuration File](#create-a-spreadsheet-and-convert-it-to-a-building-configuration-file) * [Generate a Building Configuration File from an Existing Spreadsheet](#generate-a-building-configuration-file-from-an-existing-spreadsheet) * [Initializing a New Building](#initializing-a-new-building) @@ -65,7 +65,7 @@ Contact your IoT Technical Program Manager and ask for an OAuth client credentia ## Using ABEL -When using ABEL there is a set of commands that should be run each time to ensure that the tool and all its requirements are up to date within your local repo. These instructions can be found in the [General Use](#general-use) section. The actual command to run ABEL will use a different set of arguments depending on the action being performed. Please go to the section for the action you are trying to perform to see instructions on which [arguments](#abel-arguments) to pass in. +When using ABEL there is a set of commands that should be run each time to ensure that the tool and all its requirements are up to date within your local repository. These instructions can be found in the [General Use](#general-use) section. The actual command to run ABEL will use a different set of arguments depending on the action being performed. Please go to the section for the action you are trying to perform to see instructions on which [arguments](#abel-arguments) to pass in. ### General Use @@ -154,7 +154,7 @@ During the conversion of the ABEL spreadsheet to a building configuration YAML f “instance_validation_.log”. -#### Create a Spreadsheet then Convert it to a Building Configuration File +#### Create a Spreadsheet and Convert it to a Building Configuration File Use the commands below if you do not have an ABEL spreadsheet already created for the entities you would like to onboard. This process will generate a spreadsheet template for you to fill out, then allow you to convert that spreadsheet into a building configuration YAML file once completed. From 6d6451457a2a9ca31e13a8224bdc3748811686a3 Mon Sep 17 00:00:00 2001 From: shambergoldstein <124837286+shambergoldstein@users.noreply.github.com> Date: Tue, 23 Jan 2024 10:37:07 -0800 Subject: [PATCH 6/6] Update README.md --- tools/abel/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/abel/README.md b/tools/abel/README.md index 32a17c2ba..1e0e95e75 100644 --- a/tools/abel/README.md +++ b/tools/abel/README.md @@ -4,7 +4,7 @@ ABEL, or **A**utomated **B**uilding **E**ntity **L**oader, is a tool that assists systems integrators (and other DBO users) in efficiently creating and modifying [Building Configuration YAML files](../../ontology/docs/building_config.md) through use of a [spreadsheet](./validators/README.md). -Building configuration files contain information about equipment, devices, and spaces within a building such as their device type, input and output points (fields) used to control them, and their relationship to one another. Building configuration files are onboarded in order to create a digital representation of the devices in the cloud and make use of the data coming from their associated fields. Due to the sheer size of building configuration files (a complete building configuration file for a building can contain thousands of devices with tens of thousands of fields), creating or editing one manually in YAML format can be very time consuming and prone to mistakes. ABEL streamlines this process by allowing the systems integrator to create and edit these files in a spreadsheet format to make it quicker and easier to edit multiple entities or fields at once. +Building configuration files are concrete instances of the ontology that contain information about equipment, devices, and spaces within a building such as their device type, input and output points (fields) used to control them, and their relationship to one another. Building configuration files are onboarded in order to create a digital representation of the devices in the cloud and make use of the data coming from their associated fields. Due to the sheer size of building configuration files (a complete building configuration file for a building can contain thousands of devices with tens of thousands of fields), creating or editing one manually in YAML format can be very time consuming and prone to mistakes. ABEL streamlines this process by allowing the systems integrator to create and edit these files in a spreadsheet format to make it quicker and easier to edit multiple entities or fields at once. ABEL has the following key features: @@ -170,7 +170,7 @@ python3 abel.py -c -b