Docs Release 2023-10-003

docs/platform/how-to/create-environment.md

@@ -0,0 +1,109 @@
+# Create an environment
+
+When you [create a project](create-project.md), you'll need to create at least one [environment](../changes.md#environments). This page describes how to create an environment in more detail.
+
+There are several ways you can create a new environment, if you've not created one as part of creating the project. For example, you can do it from the top-level kebab menu:
+
+![Create environment - kebab menu](../images/how-to/create-environment/create-environment-kebab-menu.png){width=60%}
+
+You can also create an environment from the main project screen:
+
+![Create environment - main project screen](../images/how-to/create-environment/create-environment-project-screen.png){width=60%}
+
+## Name and branch
+
+You now need to give your environment a name, and select (or create) the branch it is associated with:
+
+![Name and branch](../images/how-to/create-environment/name-branch.png){width=60%}
+
+You can enter any suitable name for your environment. Examples could be: production, testing, develop, staging, prototype, and so on. Any name that suits the use case can be used.
+
+An environment is always associated with a branch. You can select a branch that already exists (such as when you have connected your project to an existing Git repository), but you are also free to create any new branches you need - simply click `+ New branch` from the branch dropdown, in the `Environment settings` dialog.
+
+## Kafka hosting option
+
+There are three hosting options:
+
+1. Quix broker
+2. Self-hosted
+3. Confluent Cloud
+
+Each of these is described briefly in the following sections.
+
+### Quix broker
+
+The simplest and most convenient choice is to use Quix-managed Kafka. No installation of Kafka is required, and configuration can be done through the UI if you need to change the sensible default values. Very little knowledge of Kafka is expected, beyond basic familiarity with concepts such as [topics](../glossary.md#topic).
+
+There is a small charge for storage for messages persisted in a topic: 
+
+![Topic storage](../images/how-to/create-environment/topic-storage.png){width=80%}
+
+### Self-hosted Kafka
+
+If you want to install and manage your own Kafka installation, you can do this too. You'll need to do more configuration, and be very familiar with details of Kafka hosting and configuration. Selecting this option presents you with a setup guide:
+
+![Self-hosted Kafka](../images/how-to/create-environment/self-hosted-kafka.png){width=90%}
+
+You also have the option to test your connection with the Kafka server before continuing.
+
+!!! tip
+
+    You could use any Kafka solution here, for example [Redpanda](https://redpanda.com/){target=_blank}.
+
+### Confluent Cloud
+
+You can also use the Confluent Cloud hosting option. You can read documentation on [connecting to Confluent Cloud](../integrations/kafka/confluent-cloud.md). It is assumed you're familiar with [Confluent Cloud](https://www.confluent.io/){target=_blank}.
+
+## Data and streaming services
+
+The last step in creating an environment is to choose your data and streaming services option. 
+
+![Data and streaming services](../images/how-to/create-environment/data-streaming-services.png){width=60%}
+
+There are two options here:
+
+* Standard
+* High performance
+
+These options determine the following:
+
+* The amount of storage available to persisted topics
+* The level of resources (CPU, RAM) allocated to streaming services
+
+### Persisted storage
+
+Persisted storage is when you enable persistence on a topic: 
+
+![Topic persistence](../images/how-to/create-environment/topic-persistence.png){width=80%}
+
+When this option is selected, data in the topic is persisted to a Quix database (InfluxDB). This data can then be queried using the [Query API](../../apis/query-api/index.md), or played back into a topic using the [replay service](replay.md). 
+
+!!! note
+
+    Persisted storage is not the same as topic storage. Topic storage is charged separately, and relates to storage allocated to messages retained in a Kafka topic, for the [Quix broker option](#quix-broker).
+
+### Streaming services
+
+Services that experience improved performance when selecting the "High performance" option include the following:
+
+* GitService - this is the service that synchronizes your Quix environment with the project's Git repository.
+* [Replay Service](replay.md) - enables replay of persisted data into a topic.
+* [Streaming Reader](../../apis/streaming-reader-api/index.md) - service that enables a client to subscribe to a Quix topic.
+* [Streaming Writer](../../apis/streaming-writer-api/index.md) - service that enables a client to publish to a Quix topic.
+* [Query API](../../apis/query-api/index.md) - query data persisted in the Quix database.
+
+Generally, if you notice sluggish performance in one of these services, it may mean for the volumes and frequency of data you are processing, you might need the High performance option.
+
+!!! tip
+
+    While you can't directly upgrade a standard environment to a high performance environment, you can create a new environment that uses the high performance option. You can create this environment using any branch (or even a new branch) suitable for your use case. For example, if you had a `staging` branch that was currently a standard environment, and you needed to upgrade it to a high performance environment, you could delete the environment, and create a new environment using the high performance option, and link it to the `staging` branch.
+
+### Use cases
+
+Standard is designed for proof-of-concept, experimentation, and testing environments.
+
+High performance is designed for production environments.
+
+## Next steps
+
+* [See the pricing page](https://quix.io/pricing){target=_blank}

docs/platform/how-to/create-project.md

@@ -6,6 +6,8 @@ This documentation describes how to create a new project, and populate it with t
 
     You can create as many environments in a project as you need. You can mark them as protected, and name them as needed, to align with your own development processes. This how-to simply shows one example project.
 
+You can also read more details about environments in [How to create an environment](create-environment.md).
+
 ## Creating a project
 
 To do anything useful with Quix, you'll need at least one project, and one environment. You can think of a project as corresponding to a Git repository, and an environment as corresponding to Git branch within that repository. 

docs/platform/integrations/kafka/confluent-cloud.md

@@ -5,7 +5,7 @@ Quix requires Kafka to provide streaming infrastructure for your Quix environmen
 When you create a new Quix environment, there are three hosting options:
 
 1. **Quix Broker** - Quix hosts Kafka for you. This is the simplest option as Quix provides hosting and configuration.
-2. **Self-Hosted Kafka** - This is where you already have existing Kafka infrastructure that you use, and you want to enable Quix to provide the stream processing platform on top of it. You can configure Quix to work with your existing Kafka infrastructure using this option.
+2. **Self-Hosted Kafka** - This is where you already have existing Kafka infrastructure that you use, and you want to enable Quix to provide the stream processing platform on top of it. You can configure Quix to work with your existing Kafka infrastructure using this option. You could use any Kafka solution here, for example [Redpanda](https://redpanda.com/){target=_blank}.
 3. **Confluent Cloud** - if you use Confluent Cloud for your Kafka infrastructure, then you can configure Quix to connect to your existing Confluent Cloud account.
 
 This documentation covers the third hosting option, Confluent Cloud. 

docs/platform/tutorials/computer-vision/add-service.md

@@ -2,7 +2,7 @@
 
 In this lab you use everything you've learned so far, to add a new service to the pipeline. Specifically, you add a service to publish the number of cars captured by the TfL cams to a new topic. You will then observe the number of cars change in real time using the waveform view of the Quix Data Explorer. This service could be useful if you want to easily store the number of cars, or perhaps create an alarm if the number of cars rises above a certain threshold. This service is a simple example of filtering - where you filter out data you are not interested in for subsequent processing.
 
-You develop this service on a feature branch, and then you create a PR to merge your new feature into the develop branch. This is a common pattern for development - you can test your new service on the feature branch, and then test again on the develop branch, before final integration into the production `main` branch.
+You develop this service on a feature branch, and then you create a PR to merge your new feature into the tutorial branch. This is a common pattern for development - you can test your new service on the feature branch, and then test again, before final integration into the production `main` branch.
 
 ## Create an environment
 
@@ -14,13 +14,11 @@ To create a new environment (and branch):
 
 2. Create a new environment called `Cars Only`.
 
-3. Create a new branch called `cars-only`. To do this, from the branch dropdown click `+ New branch` which displays the New branch dialog:
-
-    ![New branch](./images/new-branch.png)
+3. Create a new branch called `cars-only`. To do this, from the branch dropdown click `+ New branch` which displays the New branch dialog.
 
     !!! important
 
-        Make sure you branch from the `develop` branch, not `main`, as you are going to merge your changes onto the `develop` branch.
+        Make sure you branch from the `tutorial` branch, not `main`, as you are going to merge your changes back to the `tutorial` branch.
 
 4. Complete creation of the environment using the default options.
 
@@ -177,31 +175,25 @@ You now use the Quix Data Explorer to view the cars data in real time.
 
 ## Merge the feature
 
-Once you are sure that the changes on your feature branch are tested, you can then merge your changes onto the develop branch. Here your changes undergo further tests before finally being merged into production. 
+Once you are sure that the changes on your feature branch are tested, you can then merge your changes onto the tutorial branch. Here your changes undergo further tests before finally being merged into production. 
 
-To merge your feature branch, `cars-only` into `develop`:
+To merge your feature branch, `cars-only` into `tutorial`:
 
 1. Select `Merge request` from the menu as shown:
 
     ![Merge request menu](./images/merge-request-menu.png)
 
-2. In the `Merge request` dialog, set the `cars-only` branch to merge into the `develop` branch, as shown:
-
-    ![Merge request dialog](./images/merge-request-dialog.png)
+2. In the `Merge request` dialog, set the `cars-only` branch to merge into the `tutorial` branch.
 
 You are going to create a pull request, rather than perform a direct merge. This enables you to have the PR reviewed in GitHub (or other Git provider). You are also going to do a squash and merge, as much of the feature branch history is not required.
 
 To create the pull request:
 
 1. Click `Create pull request`. You are taken to your Git provider, in this case GitHub.
 
-2. Click the `Pull request` button:
-
-    ![Pull request GitHub](./images/pull-request-github.png)
+2. Click the `Pull request` button.
 
-3. Add your description, and then click `Create pull request`:
-
-    ![Pull request description](./images/pr-add-description.png)
+3. Add your description, and then click `Create pull request`.
 
 4. Get your PR reviewed and approved. Then squash and merge the commits:
 
@@ -213,21 +205,19 @@ To create the pull request:
 
         You can just merge, you don't have to squash and merge. You would then retain the complete commit history for your service while it was being developed. Squash and merge is used in this case by way of example, as the commit messages generated while the service was being developed were deemed to be not useful in this case.
 
-## Resync the Develop environment
-
-You have now merged your new feature into the `develop` branch in the Git repository. Your Quix view in the Develop environment is now out of sync with the Git repository. If you click on your Develop environment in Quix, you'll see it is now a commit (the merge commit) behind:
+## Resync the environment
 
-![Develop behind](./images/develop-behind.png)
+You have now merged your new feature into the `tutorial` branch in the Git repository. Your Quix view in the Tutorial environment is now out of sync with the Git repository. If you click on your Tutorial environment in Quix, you'll see it is now a commit (the merge commit) behind.
 
-You now need to make sure your Develop environment in Quix is synchronized with the Git repository. To do this:
+You now need to make sure your Tutorial environment in Quix is synchronized with the Git repository. To do this:
 
 1. Click on `Sync environment`. The `Sync environment` dialog is displayed.
 
 2. Review the changes and click `Sync environment`.
 
 3. Click `Go to pipeline`.
 
-Your new service will build and start in the Develop environment, where you can now carry out further testing. When you are satisfied this feature can be released tp production, then you would repeat the previous process to merge your changes to Production `main`.
+Your new service will build and start in the Tutorial environment, where you can now carry out further testing. When you are satisfied this feature can be released to production, then you would repeat the previous process to merge your changes to Production `main`.
 
 ## 🏃‍♀️ Next step
 

docs/platform/tutorials/computer-vision/get-project.md

@@ -29,7 +29,7 @@ To fork the repository:
 
 1. Navigate to the [Quix GitHub repository](https://github.com/quixio/computer-vision-demo){target="_blank"}.
 
-2. Click the `Fork` button to fork the repo into your GitHub account (or equivalent Git provider if you don't have a GitHub account). Make sure you fork all branches, as you will be looking at the `develop` branch.
+2. Click the `Fork` button to fork the repo into your GitHub account (or equivalent Git provider if you don't have a GitHub account). Make sure you fork all branches, as you will be looking at the `tutorial` branch.
 
     !!! tip 
 
@@ -76,15 +76,15 @@ To link Quix to this forked repository:
 
 You now need to add an environment to your project. This is explained in the following section.
 
-## Create your Develop environment
+## Create your environment
 
-A Quix project contains at least one branch. For the purposes of this tutorial you will examine the `develop` branch of the project. In a Quix project a branch is encapsulated in an environment. You'll create a `Develop` environment mapped to the `develop` branch of the repository.
+A Quix project contains at least one branch. For the purposes of this tutorial you will examine the `tutorial` branch of the project. In a Quix project a branch is encapsulated in an environment. You'll create a `Tutorial` environment mapped to the `tutorial` branch of the repository.
 
-Now create an environment called `Develop` which uses the `develop` branch:
+Now create an environment called `Tutorial` which uses the `tutorial` branch:
 
-1. Enter the environment name `Develop`.
+1. Enter the environment name `Tutorial`.
 
-2. Select the `develop` branch from the dropdown.
+2. Select the `tutorial` branch from the dropdown.
 
     Make sure the branch is protected, as shown in the following screenshot:
 
@@ -104,42 +104,41 @@ At this point you can wait a few minutes for the pipeline services to completely
 
 ## Configure credentials
 
-As the project uses Quix API credentials, you'll now need to configure your details for the services that require API keys.
+As the project uses Quix API credentials, you'll now need to configure your credentials for the services that require API keys. The main ones are:
+
+* TfL camera feed - TfL API key
+* Web UI - a bearer token ([PAT](../../how-to/personal-access-token-pat.md)) 
+
+You need to [create secrets](../../how-to/environment-variables.md#create-a-secret-variable) for these and then assign them to the appropriate [environment variables](../../how-to/environment-variables.md).
 
 ### TfL camera feed
 
-Open the service and edit the environment variable as shown here:
+To get this service to run, you'll need to configure it with your TfL API key.
+
+Create a new secret that contains your TfL API key. Now link that secret to the `tfl_api_key` environment variable in the TfL camera feed service.
 
-![TfL credentials](./images/tfl-credentials.png){width=60%}
+[Read more about environment variables and secret management](../../how-to/environment-variables.md).
 
 ### Web UI service
 
 Note if you just want to try out the UI without performing the following steps, you can do that in the [demo](https://app-demo-computervisiondemo-prod.deployments.quix.ai/){target=_blank}.
 
-When testing the UI you might find Google Maps does not load correctly for you - this is because the code has the Quix Google Maps API key. To work around this, you can set the Google Maps API key to an empty string. Google maps will then default to "developer mode" - the map will display correctly, but will be watermarked with 'for developer use'. To do this, in the Applications list, click on the `TfL image processing UI` application.
+You'll need a [PAT](../../how-to/personal-access-token-pat.md) for the UI as it uses the Streaming ReaderAPI which needs to be authenticated. 
 
-To set the Google Maps API key to an empty string, you need to edit `src/app/app.module.ts` and modify the `apiKey` field in `AgmCoreModule.forRoot` to the following:
+Once you have your PAT copied to the clipboard, create a new secret to contain it. Then link that secret to `bearerToken` environment variable for the UI.
 
-``` typescript
-AgmCoreModule.forRoot({
-      apiKey: ''
-    }),
-```
+[Read more about environment variables and secret management](../../how-to/environment-variables.md).
 
-Also, you need your [workspace ID](../../how-to/get-environment-id.md) and a [PAT](../../how-to/personal-access-token-pat.md) to get the UI fully working. Once you have these, you'll need to access the code for the service, and set these values.
+Other optional services may require similar configuration, for example, the Quix Amazon S3 connector service requires your S3 credentials if you want to use it.
+
+When testing the UI you see "For development purposes only" displayed on the map. Optionally, if you have a Google Maps API key, you can avoid this.
 
-In the file `src/app/services/quix.service.ts`, locate the following code, and replace the place holders with your values:
+To add your own Google Maps API key you need to edit `src/app/app.module.ts` and modify the `apiKey` field in `AgmCoreModule.forRoot` to include your Google Maps API key:
 
 ``` typescript
-/*~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-*/
-  /*WORKING LOCALLY? UPDATE THESE!*/
-  private workingLocally = false; // set to true if working locally
-  private token: string = '<your_pat>'; // Create a token in the Tokens menu and paste it here
-  public workspaceId: string = '<your_workspace_id>'; // Look in the URL for the Quix Portal your workspace ID is after 'workspace='
+AgmCoreModule.forRoot({apiKey: '<your_google_maps_api_key>'}),
 ```
 
-Other optional services may require similar configuration, for example, the Quix Amazon S3 connector service requires your S3 credentials if you want to use it.
-
 ## See also
 
 If you are new to Quix it is worth reviewing the [recent changes page](../../changes.md), as that contains very useful information about the significant recent changes, and also has a number of useful videos you can watch to gain familiarity with Quix.