From ece0fbab7e477c501e120a4f765241cfcdb63a02 Mon Sep 17 00:00:00 2001 From: Nick O'Leary Date: Thu, 19 Oct 2023 14:05:46 +0100 Subject: [PATCH 1/4] Update engineering section with more details on ops --- src/handbook/design/index.md | 2 +- .../development/how-we-work/production.md | 40 +++++++++++++ .../development/how-we-work/staging.md | 56 +++++-------------- src/handbook/development/index.md | 43 ++++++++++---- src/handbook/development/observability.md | 12 ++++ src/handbook/development/releases/process.md | 2 +- 6 files changed, 100 insertions(+), 55 deletions(-) create mode 100644 src/handbook/development/how-we-work/production.md diff --git a/src/handbook/design/index.md b/src/handbook/design/index.md index 7a8d0865f4..c69a524b98 100644 --- a/src/handbook/design/index.md +++ b/src/handbook/design/index.md @@ -1,6 +1,6 @@ --- navTitle: Design -navGroup: Development & Design Practices +navGroup: Engineering & Design Practices --- # Design diff --git a/src/handbook/development/how-we-work/production.md b/src/handbook/development/how-we-work/production.md new file mode 100644 index 0000000000..b6c5d45770 --- /dev/null +++ b/src/handbook/development/how-we-work/production.md @@ -0,0 +1,40 @@ +--- +navTitle: Production Environment +--- + +# Production Environment + +Our production environment runs in `EU-West-1` in a dedicated AWS account. + +Access to the AWS account is restricted and not generally available. If you believe +you have a need to access the AWS account, raise an issue in [CloudProject](https://github.com/FlowFuse/CloudProject) +and assign to the CTO for review. + +## Using production + +SSO is enabled for all `@flowfuse.com` employees. Everyone is a member of `FlowForge Team` +where Applications/Instances can be created. + +By default, FlowFuse employee accounts do not have Admin level access. A dedicated +admin account exists with details in `1Password` for those with approved admin level +access. + +If you want to create your own Team for work purposes, use your Brex card when setting +up the billing details. You can then request a Stripe Coupon to cover the costs +by raising an issue in [CloudProject](https://github.com/FlowFuse/CloudProject) + +## Deployment + +We deploy the latest product code on a weekly basis. This is done by creating a +new [maintenance release](../releases/process.md#unmanaged-releases), including updates +to the [`helm`](https://github.com/FlowFuse/helm) templates. They are then applied +manually to production by the engineering team. + +The automated deployment to staging has been prepared to apply to production with +a manual verification step blocking the final deploy. We have not yet fully enabled +this mode but we are getting closer to doing so. + +## Observability + + - [Observability](../observability.md) - how we monitor production + diff --git a/src/handbook/development/how-we-work/staging.md b/src/handbook/development/how-we-work/staging.md index 9fc98a4682..c5786d076d 100644 --- a/src/handbook/development/how-we-work/staging.md +++ b/src/handbook/development/how-we-work/staging.md @@ -8,27 +8,29 @@ We have a staging environment running on AWS which is a scaled down replica of our managed FlowFuse offering, with a separate domain. Staging URL and sign in details can be found in the Developer Vault in 1Password. -## AWS Account +## Deployment -It uses a separate AWS account ending in ..9937 +Any change to core product repositories triggers a series of GitHub Actions that results +in staging being updated with the latest code. This can take up to 30 minutes to complete. -Ben or ZJ can provision a user account for this account. +The deploy action can be monitored [here](https://github.com/FlowFuse/helm/actions/workflows/flowforge-container.yml). -The services are running in EU-West-1. -## Nodes +## AWS Account -The staging environment uses one node running on a t2.small for the management -app and a pair of t2.small nodes for the projects cluster. t2.small is the -smallest instance that can be used with EKS. +It uses a separate AWS account ending in ..9937. The services are running in EU-West-1. +Access to the AWS account is restricted and not generally available. If you believe +you have a need to access the AWS account, raise an issue in [CloudProject](https://github.com/FlowFuse/CloudProject) +and assign to the CTO for review. ## Email -Amazon SES is setup on staging however it is still running in sandbox mode which means only verified address & domains can RECEIVE emails from it, this is currently limited to flowforge.com email addresses and a small set of pre-approved disposable emails. +Amazon SES is setup on staging however it is still running in sandbox mode which means only verified address & domains can RECEIVE emails from it, this is currently limited to flowfuse.com email addresses and a small set of pre-approved disposable emails. -There is no intention to move this from sandbox as this helps to limit access to staging. +If you need to use another email address with staging then you should raise an issue +in [CloudProject](https://github.com/FlowFuse/CloudProject) and assign to the CTO for review. -If you need to use another email address with staging then you should verify the address through SES in the AWS Console. +If approved, the email will need to be added to the SES configuration in the AWS console. ### Test Email accounts @@ -38,38 +40,10 @@ of short-lived testing of sign-up and user management. The inboxes for these email addresses are publicly accessible if known, so the list is available on this private issue: https://github.com/FlowFuse/CloudProject/issues/135 -## Deployment - -Currently there is no auto deployment to staging, this should be rectified in the future so that staging is running the code in the main branches of the respective repos. - ## Using staging When setting up a team you'll need to enter billing details. For credit card details, use [the Stripe mock data](https://stripe.com/docs/testing#testing-interactively). -## Using the FlowFuse Device Agent with staging - -Staging uses pre-release npm packages stored in a GitHub npm repository. To be able to use these packages you will need to authenticate with the repository. - -You will need to create a GH Personal token - -1. Go to your [classic personal access tokens](https://github.com/settings/tokens) -1. Click Generate New Token (and again pick the classic option) -1. You will probably be prompted for 2FA now -1. Give the token a meaningful name -1. Pick an expiration. I went with no expiration so I don't have to do this again and I'm going to limit the scope -1. Tick the box next to `read:packages` -1. Click generate token button at bottom of page - -Store the token in your private 1Password vault - -Create the following .npmrc file: - -``` -//npm.pkg.github.com/:_authToken=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx -@flowforge:registry=https://npm.pkg.github.com/ -``` - -where `ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx` is the token you just generated. - -You need to place this in the project directory, e.g. `/opt/flowforge-device/project` or if you are running it in the dev env `flowforge-device-agent/var/project` (assume starting with `node index -d ./var -c ./var/device.yml`) +As the staging cluster is purposefully smaller than production, please be mindful of deleting +resources after use. \ No newline at end of file diff --git a/src/handbook/development/index.md b/src/handbook/development/index.md index 55fbbdcb52..2e37634b6f 100644 --- a/src/handbook/development/index.md +++ b/src/handbook/development/index.md @@ -1,26 +1,45 @@ --- -navTitle: Development -navGroup: Development & Design Practices +navTitle: Engineering +navGroup: Engineering & Design Practices --- -# Development +# Engineering + +The Engineering team is responsible for writing and maintaining the code for +FlowFuse's product and infrastructure. + +This includes, but is not limited to: + + - Developing new product features in accordance to the Product plan + - Providing technical input into the Product planning process - assisting with + scoping of items, technical prioritization and sizing. + - Providing technical support to our customers and community members + - Ensuring the ongoing operations of FlowFuse Cloud + ## How we work -- [Packaging](../development/how-we-work/packaging.md) -- [Contributing](../development/how-we-work/contributing.md) -- [Security Policy](../development/how-we-work/security.md) -- [Staging Environment](../development/how-we-work/staging.md) -- [Observability](../development/observability.md) +- [Planning](./releases/planning.md) - how we plan what we're doing +- [Release Process](./releases/process.md) - how we ship -## Releases -- [Planning](./releases/planning.md) -- [Process](./releases/process.md) +## Engineering Guides + +- [Packaging](../development/how-we-work/packaging.md) - how we manage repos and npm packaging +- [Contributing](../development/how-we-work/contributing.md) - tips on our development style to help get started contributing +- [Security Policy](../development/how-we-work/security.md) ## Front-End A collection of how-to's and best practice guides for FlowFuse's front-end development. + - [Data Attributes](./frontend/data-attributes.md) - [Services](./frontend/services.md) -- [Testing](./frontend/testing.md) \ No newline at end of file +- [Testing](./frontend/testing.md) + +## Operations + +How we run and manage our FlowFuse Cloud platform + +- [Staging Environment](../development/how-we-work/staging.md) +- [Observability](../development/observability.md) diff --git a/src/handbook/development/observability.md b/src/handbook/development/observability.md index a0b0a7f930..45ea75889a 100644 --- a/src/handbook/development/observability.md +++ b/src/handbook/development/observability.md @@ -34,3 +34,15 @@ Grafana is a popular open-source platform for creating, sharing, and managing da AWS CloudWatch is a monitoring and observability service that provides data and actionable insights for AWS, hybrid, and on-premises applications and infrastructure resources. In our case we use it to monitor infrastructure-related resources in AWS. [Link](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#home:) + +### Uptime Robot + +[Uptime Robot](https://uptimerobot.com/) is used to monitor our public facing sites, including FlowFuse Cloud. This polls +each endpoint at regular intervals and raises an alarm if an error is detected. The alerts are sent to `#ops-uptime-alerts` in slack +and emailed to the CTO. + +## Alerting + +Any [alerts](https://grafana.flowforge.com/alerting/list) that have been configured +in Grafana will post to the `#ops-alerts` channel in slack. The alert, where appropriate, +will include a link to the relevant section of the [Incident Playbook](https://docs.google.com/document/d/1NMPWEFgHkVNN7RqHXUgijEGdNwZH-SlaAspOQr9Vg9k/edit#heading=h.a7jq4bkz66hv). diff --git a/src/handbook/development/releases/process.md b/src/handbook/development/releases/process.md index f3ea22d347..666fadf254 100644 --- a/src/handbook/development/releases/process.md +++ b/src/handbook/development/releases/process.md @@ -3,7 +3,7 @@ navTitle: Release Process --- # Release Process -We do two types or releases: +We do two types of releases: - [Scheduled Releases](#scheduled-releases) - _for the planned release of the entire set of product repositories._ - [Fix Releases](#fix-releases) - _for ad-hoc release of individual repositories._ From 542d3c51f6538be89541e09d2f1b8572fd6010c1 Mon Sep 17 00:00:00 2001 From: Nick O'Leary Date: Thu, 19 Oct 2023 14:18:06 +0100 Subject: [PATCH 2/4] Add production info to eng index --- src/handbook/development/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/handbook/development/index.md b/src/handbook/development/index.md index 2e37634b6f..5042bc5ccc 100644 --- a/src/handbook/development/index.md +++ b/src/handbook/development/index.md @@ -41,5 +41,6 @@ A collection of how-to's and best practice guides for FlowFuse's front-end devel How we run and manage our FlowFuse Cloud platform +- [Production Environment](../development/how-we-work/production.md) - [Staging Environment](../development/how-we-work/staging.md) - [Observability](../development/observability.md) From 0a61a189fec257e06718221944d6dbfaf69712a1 Mon Sep 17 00:00:00 2001 From: Nick O'Leary Date: Thu, 19 Oct 2023 17:30:10 +0100 Subject: [PATCH 3/4] Big tidy up of dev section of handbook --- src/handbook/development/contributing.md | 2 +- .../{git-how-to.md => guides/git.md} | 12 +- src/handbook/development/guides/index.md | 11 ++ .../markdown.md} | 8 +- .../development/how-we-work/contributing.md | 49 ----- src/handbook/development/how-we-work/index.md | 14 -- .../development/how-we-work/packaging.md | 186 ------------------ .../development/how-we-work/security.md | 55 ------ src/handbook/development/index.md | 28 +-- src/handbook/development/ops/index.md | 0 .../development/{ => ops}/observability.md | 0 .../{how-we-work => ops}/production.md | 0 .../{how-we-work => ops}/staging.md | 0 src/handbook/development/packaging.md | 64 +++--- src/handbook/development/security.md | 5 +- src/handbook/development/staging.md | 68 ------- src/redirect.md | 10 + 17 files changed, 83 insertions(+), 429 deletions(-) rename src/handbook/development/{git-how-to.md => guides/git.md} (95%) create mode 100644 src/handbook/development/guides/index.md rename src/handbook/development/{markdown-how-to.md => guides/markdown.md} (91%) delete mode 100644 src/handbook/development/how-we-work/contributing.md delete mode 100644 src/handbook/development/how-we-work/index.md delete mode 100644 src/handbook/development/how-we-work/packaging.md delete mode 100644 src/handbook/development/how-we-work/security.md create mode 100644 src/handbook/development/ops/index.md rename src/handbook/development/{ => ops}/observability.md (100%) rename src/handbook/development/{how-we-work => ops}/production.md (100%) rename src/handbook/development/{how-we-work => ops}/staging.md (100%) delete mode 100644 src/handbook/development/staging.md diff --git a/src/handbook/development/contributing.md b/src/handbook/development/contributing.md index bcdb689369..bbd5420ce3 100644 --- a/src/handbook/development/contributing.md +++ b/src/handbook/development/contributing.md @@ -47,7 +47,7 @@ Take your time when committing files. Review each file carefully and ensure what #### Branching vs. Forking -Commits should never be pushed directly to `main`. Instead, branch or fork from the relevant branch (most likely `main`) and work from there. +Commits must never be pushed directly to `main`. Instead, branch or fork from the relevant branch (most likely `main`) and work from there. It is preferred that new work be added on a branch (rather than in a forked repository), although this is not enforced. Branch names should be short, informative, and if directly linked to a single issue number, reference such issue number, e.g. `29-issue-summary`. diff --git a/src/handbook/development/git-how-to.md b/src/handbook/development/guides/git.md similarity index 95% rename from src/handbook/development/git-how-to.md rename to src/handbook/development/guides/git.md index 6877c13b0d..f98283c439 100644 --- a/src/handbook/development/git-how-to.md +++ b/src/handbook/development/guides/git.md @@ -62,13 +62,13 @@ The first step to editing content is to create a [Branch](#branch) of that conte Navigate to the Project within [Github](#github) you want to work on, that would usually be [our website](https://github.com/FlowFuse/website) or [Handbook](https://github.com/FlowFuse/handbook). -![Create a new Branch](../images/git-how-to/new-branch-1.png) +![Create a new Branch](../../images/git-how-to/new-branch-1.png) The [Branch](#branch) name should give a brief overview of what you are planning to change in the [Project](#project), for example ‘Add Git guide for non-tech projects & staff’ then click the ‘create branch’ link. You should now notice that where the drop down said ‘main’ before it now says the name of the [Branch](#branch) you just created. -![Create a new Branch](../images/git-how-to/new-branch-2.png) +![Create a new Branch](../../images/git-how-to/new-branch-2.png) You can now start the process of actually creating or editing content, any changes you make will not yet be added to the [Published](#publish) version of the [Live](#live) [Project](#project) so don't worry if you make mistakes or are not yet happy with the finished product. @@ -78,7 +78,7 @@ In this example I am going to create a new document in the [Handbook](#handbook) Firstly I will create a new document called git-how-to.md in the design folder. -![Create a new file](../images/git-how-to/new-file-1.gif "@skip") +![Create a new file](../../images/git-how-to/new-file-1.gif "@skip") The file type is .md (Markdown). A [Markdown](#markdown) file is similar to a .docx or .txt. It allows you to lay out content in a document including text, images, titles, headers and tables. You can read more about [Markdown](#markdown) here. @@ -88,7 +88,7 @@ In the video above I pressed ‘[Commit](#commit) changes’ which is the same a I can now start the process of writing my document, first I will reopen it in the editor, then I will add the content. -![Edit the new file](../images/git-how-to/edit-file-1.gif "@skip") +![Edit the new file](../../images/git-how-to/edit-file-1.gif "@skip") I will work on the file until I think it's ready for a colleague to review the changes I have made. Once I am happy with the content I will [Commit](#commit) the changes as I did before. @@ -100,13 +100,13 @@ I am now ready to request a [Review](#review) of my work from a colleague. To do [Github](#github) gives you an easy to find button to create a [Pull Request](#pull-request) for your current work. -![Create a Pull Request](../images/git-how-to/create-a-pr-1.gif "@skip") +![Create a Pull Request](../../images/git-how-to/create-a-pr-1.gif "@skip") It's a good idea to provide your colleagues descriptive comments explaining the goals of the changes you have made as well as anything else you think would help them [review](#review) your work. Once you press the 'Create pull request' button an alert will be sent to one of FlowFuse's Slack channels letting everyone know you'd like your work reviewed. You can also request a [review](#review) from a specific colleague using the [Reviewers](#reviewer) section of your [Pull Request](#pull-request) Click on [Reviewers](#reviewer) then select the colleague you think would be best placed to [review](#review) your work. -![Request a review from a specific colleague](../images/git-how-to/nominate-a-reviewer-1.gif "@skip") +![Request a review from a specific colleague](../../images/git-how-to/nominate-a-reviewer-1.gif "@skip") ### Requesting a review of your work diff --git a/src/handbook/development/guides/index.md b/src/handbook/development/guides/index.md new file mode 100644 index 0000000000..194ffa6199 --- /dev/null +++ b/src/handbook/development/guides/index.md @@ -0,0 +1,11 @@ +--- +navTitle: Guides +--- + +# Guides + +A collection of guides for some of things that are foundational to working at +FlowFuse + +- [Git Intro](./git.md) - get started with Git +- [Markdown Guide](./markdown.md) - a quick primer for Markdown diff --git a/src/handbook/development/markdown-how-to.md b/src/handbook/development/guides/markdown.md similarity index 91% rename from src/handbook/development/markdown-how-to.md rename to src/handbook/development/guides/markdown.md index 36cb46ea42..1eb6f92fef 100644 --- a/src/handbook/development/markdown-how-to.md +++ b/src/handbook/development/guides/markdown.md @@ -1,8 +1,8 @@ --- -navTitle: Markdown How-To +navTitle: Markdown Guide --- -# Markdown How-To +# Markdown Guide This is a very short reference guide for those trying to write Markdown. For more comprehensive guides, we recommend looking at: @@ -76,10 +76,10 @@ from the previous paragraph. ## Links -[this will be a link](http://localhost:8080/blog/2023/02/flowforge-1-4-0-released/) +[this will be a link](http://example.com/) ```md -[this will be a link](http://localhost:8080/blog/2023/02/flowforge-1-4-0-released/) +[this will be a link](http://example.com/) ``` ## Images diff --git a/src/handbook/development/how-we-work/contributing.md b/src/handbook/development/how-we-work/contributing.md deleted file mode 100644 index 7c45563e3e..0000000000 --- a/src/handbook/development/how-we-work/contributing.md +++ /dev/null @@ -1,49 +0,0 @@ -# Contributing - -## Coding Best Practices - -### Linting - -All code repositories adopt our standard linting rules found in the [flowforge/.github repository](https://github.com/FlowFuse/.github/blob/main/.eslintrc). - -We use [StandardJS](https://standardjs.com/), with one exception - 4 spaces not 2. - -If you're using VSCode, then we recommend using the [ESLint extension](https://github.com/Microsoft/vscode-eslint) and setting `all` for the `Eslint › Code Actions On Save: Mode` setting: - -ESLint - Action on Save - -In the case of working with `vue` or `njk` files (found in the [frontend](https://github.com/FlowFuse/flowfuse/tree/main/frontend) and [website](https://github.com/FlowFuse/website) repositories), then you can add `vue` and `njk` to the `Eslint: Probe` setting in order to enable auto-formatting on save for these file types. - -ESLint - Probe - -## Git Best Practices - -### Committing - -Take care when adding files to a commit. It's easy just to `git add -A` (i.e. add all local changes to a commit) but this can result in commits and PRs being clogged with excessive changes that aren't linked to the actual issue/feature at hand. - -Take your time when committing files. Review each file carefully and ensure what you're adding to a commit is relevant and necessary. - -### Git Commit Messages - -- Capitalise the first letter, no trailing dot, 72 chars or less. -- First line should be an imperative/present tense, e.g. `Change` (not `Changed` or `Changes`) -- Do not include the issue number in the first line, this means that commit message are then suitable to include in a changelog as-is. -- Second line should either be blank, or reference to an issue/PR using one of the GitHub recognised keywords, e.g. `closes #...` `fixes #...` `part of #...` -- The remainder should be any further narrative that is needed. Wrapped at 72 chars. - -### Branching vs. Forking - -Commits should never be pushed directly to `main`. Instead, branch or fork from the relevant branch (most likely `main`) and work from there. - -It is preferred that new work be added on a branch (rather than in a forked repository), although this is not enforced. Branch names should be short, informative, and if directly linked to a single issue number, reference such issue number, e.g. `29-issue-summary`. - -Once code is merged, please close any related branches in order to keep the repository tidy. - -### Pull Requests - -PRs, when opened, should have at least one reviewer assigned, and a consequent review approved, before any merge takes place. If a PR is opened for review/discussion purposes, this PR should be set to `draft` state. - -When merging a PR, you should choose the "Merge pull request" option. There is no need to rebase or squash the PR commits. - -When conducting a PR review, if you are the last (or only) reviewer and all reviews (including your own) are approvals, unless there is a comment from the author stating otherwise, you are free to conduct the merge. Otherwise, leave the merge to the author of the PR, or a future reviewer. diff --git a/src/handbook/development/how-we-work/index.md b/src/handbook/development/how-we-work/index.md deleted file mode 100644 index a2bc7110f2..0000000000 --- a/src/handbook/development/how-we-work/index.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -navTitle: How We Work ---- - -# How we Work - Development - -This section is useful to read if you are going to be contributing to FlowFuse. It contains -details on our development best practices, security practices and how to navigate and understand -our GitHub processes. - -- [Contributing](./contributing.md) -- [Packaging](./packaging.md) -- [Security](./security.md) -- [Staging](./staging.md) \ No newline at end of file diff --git a/src/handbook/development/how-we-work/packaging.md b/src/handbook/development/how-we-work/packaging.md deleted file mode 100644 index 6577c4c379..0000000000 --- a/src/handbook/development/how-we-work/packaging.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -navTitle: Packaging Guidelines ---- - -# Packaging Guidelines - -This section describes the requirements we have for all GitHub repositories, -and npm modules we maintain. - -To help ensure all of the requirements are met, an issue should be raised in -[`flowforge/admin`](https://github.com/FlowFuse/admin/issues/new/choose){rel="nofollow"} -using the `New Repository Checklist` and then worked through. - -## GitHub projects - -### Naming - -- FlowFuse Components should start with `flowfuse-` -- If a Node-RED plugin/node should start with `flowfuse-nr-` -- Installers or Orchestration projects are named without the leading `flowfuse-` e.g. `installer` or `helm` - -### Settings - -- A rule should be added under the projects `settings/branches` to prevent pushing directly to the `main` branch - -### Content - -All Git Repositories must contain the following files: - - - `README.md` - - `LICENSE` - -### Linting - -All code repositories should adopt our standard linting rules by copying the -`.eslintrc` from the [flowforge/.github repository](https://github.com/FlowFuse/.github/blob/main/.eslintrc). - -If a repository has any additional requirements for linting, such as handling Vue -code, then additional plugins can be added. - -We use [StandardJS](https://standardjs.com/), with one exception - 4 spaces not 2. - -### Notifications - -Repositories should be added to the appropriate Slack channel. For core code repositories, -this would be in the `#gh-flowforge` channel. - -To create a subscription, go to that channel and type the message: - -``` -/github subscribe flowforge/NAME-OF-REPO comments reviews -``` - -This will subscribe to any notifications covering: `issues`, `pulls`, `commits`, `releases`, `deployments`, `comments` and `reviews`. - -### Workflows - -#### Project Automation - -All code repositories must have the Project Automation workflow added. This is done -by adding [`.github/workflows/project-automation.yml`](https://github.com/FlowFuse/flowfuse/blob/main/.github/workflows/project-automation.yml). -This workflow will ensure any opened issues are automatically added to the [Product board](https://github.com/orgs/FlowFuse/projects/3) where it can be triaged and prioritised. - -#### Release Publish - -For any repositories that contain modules to be published to npm, they should also -have a copy of [`.github/workflows/release-publish.yml`](https://github.com/FlowFuse/flowfuse/blob/main/.github/workflows/release-publish.yml). - -This workflow will publish to npm whenever the repository is tagged with a `vX.Y.Z` format -tag. - -Note that each repository may have slightly different pre-publish requirements - for -example if there is a build step or not. You may need to customise the workflow -to match what is needed. - -#### Private Repositories - -For *private* repositories, you will also need to add a Repository Secret as -they cannot access the organisation-wide secret we have in place. - -1. Generate a [Personal Access Token](https://github.com/settings/tokens) with - `repo, write:org` scope. - -2. Add it as a Repository Secret to the Private Repo (https://github.com/FlowFuse//settings/secrets/actions) - with the name `PROJECT_ACCESS_TOKEN` - -### Labels - -We have a standard set of labels that should be applied to all repositories. This -ensures we have a consistent approach to planning and tracking of work. - - - Type: `epic`, `story`, `task`, `bug`, `artwork` - - Sizing: `XS - 1`, `S - 2`, `M - 3`, `L - 5`, `XL - 8`, `XXL - 13` - - Area: `area:docs`, `area:db`, `area:migration`, `area:frontend`, `area:api`, `area:device`, `area:billing`, `area:infrastrucutre`, `area:install`, `design` - - Priority: `priority:high`, `priority:medium`, `priority:low` - - Status: `blocked`, `consideration` - - Product Scope: `scope:devices`, `scope:enterprise`, `scope:node-red`, `scope:collaboration` - - Other: `good first issue`, `upstream`, `needs-triage`, `headline`, `backport`, `research` - -The labels are synchronized across the repositories via a GitHub Action in the [`.github`](https://github.com/FlowFuse/.github) -repository. - -New repositories must be added to the list in [`flowforge-repositories.yml`](https://github.com/FlowFuse/.github/blob/main/flowforge-repositories.yml), -and then the [Synchronize Labels](https://github.com/FlowFuse/.github/actions/workflows/sync-labels.yml) action manually run. - -## NPM packages - -### Naming - -- All packages should be scoped to `@flowforge` - -Node-RED plugins should start with `nr-` e.g. - - @flowforge/nr-storage - - @flowforge/nr-auth - -FlowFuse plugins should start with `forge-` e.g. - - - @flowforge/forge-driver-localfs - - @flowforge/forge-driver-docker - - @flowforge/forge-driver-k8s - -### Content - -The `package.json` must contain the following keys - - - description - - repository - ``` - "repository": { - "type": "git", - "url": "git+https://github.com/FlowFuse/flowforge.git" - }, - ``` - - homepage - ``` - "homepage": "https://github.com/FlowFuse/flowforge#readme", - ``` - - bugs - ``` - "bugs": { - "url": "https://github.com/FlowFuse/flowfuse/issues" - }, - ``` - - license - - Apache-2.0 - - SEE LICENSE IN ./LICENSE - - author - ``` - "author": { - "name": "FlowFlow Inc." - }, - ``` - - engines - ``` - "engines": { - "node": ">=16.x" - } - ``` - -### Package Version Numbering - -Package numbers should follow the Semantic Versioning Scheme as laid out on [semver.org](https://semver.org/). - -Each component will stay in step with the core flowforge/flowforge release numbering for `major.minor` but will increment their own `fix` values as needed. e.g. On release of v0.2.0 all components will tag and release v0.2.0, but can independently release v0.2.1 as needed. - -Major and minor releases will follow the schedule laid out in the [Cadence](../releases/planning#cadence) section of the handbook. - -A Fix release can be made at any time, depending on the best judgement of the engineer making the fix but requires a review by another team member. - -The process for making a release is documented [here](../releases/process.md). - -### Adding NPM packages to Stacks - -As we build more FlowFuse specific nodes we will need to add these to the Stacks - -#### Localfs - -Currently bundled packages for the localfs driver need to be added to the [flowforge-nr-launcher](https://github.com/FlowFuse/flowforge-nr-launcher) `package.json` and the path to the node needs to be added to the `nodesDir` array in the `lib/launcher.js` file (around line 70). This will be updated in the next release to be controlled by the [flowforge-driver-localfs](https://github.com/FlowFuse/flowforge-driver-localfs) project - -#### Docker - -Any nodes or themes should be added to the `package.json` in `node-red-container/` directory of the [docker-compose](https://github.com/FlowFuse/docker-compose) project - -#### K8s - -Any nodes or themes should be added to the `package.json` in `node-red-container/` directory of the [helm](https://github.com/FlowFuse/helm) project diff --git a/src/handbook/development/how-we-work/security.md b/src/handbook/development/how-we-work/security.md deleted file mode 100644 index 59de7ad14f..0000000000 --- a/src/handbook/development/how-we-work/security.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -navTitle: Security Policy ---- -# Security Policy - -## Reporting a Vulnerability - -Please report any vulnerabilities discovered in FlowFuse products to security@flowfuse.com. - -We will endeavour to acknowledge and fix any reported vulnerabilities ASAP based -on its severity and assessed impact to our users. - -## Bug Bounties - -At our sole discretion, we offer rewards for responsibly disclosed issues according -to their severity. - -Please note the following are general guidelines and any reward decisions are up -to the discretion of FlowFuse. We keep these reward levels under review to ensure -they provide a fair reflection of the issues being reported. - - -Low | Medium | High | Critical -------|----------|--------|---------- -$128 | $256 | $512 | $1024 - - -Multiple reports of the same issue manifesting in different ways will be treated -as a single report. - -The quality of bug report will also impact any ultimate reward decision. - -### Sample criteria - -The following are examples of the types of bug for each severity. It is not -definitive and each reported bug will be considered on its own merits. - -*Critical Severity Bugs* - - - SQL Injection - - Remote Code Execution - - Privilege Escalation - -*High Severity Bugs* - - - Cross-Site Request Forgery - - Information leaks of user data - -*Medium Severity Bugs* - - - Information leaks of non-user data - -*Low Severity Bugs* - - - Exposure of any integer resource IDs (primary keys in our database) diff --git a/src/handbook/development/index.md b/src/handbook/development/index.md index 5042bc5ccc..7e7b0ac76a 100644 --- a/src/handbook/development/index.md +++ b/src/handbook/development/index.md @@ -16,19 +16,17 @@ This includes, but is not limited to: - Providing technical support to our customers and community members - Ensuring the ongoing operations of FlowFuse Cloud +## General -## How we work +- [Security Policy](./security.md) +- [Packaging](./packaging.md) - how we manage repos and npm packaging +- [Contributing](./contributing.md) - tips on our development style to help get started contributing + +## Releases - [Planning](./releases/planning.md) - how we plan what we're doing - [Release Process](./releases/process.md) - how we ship - -## Engineering Guides - -- [Packaging](../development/how-we-work/packaging.md) - how we manage repos and npm packaging -- [Contributing](../development/how-we-work/contributing.md) - tips on our development style to help get started contributing -- [Security Policy](../development/how-we-work/security.md) - ## Front-End A collection of how-to's and best practice guides for FlowFuse's front-end development. @@ -41,6 +39,14 @@ A collection of how-to's and best practice guides for FlowFuse's front-end devel How we run and manage our FlowFuse Cloud platform -- [Production Environment](../development/how-we-work/production.md) -- [Staging Environment](../development/how-we-work/staging.md) -- [Observability](../development/observability.md) +- [Production Environment](./ops/production.md) +- [Staging Environment](./ops/staging.md) +- [Observability](./ops/observability.md) + +## Guides + +A collection of guides for some of things that are foundational to working at +FlowFuse + +- [Git Intro](./guides/git.md) - get started with Git +- [Markdown Guide](./guides/markdown.md) - a quick primer for Markdown diff --git a/src/handbook/development/ops/index.md b/src/handbook/development/ops/index.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/handbook/development/observability.md b/src/handbook/development/ops/observability.md similarity index 100% rename from src/handbook/development/observability.md rename to src/handbook/development/ops/observability.md diff --git a/src/handbook/development/how-we-work/production.md b/src/handbook/development/ops/production.md similarity index 100% rename from src/handbook/development/how-we-work/production.md rename to src/handbook/development/ops/production.md diff --git a/src/handbook/development/how-we-work/staging.md b/src/handbook/development/ops/staging.md similarity index 100% rename from src/handbook/development/how-we-work/staging.md rename to src/handbook/development/ops/staging.md diff --git a/src/handbook/development/packaging.md b/src/handbook/development/packaging.md index 0d8060128f..7e046ea9b3 100644 --- a/src/handbook/development/packaging.md +++ b/src/handbook/development/packaging.md @@ -1,23 +1,23 @@ --- -originalPath: development/packaging.md -updated: 2022-11-24 22:12:28 +0000 +navTitle: Packaging Guidelines --- -# FlowFuse Packaging Guidelines + +# Packaging Guidelines This section describes the requirements we have for all GitHub repositories, and npm modules we maintain. To help ensure all of the requirements are met, an issue should be raised in -[`flowforge/admin`](https://github.com/FlowFuse/admin/issues/new/choose) using -the `New Repository Checklist` and then worked through. +[`FlowFuse/admin`](https://github.com/FlowFuse/admin/issues/new/choose){rel="nofollow"} +using the `New Repository Checklist` and then worked through. -## Github projects +## GitHub projects ### Naming -- Flow Forge Components should start with `flowforge-` -- If a Node-RED plugin/node should start with `flowforge-nr-` -- Installers or Orchestration projects are named without the leading `flowforge-` e.g. `installer` or `helm` +- FlowFuse Components should start with `flowfuse-` +- If a Node-RED plugin/node should start with `nr-` +- Installers or Orchestration projects are named without the leading `flowfuse-` e.g. `installer` or `helm` ### Settings @@ -33,7 +33,7 @@ All Git Repositories must contain the following files: ### Linting All code repositories should adopt our standard linting rules by copying the -`.eslintrc` from the [flowforge/.github repository](https://github.com/FlowFuse/.github/blob/main/.eslintrc). +`.eslintrc` from the [FlowFuse/.github repository](https://github.com/FlowFuse/.github/blob/main/.eslintrc). If a repository has any additional requirements for linting, such as handling Vue code, then additional plugins can be added. @@ -43,12 +43,12 @@ We use [StandardJS](https://standardjs.com/), with one exception - 4 spaces not ### Notifications Repositories should be added to the appropriate Slack channel. For core code repositories, -this would be in the `#gh-flowforge` channel. +this would be in the `#gh-flowfuse` channel. To create a subscription, go to that channel and type the message: ``` -/github subscribe flowforge/NAME-OF-REPO comments reviews +/github subscribe FlowFuse/NAME-OF-REPO comments reviews ``` This will subscribe to any notifications covering: `issues`, `pulls`, `commits`, `releases`, `deployments`, `comments` and `reviews`. @@ -89,19 +89,18 @@ they cannot access the organisation-wide secret we have in place. We have a standard set of labels that should be applied to all repositories. This ensures we have a consistent approach to planning and tracking of work. - - Type: `epic`, `story`, `task`, `bug` - - Sizing: `1`, `2`, `3`, `5`, `8`, `13` - - Area: `area:docs`, `area:db`, `area:migration`, `area:frontend`, `area:api`, `area:device`, `area:marketing`, `area:customer-success` - - Priority: `priority:high`, `priority:medium`, `priority:low`, `Customer Request` - - Status: `blocked` +The definitive list of labels is maintained [here](https://github.com/FlowFuse/.github/blob/main/labels.json) +and are synchronized across all repositories via GitHub Actions. + + - Type: `epic`, `story`, `task`, `bug`, `artwork` + - Sizing: `XS - 1`, `S - 2`, `M - 3`, `L - 5`, `XL - 8`, `XXL - 13` + - Area: `area:docs`, `area:db`, `area:migration`, `area:frontend`, `area:api`, `area:device`, `area:billing`, `area:infrastrucutre`, `area:install`, `design` + - Priority: `priority:high`, `priority:medium`, `priority:low` + - Status: `blocked`, `consideration` - Product Scope: `scope:devices`, `scope:enterprise`, `scope:node-red`, `scope:collaboration` - - Other: `good first issue`, `upstream`, `needs-triage` + - Other: `good first issue`, `upstream`, `needs-triage`, `headline`, `backport`, `research` -The labels are synchronized across the repositories via a GitHub Action in the [`.github`](https://github.com/FlowFuse/.github) -repository. -The `Customer Request` label is intended solely for issues requested by an important enterprise customer, not for leads or individuals like [Harry the Hobbyist](../product/personas/). - New repositories must be added to the list in [`flowforge-repositories.yml`](https://github.com/FlowFuse/.github/blob/main/flowforge-repositories.yml), and then the [Synchronize Labels](https://github.com/FlowFuse/.github/actions/workflows/sync-labels.yml) action manually run. @@ -109,13 +108,16 @@ and then the [Synchronize Labels](https://github.com/FlowFuse/.github/actions/wo ### Naming +*Note:* we are migrating to the `@flowfuse` scope of packages. See [this issue](https://github.com/FlowFuse/admin/issues/211) +for details. This section of the handbook will be updated as part of the migration. + - All packages should be scoped to `@flowforge` Node-RED plugins should start with `nr-` e.g. - @flowforge/nr-storage - @flowforge/nr-auth -Flow Forge plugins should start with `forge-` e.g. +FlowFuse plugins should start with `forge-` e.g. - @flowforge/forge-driver-localfs - @flowforge/forge-driver-docker @@ -130,12 +132,12 @@ The `package.json` must contain the following keys ``` "repository": { "type": "git", - "url": "git+https://github.com/FlowFuse/flowfuse.git" + "url": "git+https://github.com/FlowFuse/flowforge.git" }, ``` - homepage ``` - "homepage": "https://github.com/FlowFuse/flowfuse#readme", + "homepage": "https://github.com/FlowFuse/flowforge#readme", ``` - bugs ``` @@ -149,7 +151,7 @@ The `package.json` must contain the following keys - author ``` "author": { - "name": "FlowFlow Inc." + "name": "FlowFuse Inc." }, ``` - engines @@ -163,15 +165,13 @@ The `package.json` must contain the following keys Package numbers should follow the Semantic Versioning Scheme as laid out on [semver.org](https://semver.org/). -Each component will stay in step with the core FlowFuse/flowfuse release numbering for `major.minor` but will increment their own `fix` values as needed. e.g. On release of v0.2.0 all components will tag and release v0.2.0, but can independently release v0.2.1 as needed. +Each component will stay in step with the core flowforge/flowforge release numbering for `major.minor` but will increment their own `fix` values as needed. e.g. On release of v0.2.0 all components will tag and release v0.2.0, but can independently release v0.2.1 as needed. -Major and minor releases will follow the schedule laid out in the [Cadence](./releases/planning#cadence) -section of the handbook. +Major and minor releases will follow the schedule laid out in the [Cadence](../releases/planning#cadence) section of the handbook. A Fix release can be made at any time, depending on the best judgement of the engineer making the fix but requires a review by another team member. -The process for making a release is documented [here](./releases/process/). - +The process for making a release is documented [here](../releases/process.md). ### Adding NPM packages to Stacks @@ -179,7 +179,7 @@ As we build more FlowFuse specific nodes we will need to add these to the Stacks #### Localfs -Currently bundled packages for the localfs driver need to be added to the [nr-launcher](https://github.com/FlowFuse/nr-launcher) `package.json` and the path to the node needs to be added to the `nodesDir` array in the `lib/lancher.js` file (around line 70). This will be updated in the next release to be controlled by the [flowforge-driver-localfs](https://github.com/FlowFuse/driver-localfs) project +Currently bundled packages for the localfs driver need to be added to the [nr-launcher](https://github.com/FlowFuse/nr-launcher) `package.json` and the path to the node needs to be added to the `nodesDir` array in the `lib/launcher.js` file (around line 70). This will be updated in the next release to be controlled by the [driver-localfs](https://github.com/FlowFuse/driver-localfs) project #### Docker diff --git a/src/handbook/development/security.md b/src/handbook/development/security.md index 8981fc3527..c0491c69c8 100644 --- a/src/handbook/development/security.md +++ b/src/handbook/development/security.md @@ -1,9 +1,8 @@ --- -originalPath: development/security.md -updated: 2022-08-04 17:05:04 +0100 +navTitle: Security Policy --- -# Security Policy +# Security Reporting Policy This policy relates to external disclosures of potential vulnerabilities in the FlowFuse platform and associated components. diff --git a/src/handbook/development/staging.md b/src/handbook/development/staging.md deleted file mode 100644 index 664bcb0ae6..0000000000 --- a/src/handbook/development/staging.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -originalPath: development/staging.md -updated: 2022-11-24 12:11:19 +0100 ---- -# Staging Environment - -We have a staging environment running on AWS which is a scaled down replica of -our managed FlowFuse offering, with a separate domain. Staging URL and sign in -details can be found in the Developer Vault in 1Password. - -## AWS Account - -It uses a separate AWS account ending in ..9937 - -Ben or ZJ can provision a user account for this account. - -The services are running in EU-West-1. - -## Nodes - -The staging environment uses one node running on a t2.small for the management -app and a pair of t2.small nodes for the projects cluster. t2.small is the -smallest instance that can be used with EKS. - -## Email - -The staging environment can only be used with a set of pre-approved email addresses. This currently includes: - - - All `@flowfuse.com` addresses - with SSO enabled - - A collection of disposable email accounts that can be used to test user sign-up/invitations etc. Details of the available addresses are in [this issue](https://github.com/FlowFuse/CloudProject/issues/135). If you use one of these addresses for some testing, please delete the user once you are done so it is available for future use. - -The list of valid email addresses is managed via the Amazon SES configuration in the staging acount. - -## Deployment - -Currently there is no auto deployment to staging, this should be rectified in the future so that staging is running the code in the main branches of the respective repos. - -## Using staging - -When setting up a team you'll need to enter billing details. For credit card -details, use [the Stripe mock data](https://stripe.com/docs/testing#testing-interactively). - -## Using the FlowFuse Device Agent with staging - -Staging uses pre-release npm packages stored in a GitHub npm repository. To be able to use these packages you will need to authenticate with the repository. - -You will need to create a GH Personal token - -1. Go to your [classic personal access tokens](https://github.com/settings/tokens) -1. Click Generate New Token (and again pick the classic option) -1. You will probably be prompted for 2FA now -1. Give the token a meaningful name -1. Pick an expiration. I went with no expiration so I don't have to do this again and I'm going to limit the scope -1. Tick the box next to `read:packages` -1. Click generate token button at bottom of page - -Store the token in your private 1Password vault - -Create the following .npmrc file: - -``` -//npm.pkg.github.com/:_authToken=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx -@flowforge:registry=https://npm.pkg.github.com/ -``` - -where `ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx` is the token you just generated. - -You need to place this in the project directory, e.g. `/opt/flowforge-device/project` or if you are running it in the dev env `flowforge-device-agent/var/project` (assume starting with `node index -d ./var -c ./var/device.yml`) diff --git a/src/redirect.md b/src/redirect.md index 7009dcd4a0..b63040a4e8 100644 --- a/src/redirect.md +++ b/src/redirect.md @@ -30,6 +30,16 @@ redirects: - { "from": "/blog/2023/07/node-explained-filter/", "to": "/node-red/core-nodes/rbe/"} - { "from": "/blog/2022/12/node-red-exec-explained/", "to": "/node-red/core-nodes/exec/"} - { "from": "/blog/2023/05/Integrating Modbus with Node-RED", "to": "/blog/2023/05/integrating-modbus-with-node-red/"} + - { "from": "/handbook/development/observability/", "to": "/handbook/development/ops/observability/" } + - { "from": "/handbook/development/staging/", "to": "/handbook/development/ops/staging/" } + - { "from": "/handbook/development/how-we-work/staging/", "to": "/handbook/development/ops/staging/" } + - { "from": "/handbook/development/how-we-work/security/", "to": "/handbook/development/security/" } + - { "from": "/handbook/development/how-we-work/packaging/", "to": "/handbook/development/packaging/" } + - { "from": "/handbook/development/how-we-work/contributing/", "to": "/handbook/development/contributing/" } + - { "from": "/handbook/development/how-we-work/", "to": "/handbook/development/" } + - { "from": "/handbook/development/git-how-to/", "to": "/handbook/development/guides/git" } + - { "from": "/handbook/development/markdown-how-to/", "to": "/handbook/development/guides/markdown" } + # The "permalink" attribute determines where the output page will be located. permalink: "{{ redirect.from }}" From 73b8206c7d5a990da152c415a34af00a03aa19bb Mon Sep 17 00:00:00 2001 From: Nick O'Leary Date: Thu, 19 Oct 2023 17:43:53 +0100 Subject: [PATCH 4/4] Fix links and update access/request issue link --- src/handbook/development/ops/index.md | 7 +++++++ src/handbook/development/ops/production.md | 2 +- src/handbook/development/ops/staging.md | 5 ++--- src/handbook/development/packaging.md | 4 ++-- 4 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/handbook/development/ops/index.md b/src/handbook/development/ops/index.md index e69de29bb2..df2a990e5b 100644 --- a/src/handbook/development/ops/index.md +++ b/src/handbook/development/ops/index.md @@ -0,0 +1,7 @@ +--- +navTitle: Platform Ops +--- + + - [Observability](./observability.md) + - [Staging Environment](./staging.md) + - [Production Environment](./production.md) \ No newline at end of file diff --git a/src/handbook/development/ops/production.md b/src/handbook/development/ops/production.md index b6c5d45770..4c744c6df8 100644 --- a/src/handbook/development/ops/production.md +++ b/src/handbook/development/ops/production.md @@ -7,7 +7,7 @@ navTitle: Production Environment Our production environment runs in `EU-West-1` in a dedicated AWS account. Access to the AWS account is restricted and not generally available. If you believe -you have a need to access the AWS account, raise an issue in [CloudProject](https://github.com/FlowFuse/CloudProject) +you have a need to access the AWS account, raise an [Access/Permission Request issue](https://github.com/FlowFuse/admin/issues/new/choose) and assign to the CTO for review. ## Using production diff --git a/src/handbook/development/ops/staging.md b/src/handbook/development/ops/staging.md index c5786d076d..e6488ddcdd 100644 --- a/src/handbook/development/ops/staging.md +++ b/src/handbook/development/ops/staging.md @@ -18,9 +18,8 @@ The deploy action can be monitored [here](https://github.com/FlowFuse/helm/actio ## AWS Account -It uses a separate AWS account ending in ..9937. The services are running in EU-West-1. Access to the AWS account is restricted and not generally available. If you believe -you have a need to access the AWS account, raise an issue in [CloudProject](https://github.com/FlowFuse/CloudProject) +you have a need to access the AWS account, raise an [Access/Permission Request issue](https://github.com/FlowFuse/admin/issues/new/choose) and assign to the CTO for review. ## Email @@ -38,7 +37,7 @@ We have enabled a small list of mailinator.com based email addresses for the pur of short-lived testing of sign-up and user management. The inboxes for these email addresses are publicly accessible if known, so the list -is available on this private issue: https://github.com/FlowFuse/CloudProject/issues/135 +is available on [this private issue](https://github.com/FlowFuse/CloudProject/issues/135). ## Using staging diff --git a/src/handbook/development/packaging.md b/src/handbook/development/packaging.md index 7e046ea9b3..00492a447b 100644 --- a/src/handbook/development/packaging.md +++ b/src/handbook/development/packaging.md @@ -167,11 +167,11 @@ Package numbers should follow the Semantic Versioning Scheme as laid out on [sem Each component will stay in step with the core flowforge/flowforge release numbering for `major.minor` but will increment their own `fix` values as needed. e.g. On release of v0.2.0 all components will tag and release v0.2.0, but can independently release v0.2.1 as needed. -Major and minor releases will follow the schedule laid out in the [Cadence](../releases/planning#cadence) section of the handbook. +Major and minor releases will follow the schedule laid out in the [Cadence](./releases/planning#cadence) section of the handbook. A Fix release can be made at any time, depending on the best judgement of the engineer making the fix but requires a review by another team member. -The process for making a release is documented [here](../releases/process.md). +The process for making a release is documented [here](./releases/process.md). ### Adding NPM packages to Stacks