Master
Vulnerability Scans
Frontend:
Server:
Staging:
Platform for sale of Christmas tree permits and intake of special use applications for the U.S. Forest Service
The U.S. Forest Service is engaged in an ongoing effort to modernize and simplify its permitting processes. One facet of this effort is to make special use permits available to obtain online. The USForest Service and GSA's 18F acquisition built this open source platform for noncommercial group use, temporary outfitter and guide, and Christmas tree permits.
As the first two-way interaction-focused Forest Service online application, Open Forest will strengthen the connection between the public and the National Forests. The application will broaden and increase the public’s responsible access to public lands through online availability; a predictable, responsive and friendly experience; and reduced administrative burden. These enhancements will drive increased customer satisfaction and quality of public engagement.
The Forest Service will use agile development methods and modular procurements to develop the application, ensuring an adaptive process that responds to emergent user needs and changing requirements.
- Welcome
- Opportunity Statement
- Table of contents
- Development
- Overview
- Requirements
- Server development
- Frontend Development
- Install angular cli
- Navigate to frontend directory
- Install dependencies
- Development server
- Build
- Running unit tests
- Running end-to-end tests
- Testing WCAG2AA compliance with pa11y
- Typedoc
- Build typedoc
- Available services
- Pay.gov error mocking in local environment
- Pay.gov in QA environment
- Christmas trees sidebar template
- Basic Auth
- Deployment
- Docs
- Content administration
- Usability testing
- Known technical debt
- Known UX debt
- Contributing
- Public domain
This repository comprises the application code for the platform. It includes a node.js server, and a Angular 2+ static frontend, currently deployed on cloud.gov.
Install Git
All contributors should use git-seekret (https://github.com/18f/laptop#git-seekret) to prevent accidental commits of sensitive data.
Project specific rules are defined on the wiki.
git clone [repository url] fs-open-forest-platform
Navigate to cloned repo
cd fs-open-forest-platform
There are two options for development - Docker or installing the dependencies independently.
Run docker-compose up. Navigate to http://localhost:4200/. The app will automatically reload if you change any of the source files.
** The following instructions outline tools and procedures required for local development without docker **
Install Node ^8.15.1
We recommend installing node such that it does not require the sudo command to run and different versions can be easily managed, for example, using NVM
Make sure NPM is up to date npm install -g npm
Install PostgreSQL
Run cd server then run npm install to install dependencies.
A running Postgresql database is required in order to run the server locally. Please make sure you have installed Postgresql.
Create a new database user with the password fs_open_forest and the CREATEDB
permission. You will be prompted for the password.
$ createuser fs_open_forest --createdb -P
Then you can initialize your development database.
$ npm run createdb
$ npm run migrate
$ npm run seed
The database can support promisified transactions by leveraging the server/dba/migrations/modules/transaction.js
in order to have a shared ID across new permit types an ALTER TABLE query must be used. See migration 36 how to construct this.
There are environment variables that are required to be set in order to run tests and to run the server in general. Please set up these environment variables either in your shell or on the command line. We use a cloud foundry (the underlying framework for cloud.gov) specific manager of environment variables.
To run any of the server commands, either the environment variables above must be available in your shell or on the command line, and you must be in the server directory.
| Function | Command | Additional information |
|---|---|---|
| Setup Database | npm run migrate |
|
| Seed Database | npm run seed |
Adds Christmas tree forest data |
| Start the server | npm run dev |
Server is accessible at http://localhost:8080 |
| Revert the last database migration | npm run undoLastMigrate |
|
| Revert all of the database migrations and start with a blank database | npm run undoAllMigrate |
|
| Run eslint for linting | npm run lint |
The linting results will be put into server/lint-results.html. |
| To run all of the tests locally | npm run test |
Be sure your Postgresql server is running |
| To run code coverage locally | npm run coverage |
Be sure your Postgresql server is running. The coverage results can be found in server/coverage/index.html |
Please see the package.json for the complete list.
With your local Node server running, browse to http://localhost:8080/docs/api in order to view the interactive Swagger API documentation. This documentation will allow interactive access to the API endpoints.
Public users must authenticate with login.gov, and Forest Service admins must authenticate with USDA eAuth. Both of these authentication techniques are handled by the Passport library for Node.js.
Login.gov uses the openid-client passport plugin for the OpenID Connect protocol, and USDA eAuth uses the passport-saml plugin for the SAML protocol.
Due to security restrictions testing can't be done locally, you must use a server on cloud.gov.
Note: we use cookie-sessions with keys bound to the environment to allow for running in a clustered environment.
Forest tree cutting start and end dates are saved in the database as a UTC DateTime object. When updating the start and end dates for the forest in the database, you must consider daylight savings, the forest timezone and calculate the offset.
Forest dates on the frontend assume the forest's timezone and do NOT specify the timezone.
For local development, go to an the changes season dates page to open a forest
VCAP service value for jwt token is used for appending a token to permit URL to view the purchased permit so that the users cannot easily guess the permit URL.
VCAP service values for Pay.Gov need to be updated for production deploy. {token_url} and {client_url} need to be supplied by Pay.Gov.
To mock Pay.Gov integration use the values in the VCAP example.
The current configuration implements email via google smtp relay. Follow the documentation at https://support.google.com/a/answer/2956491?hl=en to set up your google SMTP relay.
Authentication is set up to support safelisted IP addresses that are allowed to send emails, so no SMTP authentication is required.
The smtpserver value in your VCAP_SERVICES should be smtp-relay.gmail.com
We use winston logging and express-winston to modify logs to be viewed within cloud.gov.
cd frontend
Run npm install
Run npm start for a development server. Navigate to http://localhost:4200/. The app will automatically reload if you change any of the source files.
Run npm run dist-<env> where env is one of dev, trees (staging), prod to build the static files for the single paged app. The build artifacts that can be deployed will be stored in the dist/ directory.
Run npm test to execute the unit tests via Karma.
Add --code-coverage flag to print out code coverage statistics.
Run npm run e2e to execute the end-to-end tests via Protractor.
If you receive an error that Chromedriver requires a different version of Chrome than what is currently installed, most likely you have an older version of Chrome that is not compatible with the newest version of Chromedriver that will be installed by Protractor. You can:
Update your version of Chrome
OR
- Find the appropriate version of Chromedriver
- Navigate to
https://chromedriver.storage.googleapis.com/index.html?path=<version>/where<version>is replaced by the Chromedriver version you want - Click on the link for your operating system to download the
.zipfile - Extract the executable and copy it somewhere that makes sense to you, (ex:
~/chromedriver/) - Ensure that the environment variable
OPEN_FOREST_CHROME_DRIVERis set to the absolute path to the chromedriver executable above, either by exporting it in your profile or providing it when running the e2e tests. Ex.OPEN_FOREST_CHROME_DRIVER="/Users/<username>/Downloads/chromedriver" npm run e2e
Note: The output still shows that it downloads the latest chromedriver, but it will actually USE the one specified.
To run pa11y-ci with the single page angular app with pushstate enabled, you need to first build the static application, and then run the app from a server that supports pushstate. We are using superstatic as our server.
Run pa11y-ci via docker: cd docker; docker-compose up pa11y
Run pa11y-ci without docker: cd frontend; npm run build-test-pa11y
cd frontend
build typedoc npm run docs
typedoc are added to frontend/src/assets/typedoc and are accessible via url at /assets/typedoc/index.html
Navigate to /assets/typedoc/index.html
View services that are available to leverage in components.
-
To mock a pay.gov error when beginning the transaction, in the Christmas trees application form, set the first name to
unknown. -
To mock a pay.gov error when completing the transaction, use credit card number
000000000000XXXX.
-
A valid credit card number in staging environment is
4111111111111111. -
To mock a expired card pay.gov error when completing the transaction, use credit card number
4005550000000001.
View instructions to use the Christmas trees sidebar template.
The staging-deploy and the prod-deploy currently have a basic auth file generated from ENV vars in circle.
BASIC_AUTH_USER and BASIC_AUTH_PASS which is a hash of the password (generated by HTPassword) are copied into a frontend/src/Staticfile.auth which is used by the cloud.gov static buildpack.
This task should be removed prior to launch.
Circleci 2.0 is used for continuous integration/deployment. The configuration file for circleci are found at /.circleci/config.yml. An explaination of the checks performed by circleci are found at /docs/christmas-trees/process/circle-checks.md
The circleci configuration separates the tests into different jobs that run simultaneously to decrease build time e2e, pa11y tests, and all other tests.
Deployment to a staging server is configured to run on the dev branch only.
Check the .snyk file under frontend and server for packages ignored by snyk. This
file is managed by the snyk wizard. snyk-protect is run in the Procfile at server start-up.
Deployment to cloud.gov is configured in the .cg-deploy directory. The website's client (frontend) and server (backend) are deployed to separate servers. Each deployment environment (staging, production) require their own manifest files. The manifests are attached to the environment via the deploy script, that authenticates with cloud.gov and pushes the files.
We have a development environment configured to support ad hoc testing that cannot be done locally, for example the EAuth integration. This environment is connected to the same services as the staging environment except for the middlelayer, which is the dev environment (which is connected to staging SUDS, etc). Pushing to this environment must be done manually by making sure you are logged into Cloud.gov and:
cd frontendnpm run dist-devcd ../server./copy-frontend-assets.shcd ..cf t -o usda-forest-service -s platform-devcf zero-downtime-push open-forest-platform-frontend-dev -f .cg-deploy/manifests/dev/manifest-frontend-dev.ymlcf zero-downtime-push open-forest-platform-api-dev -f .cg-deploy/manifests/dev/manifest-api-dev.yml
The app/frontend/replace.build.js script is run from circle-ci via config.yml to update the version and date in the transpiled javascript frontend code for each build. This date and version will display in the application footer.
The database is used to store Christmas Trees permit applications for transaction purposes, and a minimal set of variables for each forest that are shared throughout the Christmas trees pages (e.g. on the permit svg and the guidelines.) The Christmas Trees Forests database contains the following fields:
View ChristmasTreesForest database structure
To update a forest’s informational content, you’ll need to find and modify markdown files specific to the forest. The content is structured so that each national forest has its own directory, which can be found under frontend/src/assets/content and then adding the forest_abbr to the url. (For example, frontend/src/assets/content/mthood.) Each forest’s directory contains several markdown files and folders that comprise the bulk of the content for each forest. (For example, /christmas-trees/forests/flathead.)
In the markdown files, database variables are indicated by curly brackets, such as {{treeHeight}}.
View a list of markdown files and their locations
JSON files for forest content are in /assets/config
Each forest has a JSON file that contains any data needed by the forest that does not come from the database or markdown, e.g., shared tree species information. The shared tree species are located in frontend/src/assets/content/common/species.
In order to enable pushstate for single page apps on cloud.gov using the static build pack, you must add a file called Staticfile to the root directory with a single line pushstate: enabled
This allows you to use urls like /some/path instead of /#/some/path
This application uses Winston library to format logs as JSON to the cloud.gov Kibana instance. The centralized logger is within the server/src/services/logger.es6 file. For route requests, the expressWinston library is used. For server controller actions the server/src/services/util.es6:logControllerAction should be used.
SMTP errors are logged in the console and prefixed with the string NODE_MAILER_SMTP_ERROR. A monitoring service, such as New Relic, can be configured to create alerts when an error with NODE_MAILER_SMTP_ERROR is logged.
While developing we spent time usability testing features with the correct users and applied feedback into the website.
The wiki includes usability testing summaries conducted by Flexion and NCI
The file frontend/src/sass/_focus-fix.scss implements a style fix in the upstream repository: https://github.com/18F/web-design-standards/pull/2112/files Eventually once these changes are released we can remove this file.
The server dependency is JSDOM is currently a fork to pass security vulnerability tests. This should be replaced with the original package once the security vulnerability is fixed.
Scaling to include more permit types: Architecture to support purchasing or applying for multiple products per transaction
The site’s architecture is not optimized to support users purchasing more than one product in a single transaction. It was built (1) with the understanding that tree cutter’s do not typically purchase permits for more than one forest and (2) for ease in modularly onboarding additional Forests, each with their own unique rules and guidelines, to sell Christmas tree permits. However, there is some evidence that users may want to purchase or apply for more than one of the permit types that the ePermits platform will eventually offer. Enabling users to purchase or apply for more than one permit type, including Christmas tree permits, online will require some rearchitecting.
Scaling to include more Forests: Form controls to help users choose from a greater number of options
The site was built to accommodate four pilot Forests. Scaling the application to include more Forests will require that a number of form controls be redesigned to support users in choosing their forest from greater than four options. Pages impacted:
*This page will need to support users’ selection from a greater number of options only if the FS Product Owner and leadership decide that Forest administrators should have access to other Forests in the application, in addition to their own.
The details of how these pages and form controls should be designed in order to support users selecting from a large number of Forests will require additional user research.
This project uses New Relic Monitoring for performance and uptime alerts. The application name and license keys are provided as environment variables that are accessed through the VCAP constants. This application uses the newrelic npm module.
See CONTRIBUTING for additional information.
This project is in the worldwide public domain. As stated in CONTRIBUTING:
This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.
All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.