New Main Contributor/Maintainer Needed

With the advent of the IBM announcement of IBM® Cloud Foundry being deprecated, this repository needs a new main contributor/maintainer. IBM announcement here: https://cloud.ibm.com/docs/cloud-foundry-public?topic=cloud-foundry-public-deprecation

Administration Web UI for Cloud Foundry NG

The Administration Web UI provides metrics and operations data for Cloud Foundry NG. It gathers data from the varz providers and doppler_logging_endpoint for the various Cloud Foundry components as well as from the Cloud Controller and UAA REST APIs.

See the Using the Administration UI section for more information on using it and for sample screen shots.

cf-release 245

In order to run the Administration UI on a release prior to cf-release v245, use the cf-release-pre-v245 branch.

CF releases for v245 and later continue to use the master branch.

This is due to the CCDB changes for applications in cf-release v245.

Placement

In order to execute, the Administration UI needs to be able to access the following resources:

• NATS
• Cloud Controller REST API
• Cloud Controller DB URI
• UAA REST API
• UAA DB URI
• doppler logging endpoint

Installation of the Administration UI and its prerequisites requires access to the Internet to access github.com, RubyGems.org, Ubuntu software repositories, etc.

Installation Steps

Ubuntu 18.04.5 64 bit

The last version of Ubuntu this has been tested upon is Ubuntu 18.04.5 64 bit.

Ubuntu Prerequisite Libraries

# Update your package listing
sudo apt-get update

# Install prerequisite libraries
sudo apt-get install -f -y --no-install-recommends git-core build-essential libreadline-dev libssl-dev libsqlite3-dev openssl libpq-dev libmysqlclient-dev

Ruby

Ruby is required to run the Administration UI. The last version used for development and testing is Ruby 3.1.2.

Here is a sample installation of ruby using rbenv:

git clone https://github.com/sstephenson/rbenv.git ~/.rbenv
git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.profile
echo 'eval "$(rbenv init -)"' >> ~/.profile
. ~/.profile
rbenv install 3.1.2
rbenv global 3.1.2

Retrieve the Administration UI code

git clone https://github.com/cloudfoundry-incubator/admin-ui.git

Install Administration UI

cd admin-ui
bundle install

Setup Administration UI group and user using uaac.

The following steps are the same if you are using the default bosh-lite install of Cloud Foundry or AWS install using BOSH AWS Bootstrap.

# Target your bosh-lite UAA and get the 'admin' token
uaac target --skip-ssl-validation https://uaa.bosh-lite.com
uaac token client get admin -s admin-secret

# Add 'scim.write' if not already there and re-get token
uaac client update admin --authorities "`uaac client get admin | \
    awk '/:/{e=0}/authorities:/{e=1;if(e==1){$1="";print}}'` scim.write"
uaac token client get admin -s admin-secret

# Create a new group and add the 'admin' user to it
uaac group add admin_ui.admin
uaac member add admin_ui.admin admin

# Create the new UAA admin_ui_client
uaac client add admin_ui_client \
 --authorities clients.write,cloud_controller.admin,cloud_controller.read,cloud_controller.write,doppler.firehose,openid,scim.read,scim.write,sps.write,uaa.admin,uaa.resource \
 --authorized_grant_types authorization_code,client_credentials,refresh_token \
 --autoapprove true \
 --redirect_uri http://<admin ui host>:8070/login \
 --scope admin_ui.admin,admin_ui.user,openid \
 -s admin_ui_secret

Note: If you are using the default bosh-lite install, then running the above commands and setting the bosh-lite-specific configuration values in config/default.yml (ccdb_uri password, cloud_controller_ssl_verify_none, mbus password and uaadb_uri password) should enable you to use the default configuration values for the Administration UI and you can skip down to the Using the Administration UI section. If you have installed Cloud Foundry on AWS using the AWS Bosh Bootstrap, then you will have to modify the Administration UI before moving forward.

Administration UI Configuration

Default configuration found in config/default.yml

Values that must be changed for your environment are marked in bold.

bind_address

The network address on which the server listens for web requests.
Example: 0.0.0.0

ccdb_uri

The URI used to connect to the Cloud Controller database for retrieval.
URI format: <protocol>://<db-user>:<db-user-password>@<host>:<port>/<db-name>
Example: postgres://ccadmin:admin@10.244.0.30:5524/ccdb
Example: mysql2://ccadmin:admin@10.244.0.30:3306/ccdb
If you have MySQL database use mysql2 protocol instead of mysql. Only PostgreSQL and MySQL have been tested to work with the Administration UI.
You can try other databases by adding relevant database gems to the Gemfile and executing bundle update.

cloud_controller_discovery_interval

Seconds between cloud controller REST API discoveries.
Example: 300

cloud_controller_ssl_verify_none

If connection to cloud_controller is https, true to ignore SSL verification.
Example: true
Example: false

cloud_controller_uri

The URI used to connect to the Cloud Controller REST API. This is also used as a title for the web UI header as well as email notification.

component_connection_retries

The number of times to try to talk to a varz component before considered failing.
Example: 2

cookie_key

The key used for the session cookie.
Example: mykey

cookie_secret

The secret used for the session cookie.
Example: mysecret

cookie_secure

Whether the session cookie is secure or not.
Example: true

data_file

Relative path location to store the Administration UI data file.
Example: data/data.json

db_uri

The URI used to connect to a sqlite database instance.

If the database instance does not exist, Admin-UI will automatically create an instance.
Example: sqlite://data/store.db
In this case, the store.db file is located in the 'data' directory. Absolute path is allowed in the uri. For example, sqlite:///tmp/store.db indicates the database file 'store.db' is located in the '/tmp' directory.

display_encrypted_values

Whether to display encrypted values like application environment variables, service instance credentials, service binding credentials, service key credentials, etc. Encrypted values are also only displayed if the signed-in user is a member of a UAA admin group for the Administration UI.
Example: true
Example: false

doppler_data_file

Relative path location to store the Administration UI doppler data file.
Example: data/doppler_data.json

doppler_logging_endpoint_override

This optional URI is used to override the doppler_logging_endpoint retrieved from /v2/info. This can be used to offload filtering of the standard dopper_logging_endpoint to allow only those metrics consumed by the Administration UI, namely container metrics and value metrics. The doppler_logging_endpoint_override is required to support the same protocol as the base /v2/info doppler_logging_endpoint.

doppler_reconnect_delay

The number of seconds to wait until a reconnect to the doppler_logging_endpoint is attempted after a prior failure.
Example: 300

doppler_rollup_interval

Seconds between rolling up doppler_logging_endpoint data to be made available.
Example: 30

doppler_ssl_verify_none

True to ignore SSL verification to doppler_logging_endpoint.
Example: true
Example: false

event_days

Maximum number of days of events in the past to include within the Events tab.
Example: 7

http_debug

Whether http invocations will provide records in the log_file or not. The default value is intentionally false as this is considered a security exposure. Records within the log_file will be of log type ANY.
Example: true
Example: false

log_file

Relative path location to the Administration UI log file.
Example: admin_ui.log

log_file_page_size

Size of each log file page in bytes.
Example: 51200

log_file_sftp_keys

Key files in a comma-delimited array to use to access logs using SFTP.
Example: [/some_directory/some_key.pem]

log_files

Log files in a comma-delimited array being exposed through the Administration UI. Note that these files must be accessible by the user that started the Administration UI. These files can either be found on a file system accessible by the local system or as an SFTP URI. In the case of SFTP, both user:password and user with pem files are supported. If the SFTP password is not specified, the key files specified in log_file_sftp_keys will be used.
Example ['/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log']
Example ['/var/vcap/sys/log/cloud_controller_ng/*.log']
Example ['/var/vcap/sys/log/**/*.log']
Example ['sftp://someuser:somepassword@10.10.10.10/path/file.log']
Example ['sftp://someuser@10.10.10.10/path/*.log']
Example ['sftp://someuser:somepassword@10.10.10.10/path/**/*.log']

mbus

URL to the NATS.
Example: nats://nats:c1oudc0w@10.10.10.10:4222

monitored_components

Components in a comma-delimited array which when down will result in notification.
Example of multiple components: [NATS, CloudController, DEA, HealthManager, Router]
Example of a wild card: [-Provisioner]
Example for all components: [ALL]

nats_discovery_interval

Seconds between NATS discoveries.
Example: 30

nats_discovery_timeout

The number of seconds to wait for the NATS to respond to vcap.component.discover.
Example: 10

nats_tls

A set of optional configuration properties for Admin-UI to securely communicate with NATS using TLS. These keys and values correspond to those documented for TLS secure client connection within the NATS repository.

ca_file

File path string referencing the NATS client certificate authority file. This is required if verify_peer has a value of true.
Example: ./ca.pem

cert_chain_file

File path string referencing the NATS client certificate file.
Example: ./cert.pem

private_key_file

File path string referencing the NATS client private key file.
Example: ./key.pem

verify_peer

True to verify peer for the NATS client. If true, ca_file configuration is required.
Example: true

port

Port for the Administration UI web server. If the environment variable PORT is set, that value will be respected. This allows the Administration UI to run as a Cloud Foundry application.
Example: 8070

receiver_emails

The receiving email(s) in a comma-delimited array.
Example: [ ]
Example: [bar@10.10.10.10, baz@10.10.10.10]

secured_client_connection

A true/false indicator about whether the Admin-UI server process will operate in secure or unprotected mode. In the secure mode, Admin-UI uses SSL security mechanism to protect communication with its users. As such, it requires its users to connect via https; in unprotected mode, Admin-UI expects http requests.
When set to 'true', a SSL certificate is required and the 'ssl' property must be present in the configuration file. Please see configuration property 'ssl' for details about how to configure Admin-UI to work with SSL certificate.
By default, Admin-UI runs in the unprotected mode.
Example: true

sender_email

Email server and account used when sending an email notifying receivers of down components.

server

The email server.
Example: 10.10.10.10

port

Port of the email server.
Example: 25

domain

HELO domain provided by the client to the server, the SMTP server will judge whether it should send or reject the SMTP session by inspecting the HELO domain.
Example: localhost

account

The email account.
Example: system@10.10.10.10

secret

Secret for the email account.

authtype

SMTP authentication scheme, can be one of: plain, login, cram_md5.
Example: login

ssl

A set of configuration properties for Admin-UI to work with SSL certificate. It is required when the 'secured_client_connection' is set to true.
Certificate can be self-signed or signed by Certificate Authority (CA). Admin-UI supports these certificates. However, intermediate certificate from CA is not yet supported.
Generate Self-signed certificate
You can generate a self-signed certificate on a server, all without involving third-party CA. The following steps illustrate a way in how you can generate a self-signed certificate: * generate private key

openssl genrsa -des3 -out /tmp/admin_ui_server.key -3 -passout pass:private_key_pass_phrase 2048

This command creates a private key at 2048 bit length which is then encrypted with passphrase 'pass:private_key_pass_phrase' by way of DES3. * Generate certificate request

openssl req -new -key /tmp/admin_ui_server.key -out /tmp/admin_ui_server.csr -passin pass:private_key_pass_phrase

This command yields a certificate request file by supplying the private key and passphrase involved in the previous step. * Generate certificate

openssl x509 -req -days 365 -passin pass:private_key_pass -in /tmp/admin_ui_server.csr -signkey /tmp/admin_ui_server.key -out /tmp/admin_ui_server.crt

This command generates a certificate that is good for the next 365 days. At this point, you no longer need to keep the certificate request (dot csr file).

certificate_file_path

File path string referencing the certificate file.
Example: certificate/server.cert

max_session_idle_length

Time duration in seconds in which an https session is allowed to stay idle.
Example: 1800 for 30 minutes.

private_key_file_path

File path string referencing the private key file.
Example: system@10.10.10.10

private_key_pass_phrase

Password string that is used to encrypt the private key.
Example: my_secret

stats_file

Deprecated. Relative path location to store the Administration UI statistics.

Admin-UI no longer stores its stats data in the stats file. Instead, the data is now stored in a database. This property is required only for the purpose of data migration. When data migration is no longer needed, you can remove this property from the configuration file. Please see the Data Migration section for details.
Example: data/stats.json

stats_refresh_time

Deprecated. See stats_refresh_schedules for details.
A daily schedule which starts at specified minutes from midnight for automatic stats collection
Example: 300. This results in stats collection starting at 5 AM.

stats_refresh_schedules

Schedules of automatic stats collection expressed in an array of strings. Each string represents a schedule and follows a syntax very similar to crontab. It consists of five fields, for specifying time, date, days of a week and etc, as follow.

       *        *        *        *        * 
       -        -        -        -        - 
       |        |        |        |        | 
       |        |        |        |        +----- day of week (0 - 6)(Sunday=0) 
       |        |        |        +------- month (1 - 12) 
       |        |        +--------- day of month (1 - 31)
       |        +----------- hour (0 - 23)
       +------------- minute (0 - 59)
   where * denotes an expression using legal values shown inside the parenthesis for the column. 

• Fields are separated by spaces.

• Fields can be expressed by a wild card * symbol which means every occurrence of the fields.

Example: ['0 * * * *'] means the collection starts once every hour at the beginning of the hour.

• Field value can be expressed in form of a range, which consists of two legal values connected by a hyphen (-).

Example: ['0 0 * * 1-5'] means the collection starts at midnight 12:00AM, Monday to Friday.

• Field value can also be a sequence of legal values separated by comma. Sequence doesn't need to be monotonic.

Example: ['0 1,11,12,13 * * *'] means the collection process starts at 1:00AM, 11:00AM, 12:00PM and 1:00PM every day.

• Mixed uses of sequence and ranges are permitted.
  Example: The example above can expressed this way as well: ['0 1,11-13 * * *']

• Step based repeat pattern like /4 is currently not supported.

• stats_refresh_schedules supports multiple schedules.

Example: [ '0 1 * * *', '0 12-13 * * 1-5' ] means the collection starts at 1:00AM everyday; on Monday to Friday, the collection process will also start at 12:00PM and 1:00PM.

This property supports the following predefined schedules

Predefined Schedule                        Description
-----------------------------------------------------------------------------
 ['@hourly']                      runs at the beginning of every hour
 ['@daily']                       runs at the 12:00AM everyday
 ['@midnight']                    runs at the 12:00AM everyday
 ['@weekly']                      runs at the 12:00AM every Sunday
 ['@monthly']                     runs at the 12:00AM on first day of the month
 ['@yearly']                      runs at the 12:00AM on every Jan 1st
 ['@annually']                    runs at the 12:00AM on every Jan 1st. It is the same as @yearly.

• When stats_refresh_schedules and stats_refresh_time are both present in the default.yml file, admin-ui will error out with an error message which reads Two mutually exclusive properties, stats_refresh_time and stats_refresh_schedules, are present in the configuration file. Please remove one of the two properties.
  
• When neither stats_refresh_schedules supports nor stats_refresh_time is present in the default.yml file, admin-ui disables stats collection.
  
• The default value of stats_refresh_schedules in default.yml file is stats_refresh_schedules: ['0 5 * * *']. This value translates to a schedule that starts daily at 5:00AM.
  

stats_retries

Number of stats retries.
Example: 5

stats_retry_interval

Seconds between stats collection saving.
Example: 300

table_height

Maximum height of the data table, will be set to 287px if not configured. If the records on the page are too large to fit in this height then a vertical scroll bar will appear in the table. Can be set to either a specific pixel size or a percentage. Using 100% will ensure that all data will be displayed without a scroll bar in the table.
Example: 100%

table_page_size

Default selection for the page size in the table, will be set to 10 if not configured. Valid values are: 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000.
Example: 25

uaa_client

UAA client to access the Cloud Controller REST API as an admin user and also support Single Sign On (SSO) for UI login.

id

Client ID.
Example: admin_ui_client

secret

Password for UAA client.
Example: admin_ui_secret

The UAA client used by the Administration UI must be added to the system via the cf-uaac command-line tool, BOSH deployment or equivalent and is required to consist of the following:

• authorities: clients.write,cloud_controller.admin,cloud_controller.read,cloud_controller.write,doppler.firehose,openid,scim.read,scim.write,sps.write,uaa.admin,uaa.resource
• authorized_grant_types: authorization_code,client_credentials,refresh_token
• autoapprove: true
• redirect_uri: http://<admin ui host>:8070/login
• scopes: openid as well as those defined in the uaa_groups_admin and uaa_groups_user configuration values.

Example:

uaac client add admin_ui_client \
     --authorities clients.write,cloud_controller.admin,cloud_controller.read,cloud_controller.write,doppler.firehose,openid,scim.read,scim.write,sps.write,uaa.admin,uaa.resource \
     --authorized_grant_types authorization_code,client_credentials,refresh_token \
     --autoapprove true \
     --redirect_uri http://<admin ui host>:8070/login \
     --scope admin_ui.admin,admin_ui.user,openid \
     -s admin_ui_secret

uaa_groups_admin

UAA scope(s) in a comma-delimited array to identify qualifying scopes for admin access to the Administration UI. If a user has any one scope, that user will be considered authorized to access the Administration UI as an admin.
Example: [admin_ui.admin, cloud_controller.admin]
The previously-unregistered groups referenced by this configuration property must be added to the system via the cf-uaac CLI command or equivalent.
Example: uaac group add admin_ui.admin
The group membership then needs to be updated for each user that has admin capabilities on the Administration UI.
Example: uaac member add admin_ui.admin your_user_name

uaa_groups_user

UAA scope(s) in a comma-delimited array to identify qualifying scopes for user access to the Administration UI. If a user has any one scope, that user will be considered authorized to access the Administration UI as a user.
Example: [admin_ui.user]
The previously-unregistered groups referenced by this configuration property must be added to the system via the cf-uaac CLI command or equivalent.
Example: uaac group add admin_ui.user
The group membership then needs to be updated for each user that has user capabilities on the Administration UI.
Example: uaac member add admin_ui.user your_user_name

uaadb_uri

The URI used to connect to the UAA database for retrieval.
URI format: <protocol>://<db-user>:<db-user-password>@<host>:<port>/<db-name>
Example: postgres://uaaadmin:admin@10.244.0.30:5524/uaadb
Example: mysql2://uaaadmin:admin@10.244.0.30:3306/uaadb
If you have MySQL database use mysql2 protocol instead of mysql. Only PostgreSQL and MySQL have been tested to work with the Administration UI.
You can try other databases by adding relevant database gems to the Gemfile and executing bundle update.

varz_discovery_interval

Seconds between VARZ discoveries.
Example: 30

Data Migration

Prior releases of Admin-UI store stats data in a file as indicated by the stats_file configuration property. This data is now stored in a database. Data migration is referring to the transfer of stats information from file to database.

Data migration takes place automatically at the start of admin-UI daemon process when the following conditions are all met:

• stats_file property is present and valid in the default.yml file
• db_uri property is present and valid in the default.yml file
• stats file contains data
• the database instance either does not exist or has not yet been initialized with schema.

When Admin-UI completes the data migration to database, it will rename the original stats file by appending '.bak' file extension. For example, 'stats.json' becomes 'stats.json.bak'.

Data migration is run only once on a given database instance. If for some reason you wish to rerun data migration, you must operate on a different database instance.

Database schema Migration

At the start of its daemon process, Admin-UI always checks its database schema migration directory and attempts to bring its database schema up to date. So it's not required to run schema migration manually. This migration takes place before data migration.

The database schema migration directory is located at 'db/migrations'. This directory contains files responsible for both populating database schema and subsequently migrating the schema. Migration files are built on the Sequel migration framework, and hence adhere to its file naming convention. i.e.

       <timestamp>_<title>.rb

The schema migration Admin-UI initiated always follows the chronological order as indicated by time stamps embedded in the migration file names.

More details about Sequel migration framework and ways to perform manual schema migration can be found at the following URLs:

• http://sequel.jeremyevans.net/rdoc/classes/Sequel/Migration.html
• http://sequel.jeremyevans.net/rdoc/classes/Sequel/Migrator.html

Execute Administration UI

You can provide an option to reference the configuration file when you execute the administration UI or you can let it default to config/default.yml

ruby bin/admin [-c <configuration file>]

Simple Health Check

You can remotely check if the Administration UI is running by accessing the /health path and checking for a status code of 200. This path does not require authorization. Currently no response body is returned.

http://<admin ui host>:8070/health

Using the Administration UI

To access the Administration UI, go to:

http://<admin ui host>:8070

You will be prompted for the credentials. Once there, by default, you will be taken to the DEA tab:

From there you will see the list of DEAs running in the environment, along with some basic statistics. Selecting one from the list will bring up another table below the DEA table showing even more details about the DEA you selected:

One important thing to note is that some of the items in the secondary table are hyperlinks. Clicking on that link will take you to the appropriate tab with a query already filled in, allowing you to see just the data related to what you clicked on. For example, in the table above if you clicked on the Apps link, meaning the 2, you'll be taken to the Apps tab and the query will be filled in such that you will only see the apps running on this DEA, as shown here:

Notice the Search entry field is pre-populated with a string and the table is filtered to show just those rows that contain that string in any column.

Also, note that each row in the table has a checkbox. While not all tables will have those, by selecting a set of rows an action can be performed on them. For example, in this case, by selecting one or more apps you can then use the buttons on the right side of the main table:

to start, stop, restart, etc. those apps.

All of the tabs will follow the same interaction pattern as described above.

There are however a few other tabs that worth calling out.

The Logs tab will display the contents of the log files that the Administration UI has access to - these need to be local to the application:

On this tab, once a particular log file is selected, you can examine its contents in the text area. Use the buttons to iterate through the file one page at a time, or use horizontal scroll bar at the top of the text area to quickly move to one section of the file.

The Stats tab:

can be used to view basic history data about the environment. Normally, a snapshot of the statistics are taken once a day, but you can force a new set of data points to be taken by using the Create Stats button.