We moved: https://github.com/mongodb/atlas-osb
Extensible Enterprise Production Deployments for MongoDB Atlas
- Intoduction
- Getting Started
- Specifications
- VMWare Tanzu Application Service
- Notes
- License
- Development
This project is a new version of the original MongoDB Atlas Serivce Broker called "atlas-osb"
Atlas-osb adds the ability to define your own plans from Atlas resource templates. This new and powerful feature brings the broker to a new level of production-readiness. One simple "create service" command is all you need to provision a complete Atlas deployment including a Project, Cluster, Database user, firewall access, and more.
Use the Atlas Service Broker to connect to MongoDB Atlas from any platform which supports the Open Service Broker API, such as Kubernetes and Pivotal Cloud Foundry.
- Provision managed MongoDB clusters on Atlas directly from your platform of choice. Includes support for all cluster configuration settings and cloud providers available on Atlas.
- Manage and scale clusters without leaving your platform.
- Create bindings to allow your applications access to clusters.
- Securely configure and deploy the broker with multiple Atlas apikeys integrated with systems such as Pivotal CredHub and Kubernetes Secrets.
atlas-osb custom plans allow cluster administrators to manage real-world production scenarios for the MongoDB Atlas Platform within any OSB-compliant environment. Custom plan's are templates of Atlas API resources.
Custom plans use the Atlas API resources and allow users to define reusable packages of related Atlas resources. These are known as "plan's". Cluster administrators can deploy these plans into Open Service Broker API marketplaces. The plans are then available through standard service broker catalog integration. This is available in Cloud Foundry, Kubernetes via Service Catalog, and other modern containerized computing environments.
Typical cli/bash commands are used for this procedure along with the following tools:
cfCloud Foundry clihammercli (recommended)- text editor
- MongoDB Atlas account - you need an Org apikey
Once you have your workstation ready, head over to http://cloud.mongodb.com and create an organization apikey. For more step by step details on this step, step over here: https://docs.atlas.mongodb.com/configure-api-access/#manage-programmatic-access-to-an-organization.
- Pull down the latest release of the atlas-osb (recommended).
curl -OL https://github.com/jasonmimick/atlas-osb/releases/download/v0.1-alpha/atlas-osb-v0.1-alpha.tar.gz
tar xvf atlas-osb-latest.tar.gz
cd atlas-osbor, you can clone this repo.
- Setup apikeys. This flow uses a simple User Provided Service to store the apikey for the broker. The broker also supports CredHub integration [TODO: add section].
Copy this template and update with your own apikey information.
{
"broker": {
"username": "admin",
"password": "admin",
},
"keys": {
"<KEY-NAME>": {
"publicKey": "<PUBLIC-KEY>",
"privateKey": "<PRIVATE-KEY>",
"orgID": "<ORG-ID>",
"id": "my-test-key",
"desc": "My first key for the atlas-osb.",
}
}
}Save this in a file called keys.
NOTE The broker.db field in the json above currently holds a connection string to another Atlas cluster we are using to store data on the mappings between serivce instance id's and the Atlas project, cluster, etc ids. The use of another external db will NOT be a requirement of the GA version of atlas-osb. This is only a temporary solution, to be addressed in Sprint 3.
- Create a user provided service:
# create simple user-provided-service with keys file
cf cups atlas-osb-keys -p keys- Installing the service broker app
The initial setup includes deploying the cf app and configuring some environment variables.
By default, the broker will only listen on localhost (127.0.0.1) and port 4000.
Use the set-env command to update this for Cloud Foundry defaults (port 8080).
# push the cf app for the broker
cf push atlas-osb --no-start
# connect the keys to the broker & configure settings
cf bind-service atlas-osb atlas-osb-keys
# use the provided sample plan templates
cf set-env atlas-osb ATLAS_BROKER_TEMPLATEDIR ./samples/plans
cf set-env atlas-osb BROKER_HOST 0.0.0.0
cf set-env atlas-osb BROKER_PORT 8080
# start it up
cf start atlas-osb
```bash
Check the logs, you should see output included the loaded plan templates.
```bash
cf logs atlas-osb --recent- Register the app as a real service broker
Grab the routes value from cf app atlas-osb and use this URL in the following command.
# create the actual broker
cf create-service-broker atlas-osb admin admin <YOUR-DEPLOYMENT-URL>
# enable access
cf enable-service-access mongodb-atlas-templateCheck out the results.
cf marketplace
cf apps- Create an Atlas cluster
For this flow, we'll create an instance of the "basic-plan" called "hello-atlas-osb". See /samples/plans/sample_basic.yml.tpl.
Use your own in the following command.
cf create-service mongodb-atlas-template basic-plan hello-atlas-osb -c '{"org_id":"<YOUR-ORG-ID>"}'Head over to http://cloud.mongodb.com to follow progress as your Atlas resources are provisioning.
- (Optional) You can test connecting to your new Atlas cluster with a very simple app in package.
Push, bind, and test sample app.
cd test/hello-atlas-cf
cf push hello-atlas-cf --no-start
cf bind-service hello-atlas-cf hello-atlas-osb
cf start hello-atlas-cf
cf app hello-atlas-cfGrab the route and load it up in your browser. You should see connection information to your new Atlas cluster.
You need to override the role when creating the binding like this:
cf bind-service hello-atlas-cf wed-demo-1 -c '{"User" : {"roles" : [ { "roleName" : "readWrite", "databaseName" : "mydatabase"} ] } }'You can check this works with the hello-atlas-cf app with something like this:
cf app hello-atlas-cf | grep routes | cut -d: -f2 | xargs -I {} google-chrome "{}" cf update-service <SERVICE-INSTANCE-NAME> -c '{ "paused":true }'Note - do not put quotes around the true/value.
First - note not all possible updates are supported at this time.
In general, to update a plan instance:
cf create-service -b dyno-ss mongodb-atlas-template basic-plan dyno-ss-4oh -c '{"org_id":"5ea0477597999053a5f9cbec", "cluster" : { "mongoDBMajorVersion": "4.0" } }'
Creating service instance dyno-ss-4oh in org atlas-broker-demo / space dynoss as admin...
OK
The syntax for the -c json you send in create-service or update-service is processed in two ways:
- The document passed is parsed and matched into the template dot-variables, then the template is executed.
- The passed document treated as a partial plan-instance and then merged into the results from step 1.
This allows service settings to be updated.
Configuration is handled with environment variables. Logs are written to
stdout/stderr as appropriate and each line is in a structured JSON format.
| Variable | Default | Description |
|---|---|---|
| ATLAS_BASE_URL | https://cloud.mongodb.com |
Base URL used for Atlas API connections |
| BROKER_HOST | 127.0.0.1 |
Address which the broker server listens on |
| BROKER_PORT | 4000 |
Port which the broker server listens on |
| BROKER_LOG_LEVEL | INFO |
Accepted values: DEBUG, INFO, WARN, ERROR |
| BROKER_TLS_CERT_FILE | Path to a certificate file to use for TLS. Leave empty to disable TLS. | |
| BROKER_TLS_KEY_FILE | Path to private key file to use for TLS. Leave empty to disable TLS. | |
| PROVIDERS_WHITELIST_FILE | Path to a JSON file containing limitations for providers and their plans. | |
| BROKER_APIKEYS | Path to file or JSON string containing credentials. | |
| ATLAS_BROKER_TEMPLATEDIR | Path to folder containing plans e.g. ./samples/plans |
The values for the OSB "Service" for a given atlas-osb instance can be customized with a set of
additional environement variables. Each of these are optional, and has default content.
Here are the settings for the domain.Service returned for each plan:
domain.Service:
| Field | Env variable | Default |
|---|---|---|
ID |
NONE | "<id>-template-service" |
Name |
BROKER_OSB_SERVICE_NAME |
"atlas" |
Description |
BROKER_OSB_SERVICE_DESC |
"MonogoDB Atlas Plan Template Deployments" |
Metadata.DisplayName |
BROKER_OSB_SERVICE_DISPLAY_NAME |
"MongoDB Atlas - %s" with "Template Services" |
Metadata.ImageUrl |
BROKER_OSB_IMAGE_URL |
"https://webassets.mongodb.com/_com_assets/cms/vectors-anchor-circle-mydmar539a.svg" |
DocumentationUrl |
BROKER_OSB_DOCS_URL |
"https://support.mongodb.com/welcome" |
ProviderDisplayName |
BROKER_OSB_PROVIDER_DISPLAY_NAME |
"MongoDB" |
LongDescription |
"Complete MongoDB Atlas deployments managed through resource templates. See https://github.com/jasonmimick/atlas-osb" |
These are the requirements for supporting multiple apikeys. Note: Sometimes we use the term "CredHub" to refer to this overall feature to support multiple keys.
- The broker will accept a json/yaml object containing authentication credentials. The format for this will be:
bindingName: 'Main-Creds'
broker:
username: 'admin'
password: 'admin'
projects:
- id: 'first-key'
desc: 'the first key'
publicKey: '12345'
privatekey: '12345'
roles:
- groupId: '12345' # TODO update to support `projectId` here
- id: 'the 2nd key'
publicKey: '12345'
privateKey: '12345'
roles:
- groupId: '545454' # TODO update to support `projectId` here
orgs:
- publicKey: '12345'
privateKey: '12345'
id: 'the org 1 key'
roles:
- orgId: '3030303030'Where each key conforms to the following schema defined by the ApiKey struct.
Multi-apikey support requirements (P# where # is priority with 0 highest.)
-
P0 Reading keys as a string from the
BROKER_APIKEYSenvironment variable. -
P0 Reading path from the
BROKER_APIKEYSenvironment variable and then loading from file. -
P0 Reading from Cloud Foundry
VCAP_SERVICESto support CredHub integration. -
P1 Support reading from Kubernetes Secrets.
-
The broker will allow selection of appropriate apikey during plan provisioning by discovery of the
ApiKeyin the plan-template instance context (see below Plans).
Providing support for all the myriad of combinations of Atlas resources and arbitrary validation logic is not possible with the current broker "service"/”plan” design. atlas-osb introduces an additional way to define new “services” and “plans” which represent arbitrary Atlas resource objects (object graphs) by leveraging the declarative design of the Atlas API and JSON templates.
Most users and most apps need the following resources at minimum for typical usage:
- 1 Atlas Project to contain everything
- 1 Standard 3-node replica set for your database
- 1 IP-Whitelist so that your app can connect
- 1 DB credential, again, so your app can connect
We can model the above with the following set of templates:
name: basic-plan
description: This is the `Basic Plan` template for 1 project, 1 cluster, 1 dbuser, and 1 secure connection.
free: true
apiKey: {{ mustToJson (index .credentials.Orgs (default "" .org_id)) }}
project:
name: {{ .instance_name }}
desc: Created from a template
orgId: {{ .org_id }}
cluster:
name: {{ .instance_name }}
providerBackupEnabled: {{ default "true" .backups }}
providerSettings:
providerName: {{ default "AWS" .provider }}
instanceSizeName: {{ default "M20" .instance_size }}
regionName: {{ default "US_EAST_1" .region }}
labels:
- key: Infrastructure Tool
value: MongoDB Atlas Service Broker
databaseUsers:
- username: {{ default "test-user" .username }}
password: {{ default "test-password" .password }}
databaseName: {{ default "admin" .auth_db }}
roles:
- roleName: {{ default "readWrite" .role }}
databaseName: {{ default "default" .role_db }}
ipWhitelists:
- ipAddress: "0.0.0.0/1"
comment: "everything"-
P0 Support loading Plan templates from json or yaml files mounted into the broker runtime at deployment-time.
-
The broker should support reading plan templates directly from Linux environment variables.
- P0 Support a default directory to load templates from
ATLAS_BROKER_TEMPLATEDIR
ATLAS_BROKER_TEMPLATEDIR = "/templates" UPS - user provided service to mount files into /templates K8s - mount configmaps as files, etc..., Docker, etc..
- P0 Support a default directory to load templates from
-
Include loaded templates with INFO level and also full json payload logging to broker logs (to help debugging).
-
Always reload templates on startup (restart to deploy new templates)
-
Template should be standard go-templates and support typical values/variable style replacements.
-
Users should be able to specify template parameters during service provisioning or update.
-
Provide a set of common resource yaml/json samples and templates.
-
Allow reading apikeys and group/org ids from the "CredHub" multi-apikey support
- Make a template variable called
Credentialsavailable which contains all the apikeys
- Make a template variable called
-
Allow reading apikey from a yaml/json file with a resource definition.
-
🚧 Support a
--dry-runflag whenever processing a provision, update, or delete operation on a custom plan. Default isfalse. Include pre & post template processing in logging output. (Not supported yet.)
Each Plan instance is managed through the OSB provision, bind, unbind, and deprovision operations. In this section we describe the relationship between a Plan's OSB operations and how the broker translates them to Atlas Client API calls.
Each plan will have at least one apikey associated with it.
🚧
When the broker gets a call to provision a plan, it will iterate through the various Atlas resources in the plan can call the corresponding service Create method. Similarily, when deprovisioning, the broker will delegate calls to the Atlas Go-client corresponding service Delete function.
Plans are loaded at startup and first validated before being made available in the Marketplace.
- read plans from disk
- do a first-pass template parse with empty context (substitute "" for everything in template)
- Allow for default values?
- Provide a way to test the plan actually works (dry-run)
- parse the result into yaml, get static metadata like plan name, description, instance size, whatever
- on provision, do a parse with full context - this is the final plan spec
- 🚧 Allow for dry-run at this step too.
This section describes how the state of plan definitions and service instance metadata will be stored.
atlas-osb introduces the use of MongoDB Realm applications to manage the state of your deployed service instances. This happens automatically, directly through the Atlas and Realm APIs.
The atlas-osb will create special project called "Atlas Service Broker Maintenance" for each Atlas Organization. Here, the broker will create a Realm App called "Broker State" :constuction: and use Realm Values to store reference data for the serivce instances you have deployed across your environments. This approach has several advantages:
- zero-footprint on OSB-marketplace environment
- single location for all serivce instance metadata
- follows the Atlas organization boundaries, allows support for multi-tenancy scenarios
- automaticalluy encrypted via Realm
The OSB bind function is used to provision a new database user credential and connection information for an application using MongoDB. This usually happens when an app is deployed into a new environment. To support this, the broker will create new Atlas resources for the binding and return the connection information appropriately.
🚧 broker needs refactor to use Plan user/ipwhitelist.
When the broker creates a binding, it will translate the Connection Details for the given cluster into the OSB Binding structure.
Connection Details types.
The format for the JSON available for binding in VCAP_SERVICES is:
{
"connectionString": "mongodb+srv://uuuuuuuuu:xxxxxxxx@chewy-123.6bikq.mongodb.net/admin",
"password": "xxxxxxxxxx",
"uri": "mongodb+srv://chewy-123.6bikq.mongodb.net",
"username": "uuuuuuuuuu",
"datbase": "admin"
}Please see the test/hello-atlas-cf sample app to see details on the binding information available to apps.
Certain customers may wish to control the exact name of the database to which apps using Atlas services can use. This is controlled by inserting the database name into the connection string (as the last forward-slash piece before the query string) which is constructed during a call to the brokers Bind function.
In general, we do not recommend using this feature. However it will be released as "Deprecated" in the Beta release for the broker. Customers are encouraged to refactor their dependencies on fixing this database name.
For example, to set the default role for any database users created via the bind() OSB operation, add these to the settings for your plan.
name: override-bind-db-plan
description: This is an extension of the `Basic Plan` template for 1 project, 1 cluster, 1 dbuser, and 1 secure connection. But it added the ability to override the bind db.
free: true
apiKey: {{ mustToJson (index .credentials.Orgs (default "" .org_id)) }}
settings:
overrideBindDB: "products"
overrideBindDBRole: "readWrite"
project:
name: {{ .instance_name }}
desc: Created from a template
...This sets gives readWrite on the products database for each new user created via bind().
The following types are supported for loading from multiple or a single yaml or json objects.
Note these types are derived from the following specifications:
Each "plan" is loaded from a single json/yaml object describing the resources managed by the plan. Plans can contain any number of various MongoDB Atlas resources, but require an Atlas Project. Since each Atlas resource is always associated with an Atlas Project, each Plan must have a project. The other resources are optional.
The Plan and Binding types are new structures to implement this new feature, while the other types are used directly from the MongoDB Atlas Go client.
The officially supported Atlas Resources are:
This represents an Open Service Broker plan (NOTE: https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#service-offering-object)
// Plan represents a set of MongoDB Atlas resources
type Plan struct {
Version string `json:"version,omitempty"`
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
ApiKey *mongodbatlas.ApiKey `json:"apiKey,omitempty"`
Project *Project `json:"project,omitempty"`
Clusters []*mongodbatlas.Cluster `json:"clusters,omitempty"`
DatabaseUsers []*mongodbatlas.DatabaseUser `json:"databaseUsers,omitempty" yaml:"databaseUsers,omitempty"`
IPWhitelists []*mongodbatlas.ProjectIPWhitelist `json:"ipWhitelists,omitempty" yaml:"ipWhitelists,omitempty"`
Bindings []*Bindings // READ ONLY! Populated by bind()
Settings map[string]string `json:"settings,omitempty"`
}The remaining resource type definitions are taken directly from the Atlas Go Client, and therefore subject to change per that project.
An Atlas project (NOTE: https://docs.atlas.mongodb.com/reference/api/projects/). Requires an organization id or valid context from apikey.
Project type.
An Atlas cluster (NOTE: https://docs.atlas.mongodb.com/reference/api/clusters-create-one/#example-request).
TODO: MARK WHICH FIELDS ARE READ-ONLY? ie. users need to understand what can be in the template
The following badges provide version and version-support information about Atlas-OSB for VMware Tanzu.
-
Deploying Atlas-OSB to CF. There are several ways to deploy atlas-osb to cloud foundry:
- with docker image
cf push APP_NAME --docker-image [REGISTRY_HOST:PORT/]IMAGE[:TAG] [--docker-username USERNAME] [-c COMMAND] [-f MANIFEST_PATH | --no-manifest] [--no-start] [--no-wait] [-i NUM_INSTANCES]- with manifest (to specify a manifest use
-fwith the path to a manifest)
-
Please note that Atlas-OSB does not support the free tier of cluster creation: M0 (Atlas API doesn't have such support)
See LICENSE. Licenses for all third-party dependencies are included in notices.
We expect to launch our beta on or before August 25, 2020 and follow up with general availability shortly thereafter.
Support for the MongoDB Atlas-OSB is provided under MongoDB Atlas support plans. Please submit support questions within the Atlas UI. Support questions submitted under the Issues section of this repo will be handled on a "best effort" basis.
Bugs should be filed under the Issues section of this repo.
Feature requests can be submitted at https://feedback.mongodb.com/forums/924145-atlas?category_id=370720 - just select "Atlas Service Broker" as the category or vote for an already suggested feature.