This repository contains the back-end APIs for the Agave Platform. These APIs are not meant to run in isolation and, in and of themselves, are not sufficient to deliver the intended Science-as-a-Service functionality. For a full deployment of the Agave Platform, please see the Agave Platform repository.
The Agave Science APIs are a set of REST APIs that provide the core Science-as-a-Service functionality to the Agave Platform. Each API runs independently as an separate process. The APIs themselves do not handle authentication. Rather, they look for a signed Java Web Token (JWT) in each request header from which they obtain the identity, scope, authorization information, and other contextual information about the caller. The JWT are generated by the Agave OAuth2 server, which is part of the Agave Deployer project.
For ease of deployment, and easier scaling from demo to heavy utilization scenarios, we build and deploy the entire Agave Platform as a collection of Docker images. This README will walk you through the process of running a development instance of Agave for yourself. For more information about the use cases satisfied by the APIs as well as functional documentation about their design, use, and performance, please see the Agave Developer’s Website. To get started using the APIs right away, pull down the Agave CLI as source or a Docker image and take it for a spin.
The Agave Developer APIs are distributed as multiple Docker images. This repository serves as a aggregation point for all the different services and a common configuration point used to propagate various configuration options when compiling from source. One of the things we love about Docker is that it simplifies our deployment environments. Each API is built into its own image and that exact same image is used in development, staging, and production. Because of this, we make all our images available publicly in the Docker Public Registry.
The Agave core services are currently comprised of over a dozen different containers. With their third-party dependencies and support containers, the total container count can surpass 30. Starting, stopping, monitoring, etc all those containers would be challenging on its own. We support Docker Compose as our development container orchestration solution for managing the bundled containers and their dependencies. Included in this repository you will find a compose directory where you will find example docker-compose.yml files to deploy the platform in various configurations.
In the following sections we walk you through what it takes to stand up and interact with the stack. For more information on how you can leverage the Agave Platform to power your own digital lab, pleas visit the Agave Platform Website.
You will need to have the Docker Engine installed to run the API containers and their dependencies. You will need to have Docker Compose installed to automate, manage, and scale the APIs. Both technologies have native installers for every platform available from the links below.
- Docker: an open platform for developers and sysadmins to build, ship, and run distributed applications.
- Docker Compose: Compose is a tool for defining and running multi-container applications with Docker.
If you intend on building the APIs from scratch, you will also need the following:
- [Java 9]
- [Go 1.14]
- [Maven 3.6+]
When you start up the APIs, each API will be assigned a dynamic port. This is purely for development purposes. In order to access the APIs, you will always point your requests to the Traefik reverse proxy container which load balances traffic to the APIs. The proxy image handles URL rewriting, load balancing, and port mapping so you do not have to worry about runtime changes if you restart or scale a running API.
The proxy container exposes ports 80, 443, and 28443. By default, SSL is enabled and all HTTP traffic is redirected to port 443.
In the event you will be interacting with an FTP or GridFTP server, you will also need to make sure that the Docker host on which the APIs are running has the standard FTP and GridFTP port ranges installed. Subsets of these port ranges are exposed by default in both the API and worker containers and are necessary to avoid performance penalties when interacting with remote servers over these protocols.
| Protocol | Ports |
|---|---|
| ftp | 21, 30000, 31000 |
| gridftp | 21, 2811, 50000, 51000 |
Finally, to make networking a little simpler, and to avoid diving into the rabbit hole that is overlay networking, please edit your Docker host's /etc/hosts file add the following hostnames as aliases to the host ip address in to your docker host's ip address file. This hostname will be used throughout the documentation to avoid problems with service discovery, firewalls, NAT, and philosophical differences in the way different operating systems handle networking.
The Agave Developer APIs require three external services in order to function:
- MariaDB: An enhanced, drop-in replacement for MySQL.
- MongoDB: an open-source document database, and the leading NoSQL database.
- Beanstalkd: a simple, fast work queue.
Each of these services is available as a Docker image. We primarily include them here to make you aware of the dependencies. In practice, these services will be deployed in Docker containers along with the core services, so you don't need to worry about them.
If you plan on testing out email notifications locally, you will need to edit the
compose/docker-compose.persistence.ymlfile to uncomment the local maildev container. To use your own email server, leave the maildev container commented out, and edit thecompose/configs/common.conffile with the proper values for theMAIL_SMTPS_*variables to match your smtp relay server. By default, email notifications are serialized to JSON and writting to the service logs.
The Agave Core APIs are automatically built and tested as part of our CI/CD workflow on every commit. Docker images for successful builds are automatically created and pushed into the Docker public registry. Unless you are actively developing against the code base, you do not need to build the images yourself. The
docker-compose.ymlfile will guarantee you have the latest version of the APIs running.
To create a development build of the APIs, you will need to check out this project, and run the Maven build. The following commands will build and package each API, build its Docker image, and tag it with the current version and project commit hash.
$ git clone http://github.com/agaveplatform/science-apis.git science-apis -b develop
$ cd science-apis
$ mvn -P agave,dev clean install
$ ./dockerbuild.sh -b agaveplatform
The compose directory contains Docker Compose files to launch the core services and its dependencies. The provided configuration is appropriate for a local development stack only. This should not be used in production environments.
To start up the science api stack run the following
$ cd compose
$ docker-compose -f docker-compose.persistence.yml -f docker-compose.core.yml down
Once the containers start (this may take a minute or two), they will be available at: http://<docker_host>:8080/. The tenants API will be available at http://<docker_host>:8080/tenants if you would like to configure your client to point to the development interfaces.
$ cd compose
$ docker-compose -f docker-compose.persistence.yml -f docker-compose.core.yml up -d
The APIs can be started using the following command. This will halt all containers while maintaining their data and runtime configuration.
$ cd compose
$ docker-compose -f docker-compose.persistence.yml -f docker-compose.core.yml down
The Agave Science APIs are under active development, so it's a good idea to update your images regularly. You can update all images and restart your container with the following commands.
$ cd compose
$ docker-compose -f docker-compose.persistence.yml -f docker-compose.core.yml pull
$ docker-compose -f docker-compose.persistence.yml -f docker-compose.core.yml down
$ docker-compose -f docker-compose.persistence.yml -f docker-compose.core.yml up -d
The APIs ship with a working instance of the Agave Live Docs. The Live Docs are a preconfigured version of Swagger UI, customized for use with Agave. You can visit the Live Docs for the APIs you just started at https://<docker_host>/docs.
Several options are available to you to explore your instance of the Agave Core Science APIs:
-
Agave CLI: a command line interface to the Agave Platform.
$ docker run -itt --rm -v $HOME/.agave:/root/.agave --name agave-cli agaveplatform/agave-cli:master bash -
Agave ToGo: a lightweight client-side, web application for interacting with Agave.
$ docker run -d -t -p 9000:9000 --name agave-togo agaveplatform/agave-togo:latest
Full documentation about the use cases satisfied by the APIs as well as functional documentation about their design, use, and performance, please see the Agave Website.
- Agave Developer Portal: the official developer portal for the Agave Platform.