id: eventarc-apihub summary: Workflow-based automation for Apigee X using API hub and Eventarc author: Matteo Longo
A common use-case for integrating Apigee with API hub using Eventarc and Workflows.
We are sharing the required artifacts and providing the details on the step-by-step process of a fictitious real production deployment together with all the concepts and corresponding design choices. We want you to also get up and running in your project implementation with minimum changes or take inspiration and further customize this solution according to your needs.
Apigee API hub enables you to consolidate and organize information about all of the APIs in your company's organization. It includes APIs at all stages of their lifecycle, from design and implementation through deprecation and retirement. This allows developers to discover and leverage the work of other teams wherever possible. It also allows API platform teams to better manage their portfolio of APIs. The API hub experience is primarily aimed at consumers and producers of APIs within an organization whose main concerns are finding and listing APIs in their organization's catalog. API hub acts as a frontend for the API Registry to view and manage your organization's APIs. Permissions in the API hub and API Registry are controlled through Google Cloud IAM.
Even if API hub is a relatively new addition to the family of products under the Apigee API management umbrella, it has already seen a significant interest by customers and system integrators, especially in large enterprises that have a clear need to organize, maintain and publish their company wide API catalogs. Such catalogs can easily grow to hundreds and even thousands of APIs; it is therefore a natural choice to study the best way to automate the creation of API proxies, their versioning, their deployment and the synchronization of their deployment status and latest revision to the catalog maintained and published in the API hub.
We will show how Google Cloud is well equipped to provide an answer to the need for complete automation of the full lifecycle of these APIs, reliably and at scale.
We will specifically look at automating the deployment workflow of APIs for which the development team is wishing to author and maintain the corresponding WSDL definitions of the SOAP-based backend interfaces and also REST APIs with their corresponding OpenAPI specifications; in fact they want to use these API definitions to autonomously generate the API proxy bundles and corresponding artifacts, with the objective of letting these govern the API gateway behavior. Obviously customers want to include autonomous deployment, minimal (i.e. zero) human intervention to prepare the proxy bundles at scale, increment their versions in correspondence of changes to the definitions, store the API proxy bundles and the corresponding versioning metadata in source code repositories and automatically publish the most up-to-date and accurate API deployment information to the API hub console and API registry.
We assume that by now you can immediately grasp the benefits of achieving this with full automation, at scale: it saves time, removes human errors, allows API administrators to revert back quickly a previous version of the definitions in case of problems with the new version while reducing chores. It also guarantees that all API consumers (both within the organization and external, according to the exposure profile and permissions for the API hub and Api Registry) are never ever presented with inaccurate, incomplete or obsolete information about the overall deployment status of the APIs in the Apigee X runtime.
We present the full automation and orchestration approach starting from the creation of APIs from their specifications using only cloud native tools. The most important one that allows to prepare a full event driven Google Cloud solution is Eventarc. Both API hub and Apigee X can natively emit signals (called triggers) to Eventarc when specific events happen. Full documentation is published for them and their references are included at the end.
We will describe how to leverage API hub, Google Cloud Workflows, Apigee X management APIs and the corresponding events generated by Eventarc to provide the triggering events for the Workflows and implement full orchestration.
Although the creation of API bundles from the SOAP WSDL definition (alternatively for REST endpoints: OpenAPI specifications) of the target backend interfaces is proposed as the first step of the automation chain, it is provided as an example. It leverages open source components that have already been developed and open sourced to handle this scenario and we want to present it for the sake of completeness. These tools have also been extended to support additional use cases and you will find the references and links to their corresponding Github repositories in a dedicated section at the bottom. You are clearly free to use the orchestration approach proposed here in more diverse use cases.
Integration and synchronization with Cloud Source repositories, GitHub or GitLab or a different source code management tool is also highly desirable and we will also show an example of this as part of this lab solution description.
We will always reference the Apigee X API management platform, but it is worth noting the automation approach we are proposing is far more generic and is the same for customers who opted for the Apigee hybrid deployment option instead of X. Moreover API hub and its API registry are not meant to be used only by Apigee API management customers: it is intended to provide a convergent and unified API documentation hub and registry for customers relying on solutions from different commercial API gateways, bespoke home grown solutions and even heterogeneous multiple vendors. It is meant as a transparent unification layer and the first consolidation step towards the migration to Apigee in large API-first enterprise transformation projects.
In the lab you will find two directories:
-
/cloud-builder: it contains a custom builder for Google Cloud build that add all the tools required for the automation workflow to the Debian 11 container image. In order to correctly build the docker image you need to download the open source tool wsdl2apigee
-
/workflows: it contains the Google Cloud Workflow templates that implement the two case of automation
In order to create a custom builder you must follow this steps:
- Download the opensource tool wsdl2apigee
$ git clone- Compile wsdl2apigee tool using
$ cd wsdl2apigee/
$ mvn install-
Copy the compiled .jar into the cloud-builder directory.
-
Build and push the custom builder to an Artifact Registry in your project, replacing values for project-id, repo-name and image-name:
$ gcloud builds submit --tag europe-docker.pkg.dev/project-id/repo-name/image-nameLike any other builder, a custom builder runs with the source mounted under /workspace, and is run with a working directory in /workspace. Any files left in /workspace by a given build step are available to other build steps.
In order to setup the automation workflow you will need to create the two separate Google workflow:
- Enable the Workflows and Cloud Build API.
$ gcloud services enable workflows.googleapis.com$ gcloud services enable cloudbuild.googleapis.com- Create the service account and give it a name; for example, sa-name
$ gcloud iam service-accounts create sa-name- To comunicate with Apigee X, grant the following role to the service account (substitute the PROJECT_ID variable):
$ gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:sa-name@PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/eventarc.eventReceiver"$ gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:sa-name@PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/apigeeregistry.editor"$ gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:sa-name@PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/cloudbuild.builds.builder"- Deploy the first workflow and associate it with the specified service account:
$ gcloud workflows deploy apigee-notifydeployment-apihub --source=apigee-notifydeployment-apihub.yaml \
--service-account=sa-name@PROJECT_ID.iam.gserviceaccount.com- Deploy the second workflow and associate it with the specified service account:
$ gcloud workflows deploy apiregistry-updatespec-workflow --source=apiregistry-updatespec-workflow.yaml \
--service-account=sa-name@PROJECT_ID.iam.gserviceaccount.com- Create the required Eventarc trigger:
$ gcloud eventarc triggers create apiregistry-spec-update-trigger \
--location=global \
--service-account="${TRIGGER_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" \
--destination-workflow=apiregistry-updatespec-workflow \
--destination-workflow-location=europe-west3 \
--event-filters="type=google.cloud.apigeeregistry.apiSpec.v1.updated"$ gcloud eventarc triggers update apigeex-deploy-trigger \
--location=global \
--service-account="${TRIGGER_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" \
--destination-workflow=apigee-notifydeployment-apihub \
--destination-workflow-location=europe-west3 \
--event-filters="type=google.cloud.audit.log.v1.written" \
--event-filters="serviceName=apigee.googleapis.com" \
--event-filters="methodName=google.cloud.apigee.v1.DeploymentService.DeployApiProxy"Customers have shown a great interest in automating at scale the publication of API proxies in their fully managed Apigee X organization in correspondence of the publication from the developers of the specifications docs onto the Apigee API hub; the API proxies could be generated autonomously either as pass-through proxies (or as compliant and pre-populated by a given set of pre-configured Policy configurations and even a standard template including flow callouts to common organization and/or environment level resources).
We want to define a sequence of events that can be autonomously generated and handled in occurrence of each. These events correspond to the first entry point for the API hub emitting an Eventarc trigger when an API developer interacts with the API hub user interface; we will describe what happens next in Step 2.
For simplicity, the high level detail above shows the scenario where the API Proxy bundle being a straightforward pass-through proxy, i.e. the proxy bundle does not exist yet and does not need to be retrieved in advance of its manipulation (it is simply created without policies) from the GIT Repository.
API hub emits an Eventarc trigger event upon one of the configured events that we want an automation task to act upon (according to step 1); we will describe in greater detail in the following steps what actions are initiated according to such conditions being met. Each individual trigger event will initiate a new instance of the Workflows execution.
All orchestration presented here is easily implemented using configuration in Google Cloud Workflows. The workflow engine is serverless and runs in Google Cloud; it is hence a cloud native tool that can be configured to initiate workflows on receiving an Eventarc specific trigger.
Workflows is configured to initiate when the Eventarc trigger coming from API hub on new events corresponding to either:
- the creation of a new API in API hub starting from its SOAP WSDL or OpenAPI specs;
- the update of an existing API in API hub, when a new version for its OpenAPI specs have been uploaded or - for example - when the corresponding SOAP WSDL file has been amended with a new version (or, for REST API endpoints, a different OpenAPI specification file is uploaded).
Also note that an alternative (separate) entry point for the orchestration pipelines described in this article is the publication from an authorized Apigee API management console user of either a new API proxy or a new version of an existing API proxy.
An example using the Google Cloud portal on how to configure at Step 2 a new trigger for the scenario described in Step 3.b is included below. Each corresponding new event will initiate a new instance of Workflows. Triggers can obviously be also configured via gcloud; this would be the optimal and recommended approach when using automation; how to do it is described here.
Cloud Build is responsible for preparing the proxy bundle, this is the one that will be deployed. Optionally (as described in this lab use case) additional open source tools will need to be invoked to prepare the API proxy bundle starting from the API spec that has been uploaded to the API hub, which contains the OpenAPI specs of the proxy and the full WSDL of the SOAP interface for which the client wants to be able to automatically prepare a pass-though or more sophisticated API proxy.
The open source tools used are also added to the additional resources section of this article and are openapi2apigee and wsdl2apigee. They are publicly available on github.com.
Upon completion of the necessary build steps, the API proxy bundle and all relevant artifacts and dependencies for the deployment have been prepared according to the Workflows orchestration and the Cloud Build configuration. This step consists of invoking the Apigee Management APIs for the deployment to the target Organization and Environment of the versioned API proxy (either a new API proxy or a new version of an existing API proxy). With a successful deployment, the Workflows orchestration job has now completed.
Apigee X is able to emit Eventarc events when an API proxy revision is created and when a new deployment is performed (see the referenced documentation for additional details). We use these upon both the following scenarios:
-
the successful deployment a new API proxy into a given target environment within a given Apigee X organization;
-
the successful deployment of a new version of an existing API proxy in a given Apigee X organization.
In both these cases, the trigger will carry information on the organization, the environment, the name of the proxy bundle, the proxy bundle version, additional relevant metadata (for example, the requester of the deployment).
This approach creates a separate (the second, separate) entry point for the orchestration and automation pipeline built with GCP Workflows. It is a cloud native pipeline and does not rely on external dependencies.
This step is necessary to complete the final publication event onto the API hub, so that it is always kept in sync with the correct and most up-to-date list of APIs deployed onto the Apigee X organization, so that the API catalog is kept in sync with the state of the API runtime gateway.
There is one major benefit of this approach: Workflows instances will be triggered autonomously also when a new API proxy deployment (or an API proxy deployment of a new version) is performed by an operator (with sufficient privileges) via the Apigee Management UI or by another independent automation service account, still with enough machine user privileges (for example, an independent automation pipeline being executed by another existing build & deploy process).
As anticipated in the description of the Step 7, this represents the final step and this is the one that guarantees that the most up-to-date list and version of each published API in the API runtime gateway for the corresponding organization is presented to all (via UI and automation) users of the API hub.
The Workflow has its own error handling logic, it is asynchronous and has eventual consistency design patterns so that even in case the API hub has a problem in receiving the notification immediately, this will not be discarded.
- openapi2apigee - An open source project that converts a standard Open API spec (previously known as Swagger) into an Apigee API Proxy bundle and deploys into an Apigee Organization.
- wsdl2apigee - An open source project that extends Apigee functionality to provide the ability to expose a SOAP Service as a REST API (over JSON or XML) or expose a SOAP Service in a pass through mode using a WSDL descriptor.
All solutions within this repository are provided under the Apache 2.0 license. Please see the LICENSE file for more detailed terms and conditions.
For questions, issues and comments raise a github issue