Skip to content

Protocol Methods Quickstart with AutoRest

Jump to bottom
Srikanta edited this page Oct 12, 2023 · 121 revisions

Before you Start

Java Azure SDK Design Guidelines is the overall design guideline of the data-plane SDK. Make sure you are familiar with concepts such as "Service Client" and "Packaging".

Make sure you are familiar with Git and Maven, especially the Build Lifecycle.

Please also take a look at Protocol Methods on the overview of the typical protocol methods from minimal data-plane client SDK.

⚠️ Note that if this is not the first time you generate Java SDK code, please go directly to Follow-up code generation.

Generate SDK

You can choose from

  1. Python script, Java developer environment required
  2. Azure pipeline

to create your first data-plane client SDK.

☑️ If you have access to the Azure pipeline, it is recommended to use the pipeline.

If you already have SDK generated, but want to update it (for example, when swagger specification updated, or new api-version added), please see Follow-up code generation.

Use Python script for first-time code generation

A Python script in Azure SDK for Java can help generate a minimal data-plane client SDK from OpenAPI 2.0 specification.

Please see Working with AutoRest for the installation of autorest CLI. It also requires the installation of Node.js 16 or above, Java 8 or above, and Python 3.

Python package PyYAML and requests is required to be installed, e.g. via pip3 install --upgrade PyYAML requests.

If you are new to Java development environment, see Java Development Environment for details.

☑️ Required options from user:

  • spec-readme: URL to README.md, which have OpenAPI 2.0 specification JSON as input-file in default tag. It is recommended over the input-file option below, as the README.md potentially includes more configurations like security and title.
  • input-file: URL to OpenAPI 2.0 specification JSON as input file.
  • service: Service name under sdk/, sample: storage.
  • module: Module name under sdk/<service>/, sample: azure-storage-blob.

Optional options:

  • security: Credential types, sample: AADToken for AAD credential for OAuth 2.0 authentication. See Authentication in AutoRest, and we recommend service to define them in OpenAPI.
  • security-scopes: OAuth 2.0 scopes, required if security includes AADToken.
  • title: Client name. The name should always ends with "Client", sample: BlobClient, which results in BlobServiceVersion as service version class.

For service, it is usually your service name in REST API specifications. For instance, if the JSON is under specification/storage, the service would normally be storage.

For module, please refer to Namespace in Java Guideline. For example, if the namespace is com.azure.storage.blob, the module should be azure-storage-blob.

☑️ Namespace is decided by SDK arch board.

As an example,

python eng/mgmt/automation/generate_data.py --spec-readme https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/purview/data-plane/Azure.Analytics.Purview.Account/readme.md --service=purview --module=azure-analytics-purview-account

This will generate the SDK under sdk/purview/azure-analytics-purview-account. And your follow-up work will mostly happen in this SDK folder.

Tips: The Python script also builds and verifies the generated project. It is possible that the verification fails due to Java code quality requirements. See Build section on how to fix.

Use Azure pipeline for first-time code generation

There is an internal pipeline java - data-plane generator, which invokes the the Python script, generate the code, and create pull request of the generated SDK.

It takes similar input parameters as the script, e.g. SPEC_README/INPUT_FILE variable for spec-readme/input-file option, SERVICE and MODULE variable for service and module option.

A draft pull request will be created at pull requests, named as [Automation] Generate Data-plane SDK for <module>. Here is an example PR created by the pipeline: https://github.com/Azure/azure-sdk-for-java/pull/28336

Follow-up code generation

An AutoRest configuration file is generated at sdk/<service>/<module>/swagger/readme.md during the first-time code generation. You can now edit it, and generate code through this configuration file.

If you have not installed the AutoRest, please see Working with AutoRest for the installation of autorest CLI. It requires the installation of Node.js 16 or above.

As an example, autorest --java --use=@autorest/java@latest sdk/<service>/<module>/swagger/readme.md

☑️ It is advised to use the latest version of @autorest/java extension (see AutoRest.Java Release).

See AutoRest flags, and Java Data-Plane Client for details.

All possible Java flags can be found either from autorest --help --java, or from AutoRest.Java readme.

Build

See Building and Adding a Module.

See the contributing guide.

Here is the list of files that you are required to modify/confirm for your new module.

  • eng/versioning/version_client.txt for versioning
  • sdk/<service>/pom.xml for project
  • sdk/<service>/ci.yml for CI

The Azure pipeline or Python script automatically created/modified these files for you. However, it is still advised that you double check them.

Install for Local Test

See Building. Install SDK to local maven repository: mvn install -f sdk/<service>/<module>/pom.xml -Dgpg.skip -Drevapi.skip -DskipTests

Tips: Javadoc might fail if other than Java 11. If you see error message like "Error while generating Javadoc", you can add option -Dmaven.javadoc.skip to skip Javadoc.

See Maven Getting Start Guide, if you would like to create a client that uses your SDK. See sdk/<service>/<module>/README.md for adding the SDK to dependency.

Improve SDK Documentation

There is many information about the SDK that AutoRest will never know. These can only be provided by a contributor.

README

A minimal README.md is generated by AutoRest. Your first task will be improving it.

☑️ You would want to modify [product_documentation] to point to your service documentation.

Samples

A blank Java sample file is generated as ReadmeSamples.java, which is synchronized with your README (see Codesnippet Maven Plugin).

Add your sample code between // BEGIN: ... and // END: ... in ReadmeSamples.java, and build the project, you will find your Examples section in README updated with the code.

A collection of generated Java sample files (based on x-ms-examples in OpenAPI specs) is generated under <module>.generated package. They might not be directly usable as runnable sample codes, however you should get the basic idea on how code works, and may even modify some to be runnable.

An example of adding samples is demonstrated in this commit.

Tests

A <client>TestBase file and a collection of generated Java test files (based on x-ms-examples in OpenAPI specs) is generated under <module>.generated package.

See Adding Tests.

Feature Checklist

Authentication and credential

AutoRest supports OAuth2 and ApiKey authentication.

However, if service requires advanced solution (for example, JWT claim or HMAC-SHA256 authentication scheme), you are responsible to provide the implementation (usually as a subclass of HttpPipelinePolicy), and customize the ClientBuilder.

Long-running operation

AutoRest.Java supports a default chained polling strategy for long-running operation.

It supports status monitor LRO and simple Location-header-based LRO. It does not support the outdated resource based LRO.

If service uses non-supported or non-standard LRO, you can contact Java SDK for guidance on the implementation of a specialized polling strategy, and customization in swagger/readme.md.

File upload

AutoRest.Java supports application/octet-stream, but it does not support multipart/form-data. An issue in azure-core tracks the status of the feature.

Review API Documentation

Generate Javadoc: mvn javadoc:javadoc -f sdk/<service>/<module>/pom.xml

Review the generated API documentation sdk/<service>/<module>/target/site/apidocs/index.html

Improve SDK Code

ClientBuilder, Client, AsyncClient is the user-facing API surface that you might be interested to improve. See Partial Update and Customization.

Code Review

Code review happens on 2 places.

One is the usual GitHub pull request, for developers.

Another is the apiview, for architects. GitHub pull request should automatically trigger the apiview. If this does not happen, upload one manually:

  1. Build the SDK.
  2. Login apiview with GitHub account.
  3. "Create review" at top-left corner.
  4. "Browse" and upload sdk/<service>/<module>/target/<module>-<version>-sources.jar, and comment the result link in PR. If you did not find the source jar in step 4, run mvn source:jar.

Comment @Azure/dpg-devs for awareness in PR to loop in SDK developers for review.

Package Release

See Release Checklist.

Prepare Release Script

In particular, you need to follow "Prepare Release Script" and make another "prepare release" PR.

Before running the script, make sure your CHANGELOG.md is correct. Except for first release, it should contain following text on dependency updates.

### Other Changes

#### Dependency Updates

- Upgraded `azure-core` from `1.32.0` to version `1.33.0`.
- Upgraded `azure-core-http-netty` from `1.12.5` to version `1.12.6`.

After running the script, check the "release date" and make sure it is your scheduled release date.

Merge Increment Versions PR

After package release, one will see an "Increment versions for releases" PR. Please ping the PR to SDK developer to approve and merge.

Appendix

Java Development Environment and AutoRest

Windows

Download Java, install.

Download Maven, unpack, set to PATH.

Download Node.js, install.

Install AutoRest with npm install -g autorest in command line.

Optional for Git and Python:

  • Download Git, install.
  • Download Python, install.

Linux with APT

sudo apt -y install openjdk-17-jdk

sudo apt -y install maven

curl -sL https://deb.nodesource.com/setup_16.x | sudo bash -
sudo apt -y install nodejs

npm install -g autorest

sudo apt -y install python3
sudo apt -y install python3-pip
pip3 install --upgrade wheel
pip3 install --upgrade PyYAML requests

AutoRest configuration in SDK repository

Prepare the same AutoRest configuration at sdk/<service>/<module>/swagger/readme.java.md. In addition to the required flags, do add input-file flag pointing to your OpenAPI specs.

Then use Use Python script for first-time code generation to generate the SDK.

Adding Tests

Adding runnable tests requires both a good understanding of the service, and the knowledge of the client and test framework. Feel free to contact SDK developers, if you encountered issues on client or test framework.

Test environment

Before testing the client, it is recommended that you prepare a Bicep or ARM template (or code, or CLI script, equivalent to the ARM template) that creates all required Azure resources and assigns necessary RBAC for your client. This ARM template will be very helpful for you to re-run the tests after version update, or for other developers to create their test environment.

Client authentication

If your service does not support connection string or API key authentication, it would likely use AzureAD OAuth2 token for authentication. A common solution is to provide an application and its service principal and to provide RBAC to the service principal for the access to the Azure resource of your service.

Client requires following 3 environment variables for the service principal using client ID/secret for authentication:

AZURE_TENANT_ID
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET

On Linux, you can call export:

export AZURE_TENANT_ID=<tenant_id>
export AZURE_CLIENT_ID=<client_id>
export AZURE_CLIENT_SECRET=<client_secret>

On Windows, it is set.

Live tests

Live tests require one more environment variable:

export AZURE_TEST_MODE=LIVE

AutoRest.Java already provided you the generated Java test files at sdk/<service>/<module>/src/test/java/<namespace>/generated folder, and a <client>TestBase class. You should double check the beforeTest() method in <client>TestBase class, and modify it if necessary.

It is recommended to copy the test files you would like to modify and use to sdk/<service>/<module>/src/test/java/<namespace> folder.

It is common for you to put configuration from your test environment to the code, as long as they are not a secret. Alternatively, you can keep rely on Configuration.getGlobalConfiguration().get("<env_variable>") and define a few additional environment variables.

After you settle down the builder, you can decide which REST APIs you want to test, and write the code and verification in the testClient() method. The generated samples under <module>.generated package can be used as reference for your test code.

After you finish the test code, call mvn test under sdk/<service>/<module> to execute the tests. If everything works fine, you will see following log.

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.776 s - in com.azure.messaging.webpubsub.ClientTests
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

An example of adding live tests is demonstrated in this commit.

Record/playback tests

The live tests only run when environment variable AZURE_TEST_MODE is set to LIVE, and hence it is usually run manually.

Next step, you will convert the live tests to record/playback tests.

⚠️ Record/playback tests MUST use the Test Proxy tool. If your previous tests is recorded in repository, please follow Test Proxy Migration.

⚠️ Use TestBase.setPlaybackSyncPollerPollInterval on the SyncPoller instance, if your test involves long-running operation API.

Record

Firstly, the tests need to be recorded.

Remove @DoNotRecord(skipInPlayback = true) annotation in ClientTests.java. And set environment variable AZURE_TEST_MODE to RECORD:

export AZURE_TEST_MODE=RECORD

Then call mvn test as if this is live tests.

Please follow this guide on updating the test recordings.

☑️ Permission is required for updating the asserts repository. If you failed in test-proxy push, please join a partner write team.

Playback

These recorded tests can now be run in playback mode automatically (it can be run locally via the same mvn test call, by setting environment variable AZURE_TEST_MODE to PLAYBACK).

Be aware that in playback mode during continuous integration, there is no test environment nor environment variables available. Hence if you rely on Configuration.getGlobalConfiguration().get("<env_variable>"), make sure you provide a second parameter as the default value, for the case when <env_variable> is not available.

Enable test coverage

Modify pom.xml to enable test coverage.

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-    <jacoco.skip>true</jacoco.skip>
  </properties>

At beta, one can use a lower minimum coverage.

Partial Update

Since autorest/java@4.0.42 and with partial-update option, you can in-place modify the generated code, and AutoRest.Java will keep the hand-written code intact when it re-generates the SDK.

The user-facing API is of pattern:

FooClient client = new FooClientBuilder()
    .endpoint(...)
    .credential(new DefaultAzureCredentialBuilder().build())
    .buildClient();

FooAsyncClient asyncClient = new FooClientBuilder()
    .endpoint(...)
    .credential(new DefaultAzureCredentialBuilder().build())
    .buildAsyncClient();

And the classes involved is FooClient, FooAsyncClient, and FooClientBuilder.

All of these classes can be customized in-place, to enhance the usability of the SDK, or to add new feature.

You can add new class variables and new class methods to these classes, and with partial-update option in swagger/README.md, new code generated via AutoRest will not override your change.

For modifying the signature or the implementation of a class method, please remove the @Generated annotation, so AutoRest.Java can recognize that you have modified the method.

If you would like to remove a class method, simply make the method package private, so it is not exposed to customer. Also you need to remove the @Generated annotation.

It is advised that you keep the modifications simple. If certain feature requires complicated code, it is better to put the implementation code in some other classes within implementation package, and only add the public class method and invocation in these Client or ClientBuilder classes.

Troubleshoot CI Failure

Check "Build" and "Analyze" first, the "Test"s.

It helps to run mvn clean verify locally, before checking CI failure.

Verify versions in POM files

See the contributing guide on versioning.

TLDR: run python eng/versioning/update_versions.py --ut library --bt client --sr

Check spelling

Fix spelling in code or in markdown.

Suppression at .vscode/cspell.json.

Verify Readmes / Verify sample metadata / Verify ChangeLogEntry

Fix corresponding files.

Run SpotBugs, Checkstyle, RevApi, and Javadoc

Fix code for SpotBugs, Checkstyle, and Javadoc.

If you run it locally, make sure your build tool is re-installed.

Avoid breaking changes for RevApi.

Suppression at eng/code-quality-reports/src/main/resources.

Run tests for From Source run

Besides test failure, it could also be caused by insufficient test coverage.

[WARNING] Rule violated for bundle azure-ai-anomalydetector: lines covered ratio is 0.21, but expected minimum is 0.40
[WARNING] Rule violated for bundle azure-ai-anomalydetector: branches covered ratio is 0.27, but expected minimum is 0.30

For beta SDK, you are allowed to lower the minimum coverage, by adding following Jacoco settings in pom.xml.

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+    <jacoco.min.linecoverage>0.2</jacoco.min.linecoverage>
+    <jacoco.min.branchcoverage>0.2</jacoco.min.branchcoverage> 
  </properties>

For GA SDK, it is expected to follow the default coverage, which is 0.40 for line coverage, 0.30 for branch coverage.

Client Library

User Guides

Extension Points

Contributor Guides

Management Library

User Guides

Contributor Guides

Spring Cloud Azure

Clone this wiki locally