AWS CloudTrail Processing Library

The AWS CloudTrail Processing Library is a Java client library that makes it easy to build an application that reads and processes CloudTrail log files in a fault tolerant and highly scalable manner.

• CloudTrail Product Page
• CloudTrail Forum
• Github Issues

Features

• Provides functionality to continuously download CloudTrail log files in a fault tolerant and scalable manner.
• Serializes the events in JSON format to Plain Old Java Objects (POJO).
• Provides interfaces to implement your own business logic for selecting which events to process, processing events, handling errors, and handling log processing status updates.

Getting Started

Minimum Requirements

• AWS Java SDK 1.12.261: To use the AWS CloudTrail Processing Library, you'll need the AWS Java SDK.
• Java 1.8: The AWS CloudTrail Processing Library requires Java 1.8 (Java SE 8) or later.

Documentation

To learn how to use the AWS CloudTrail Processing Library to build a CloudTrail log processor in Java, read the documentation:

• Using the CloudTrail Processing Library in the AWS CloudTrail User Guide.
• AWS CloudTrail Processing Library Reference

Building From Source

After you've downloaded the code from GitHub, you can build it using Apache Maven. To disable GPG signing in the build, use this command:

mvn clean install -Dgpg.skip=true

Release Notes

Release 1.6.3 (Oct 25, 2024)

• Added support for modeling InScopeOf
• Added support for modeling VpcEndpointAccountId

Release 1.6.2 (Nov 30, 2023)

• Added support for modeling EdgeDeviceDetails

Release 1.6.1 (May 19,2023)

• Added support for test scenario for UserType IdentityCenterUser

Release 1.6.0 (May 8,2023)

• Updated CloudTrailEvent version to 1.12
• Added support for UserIdentity

Release 1.5.2 (Sep 6, 2022)

• Update AWS Java SDK (S3/SQS) version to 1.12.261.

Release 1.5.1 (July 26, 2022)

• Update AWS Java SDK (S3/SQS) version to 1.12.x

Release 1.5.0 (Jan 26, 2022)

• Added support for implementing custom S3 manager.
• Added event logging to log file parsing-related exceptions.
• Added support for parsing optional errorCode field in insightDetails.
• Updated account ID parsing regex to accept non-numerical values.

Release 1.4.0 (Jan 11, 2021)

• Added support for parsing the following new top-level optional fields:
  • addendum
  • edgeDeviceDetails
  • tlsDetails
  • sessionCredentialFromConsole
• Updated the CloudTrail event version to 1.08.

Release 1.3.0 (Jul 30, 2020)

• Added support for parsing new section, attributions, in insightContext.
• Added support for parsing new fields, baselineDuration, in statistics section in insightContext.
• Added thread configuration for s3 client, sqs client, and sqs reader to enable performance tuning.
• Updated minimum required Java SE version to 1.8.

Release 1.2.0 (Nov 20, 2019)

• Added support for a new eventCategory attribute to indicate whether an event is a management, data, or Insights event.
• Added support for Insights events, including new attributes like insightDetails or insightContext.
• Updated the CloudTrail event version to 1.07.

Release 1.1.3 (Oct 18, 2018)

• Added support for automatically deleting the initial SNS validation message sent whenever an SNS topic for a trail is configured or updated. In previous releases, these messages had to be manually deleted.

Release 1.1.2 (May 16, 2018)

• Patch Release 1.1.1

Release 1.1.1 (Nov 30, 2017)

• Added support for Boolean identification of management events.
• Updated the CloudTrail event version to 1.06.

Release 1.1.0 (Jun 1, 2017)

• Add support for different formats for SQS messages from the same SQS queue to identify CloudTrail log files. This includes the following:
  • Notifications that CloudTrail sends to an SNS topic.
  • Notifications that Amazon S3 sends to an SNS topic.
  • Notifications that Amazon S3 sends directly to the SQS queue.
• Add support for the new deleteMessageUponFailure property. Use this property to delete messages that the CloudTrail Processing Library can't process, such as the following:
  • Parsing message failure:
    • File is not JSON.
    • Notification is not an s3:ObjectCreated:Put event.
    • CloudTrail digest files, and other formats such as .jpeg or txt are unsupported.
  • Consuming log failure, such as processing events in a log file.

Note: If deleteMessageUponFailure is true, the CloudTrail Processing Library may delete messages that it can’t process. The default value is false. Learn more.

Release 1.0.4 (Jan 17, 2017)

• Add support for ARN prefix to identify the ARNPrefix associated with the resource. Resource must have either ARN or ARNPrefix, but not both.
• Add support for shared event ID to identify CloudTrail events from the same AWS action that is sent to different AWS accounts.
• Add support for VPC endpoint ID to identify the VPC endpoint in which requests were made from a VPC to another AWS service, such as Amazon S3.
• Add support for annotation to identify user provided annotation tagging delivered by CloudTrail.
• Add support for identity provider to identify the principal name of the external identity provider.

Release 1.0.3 (Oct 5, 2016)

• Add support for service event, additional information is provided in the serviceEventDetails file.
• Add support for Resource type to identify the resource's type in a given CloudTrail event.
• Update AWS Java SDK to version 1.11.
• Update the latest supported CloudTrail event version to 1.05.
• Update event version is not supported by CloudTrail warning logging message to debug level.

Release 1.0.1 (Oct 28, 2015)

• Update AWS Java SDK to version 1.10.

Release 1.0.0 (Nov 3, 2014)

• Initial release.