WIP [CAMEL-18698] Add support for multiple input/output data types on components

[christophd]

WIP do not merge!

This PR is supposed to be a work in progress seeking for guidance and help on implementing multiple data types on components feature as described in https://issues.apache.org/jira/browse/CAMEL-18698

The changes made in this PR should be seen as a POC starting some discussions and refinements of the feature.

The idea is to let the user choose a specific data type that the component is producing as an output. The component itself may offer specific data transformation logic that gets automatically triggered when this data type is explicitly used on an endpoint URI:

from("kafka:topic?format=avro").to("aws2-s3:bucketName?format=string")

The format option tells the component to apply the specific data type as an input or output. The data type may be directly supported by the component or it may be a generic data type that makes use of the Camel TypeConverter logic to transform from one data to another.

In the PR the feature is implemented as a POC for aws2-ddb and aws2-s3 components. The aws2 components often use Java POJO domain objects as input and output so the user needs to know the domain model when interacting with the components.

The idea is to also add support for more generic data types on these components. As an example the user may also provide a generic Json structure as an input instead of the AWS domain object. The component takes care on transforming the data type to the required Java POJO used in the AWS client libraries.

On the output side the aws2-s3 component by default produces byte[] or InputStream output. With the addition of data types the user is able to request another data type such as String. The component makes sure to apply transformation logic from the AWS domain model InputStream implementation to a String.

This feature is beneficial for Camel users that do not have to know about domain specific Java types anymore. Also declarative Camel DSL use cases e.g. used in Camel K and Kamelets benefit from the auto data type conversion as discussed in apache/camel-k#1980

A Kamelet may use the provided data types information and expose this as part of the Kamelet specification.

apiVersion: camel.apache.org/v1alpha1
kind: Kamelet
metadata:
  name: aws-s3-source
spec:
  definition:
    title: "AWS S3 Source"
  properties:
    ...
  types:
    out:
      default: binary
      binary:
        mediaType: application/octet-stream
      stream:
        mediaType: application/octet-stream
      json:
        mediaType: application/json
        schema:
          type: object
          required:
            - key
            - fileContent
          properties:
            key:
              title: Key
              description: The S3 file name key
              type: string
            fileContent:
              title: Content
              description: The S3 file content as String
              type: string
        dependencies:
          - "camel:jackson"
      string:
        mediaType: plain/text
      avro:
        mediaType: application/avro

Also the user is able to choose the input/output data type in a binding:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: aws-s3-uri-binding
spec:
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: aws-s3-source
    output:
      type: avro
      schema:
        uri: http://schema-registry/person
    properties:
      outputFormat: avro
      bucketNameOrArn: ${aws.s3.bucketNameOrArn}
  sink:
    uri: log:info

The new data types implementation provided in this PR use a specific SPI annotation @DataType. A resolver mechanism DefaultDataTypeResolver is capable of doing a lookup of the respective data type implementation with given component scheme and a format name that identifies a data type.

The PR definitely needs more polishing and has some ToDos in the code that seek for guidance and help how to do it the Camel way (e.g. how to do an automatic resource lookup for data type implementations provided by components).

Also it would be nice to have some guidance on the already existing org.apache.camel.spi.DataType and how that could be related to this implementation.

Many thanks in advance!

[github-actions]

🌟 Thank you for your contribution to the Apache Camel project! 🌟

⚠️ Please note that the changes on this PR may be tested automatically.

If necessary Apache Camel Committers may access logs and test results in the job summaries!

[github-actions]

Components tested:

Total	Tested	Failed ❌	Passed ✅
7	7	2	5

[davsclaus]

closing this as there is another newer PR