encountered an issue try to consume the messages and get the schema, the message from Apicurio does not include the field "schemaType"

[kmandal-volvo]

We are using serializer and deserializer for protobuf components from Confluent since they are compatible with JRE 8. I have pointed our producer and consumer application to use endpoint http://localhost/apis/ccompat/v6 since that API endpoint is meant to be compatible with Confluent. Unfortunately, we have encountered an issue that prevents us from getting forward. When the consumer application tries to retrieve the schema it expects a JSON message that also includes the field “schemaType”, this field is needed by the Confluent deserializer to know that is should deserialize the records on the topic as PROTOBUF-messages. That field is not present in the response we get from Apicurio. If the schemaType-field is missing it will assume that the schema is of type “AVRO”. You can see it ourselves in below response from the CMD:

Apicurio Schema registry version we are using quay.io/apicurio/apicurio-registry-kube-sync:latest-snapshot and quay.io/apicurio/apicurio-registry-mem:2.0.1.Final.

{"schema":"package org.eclipse.tahu.protobuf;\n\noption java_package = \"org.eclipse.tahu.protobuf\";\r\noption java_outer_classname = \"SparkplugBProto\";\r\n\nmessage Payload {\n  optional uint64 timestamp = 1;\n  repeated .org.eclipse.tahu.protobuf.Payload.Metric metrics = 2;\n  optional uint64 seq = 3;\n  optional string uuid = 4;\n  optional bytes body = 5;\n\n  message Template {\n    optional string version = 1;\n    repeated .org.eclipse.tahu.protobuf.Payload.Metric metrics = 2;\n    repeated .org.eclipse.tahu.protobuf.Payload.Template.Parameter parameters = 3;\n    optional string template_ref = 4;\n    optional bool is_definition = 5;\n  \n    message Parameter {\n      optional string name = 1;\n      optional uint32 type = 2;\n    \n      oneof value {\n        uint32 int_value = 3;\n        uint64 long_value = 4;\n        float float_value = 5;\n        double double_value = 6;\n        bool boolean_value = 7;\n        string string_value = 8;\n        .org.eclipse.tahu.protobuf.Payload.Template.Parameter.ParameterValueExtension extension_value = 9;\n      }\n    \n      message ParameterValueExtension {}\n    }\n  }\n  message DataSet {\n    optional uint64 num_of_columns = 1;\n    repeated string columns = 2;\n    repeated uint32 types = 3;\n    repeated .org.eclipse.tahu.protobuf.Payload.DataSet.Row rows = 4;\n  \n    message DataSetValue {\n      oneof value {\n        uint32 int_value = 1;\n        uint64 long_value = 2;\n        float float_value = 3;\n        double double_value = 4;\n        bool boolean_value = 5;\n        string string_value = 6;\n        .org.eclipse.tahu.protobuf.Payload.DataSet.DataSetValue.DataSetValueExtension extension_value = 7;\n      }\n    \n      message DataSetValueExtension {}\n    }\n    message Row {\n      repeated .org.eclipse.tahu.protobuf.Payload.DataSet.DataSetValue elements = 1;\n    }\n  }\n  message PropertyValue {\n    optional uint32 type = 1;\n    optional bool is_null = 2;\n  \n    oneof value {\n      uint32 int_value = 3;\n      uint64 long_value = 4;\n      float float_value = 5;\n      double double_value = 6;\n      bool boolean_value = 7;\n      string string_value = 8;\n      .org.eclipse.tahu.protobuf.Payload.PropertySet propertyset_value = 9;\n      .org.eclipse.tahu.protobuf.Payload.PropertySetList propertysets_value = 10;\n      .org.eclipse.tahu.protobuf.Payload.PropertyValue.PropertyValueExtension extension_value = 11;\n    }\n  \n    message PropertyValueExtension {}\n  }\n  message PropertySet {\n    repeated string keys = 1;\n    repeated .org.eclipse.tahu.protobuf.Payload.PropertyValue values = 2;\n  }\n  message PropertySetList {\n    repeated .org.eclipse.tahu.protobuf.Payload.PropertySet propertyset = 1;\n  }\n  message MetaData {\n    optional bool is_multi_part = 1;\n    optional string content_type = 2;\n    optional uint64 size = 3;\n    optional uint64 seq = 4;\n    optional string file_name = 5;\n    optional string file_type = 6;\n    optional string md5 = 7;\n    optional string description = 8;\n  }\n  message Metric {\n    optional string name = 1;\n    optional uint64 alias = 2;\n    optional uint64 timestamp = 3;\n    optional uint32 datatype = 4;\n    optional bool is_historical = 5;\n    optional bool is_transient = 6;\n    optional bool is_null = 7;\n    optional .org.eclipse.tahu.protobuf.Payload.MetaData metadata = 8;\n    optional .org.eclipse.tahu.protobuf.Payload.PropertySet properties = 9;\n  \n    oneof value {\n      uint32 int_value = 10;\n      uint64 long_value = 11;\n      float float_value = 12;\n      double double_value = 13;\n      bool boolean_value = 14;\n      string string_value = 15;\n      bytes bytes_value = 16;\n      .org.eclipse.tahu.protobuf.Payload.DataSet dataset_value = 17;\n      .org.eclipse.tahu.protobuf.Payload.Template template_value = 18;\n      .org.eclipse.tahu.protobuf.Payload.Metric.MetricValueExtension extension_value = 19;\n    }\n  \n    message MetricValueExtension {}\n  }\n}\nenum DataType {\n  Unknown = 0;\n  Int8 = 1;\n  Int16 = 100  4634  100  4634    0     0  37072      0 --:--:-- --:--:-- --:--:-- 37370  UInt16 = 6;\n  UInt32 = 7;\n  UInt64 = 8;\n  Float = 9;\n  Double = 10;\n  Boolean = 11;\n  String = 12;\n  DateTime = 13;\n  Text = 14;\n  UUID = 15;\n  DataSet = 16;\n  Bytes = 17;\n  File = 18;\n  Template = 19;\n  PropertySet = 20;\n  PropertySetList = 21;\n  Int8Array = 22;\n  Int16Array = 23;\n  Int32Array = 24;\n  Int64Array = 25;\n  UInt8Array = 26;\n  UInt16Array = 27;\n  UInt32Array = 28;\n  UInt64Array = 29;\n  FloatArray = 30;\n  DoubleArray = 31;\n  BooleanArray = 32;\n  StringArray = 33;\n  DateTimeArray = 34;\n}\n"} ```

[kmandal-volvo]

Thanks for taking this issue. please help us once this issue fix.

[carlesarnal]

This is related to this

[carlesarnal]

Hi @kmandal-volvo,

When you say "That field is not present in the response we get from Apicurio" at which endpoint are you referring? According to the API reference, Confluent is not returning the schemaType in any endpoint (they are using it as a request parameter) so this part confuses me a bit. If they're returning the schemaType in the response then that's not documented.

That said, I've realized that we're missing the schemaType in, at least, two endpoints as a request parameter, is this what you were referring to?

In any case, thanks for your interest!

[michaelpearce-gain]

@carlesarnal quickly looking at this, because schematype and references are not returned in the endpoints, you can see this clearly in confluent code....then Confluents serdes cannot work as they need to know the type is protobuf and the linked referenced schemas for complex proto schemas with imports.

[carlesarnal]

yeah, this can be considered a bug in apicurio. We don't have these fields as they're not documented in the API reference. That said, they're clearly present in the code, so I will revisit this and try to fix it, thanks.

[carlesarnal]

Closing in favour of this since this is a bug in Apicurio Registry Compatibility API.

[kmandal-volvo]

Closing in favour of this since this is a bug in Apicurio Registry Compatibility API.

I do understand from the conversation but How I will know which version or release Apicurio will fix the issue

[carlesarnal]

This issue with the schemaType was fixed in this PR. You can expect it to be present in the next release of Apicurio Registry.

[kmandal-volvo]

When we can expect this release of Apircurio Registory

[carlesarnal]

You have a RC1 of 2.1.0 available here.