-
Notifications
You must be signed in to change notification settings - Fork 524
Java OTF User Guide
Some applications, such as network sniffers, need to process messages dynamically and thus have to use the Intermediate Representation to decode the messages on-the-fly (OTF). An example of using the OTF API can be found here.
The Java OTF decoder follows the design principles of the generated codec stubs, and it is thread safe to be reused concurrently across multiple threads for memory efficiency.
Note: Due to the dynamic nature of OTF decoding, the stubs generated by the SBE compiler will yield greater relative performance.
Before messages can be decoded it is necessary to retrieve the IR for the schema describing the messages and types. This can be done by reading the encoded IR into a ByteBuffer and then decoding it using the provided IrDecoder.
private static IntermediateRepresentation decodeIr(final ByteBuffer buffer)
throws IOException
{
final IrDecoder irDecoder = new IrDecoder(buffer);
return irDecoder.decode();
}Once the IR is decoded you can then create the OTF decoder for the message header:
// From the IR we can create OTF decoder for message headers.
final OtfHeaderDecoder headerDecoder = new OtfHeaderDecoder(ir.headerStructure());You are now ready to decode messages as they arrive. This can be done by first reading the message header then looking up the appropriate template to decode the message body.
// Now we have IR we can read the message header
int bufferOffset = 0;
final UnsafeBuffer buffer = new UnsafeBuffer(encodedMsgBuffer);
final int templateId = headerDecoder.getTemplateId(buffer, bufferOffset);
final int actingVersion = headerDecoder.getTemplateVersion(buffer, bufferOffset);
final int blockLength = headerDecoder.getBlockLength(buffer, bufferOffset);
bufferOffset += headerDecoder.encodedLength();Note: Don't forget to increment the bufferOffset to account for the message header size!
Once you have decoded the header you can lookup the IR for the appropriate message body then begin decoding.
int bufferOffset = 0;
final UnsafeBuffer buffer = new UnsafeBuffer(encodedMsgBuffer);
final int templateId = headerDecoder.getTemplateId(buffer, bufferOffset);
final int schemaId = headerDecoder.getSchemaId(buffer, bufferOffset);
final int actingVersion = headerDecoder.getSchemaVersion(buffer, bufferOffset);
final int blockLength = headerDecoder.getBlockLength(buffer, bufferOffset);
bufferOffset += headerDecoder.encodedLength();
// Given the header information we can select the appropriate message template to do the decode.
// The OTF Java classes are thread safe so the same instances can be reused across multiple threads.
final List<Token> msgTokens = ir.getMessage(templateId);
bufferOffset = OtfMessageDecoder.decode(
buffer,
bufferOffset,
actingVersion,
blockLength,
msgTokens,
new ExampleTokenListener(new PrintWriter(System.out, true)));The eagle eyed will have noticed the TokenListener. If you are wondering what this is then wonder no longer and read on.
As messages are decoded a number of callback events will be generated as the structural elements of the message are encountered. The callbacks are received by implementing the TokenListener interface. If you only want to receive some of the callbacks then extend AbstractTokenListener.
Primitive fields are the most common data element to be decoded. These are simple types such as integers, floating point numbers, or characters. Primitive field encodings can be a single value or a fixed length array of the same type. To receive primitive values override the following method:
public void onEncoding(
final Token fieldToken,
final DirectBuffer buffer,
final int index,
final Token typeToken,
final int actingVersion)
{
final CharSequence value = readEncodingAsString(buffer, index, typeToken, actingVersion);
printScope();
out.append(fieldToken.name())
.append('=')
.append(value)
.println();
}
private static CharSequence readEncodingAsString(
final DirectBuffer buffer, final int index, final Token typeToken, final int actingVersion)
{
final PrimitiveValue constOrNotPresentValue = constOrNotPresentValue(typeToken, actingVersion);
if (null != constOrNotPresentValue)
{
return constOrNotPresentValue.toString();
}
final StringBuilder sb = new StringBuilder();
final Encoding encoding = typeToken.encoding();
final int elementSize = encoding.primitiveType().size();
for (int i = 0, size = typeToken.arrayLength(); i < size; i++)
{
mapEncodingToString(sb, buffer, index + (i * elementSize), encoding);
sb.append(", ");
}
sb.setLength(sb.length() - 2);
return sb;
}
private long readEncodingAsLong(
final DirectBuffer buffer, final int bufferIndex, final Token typeToken, final int actingVersion)
{
final PrimitiveValue constOrNotPresentValue = constOrNotPresentValue(typeToken, actingVersion);
if (null != constOrNotPresentValue)
{
return constOrNotPresentValue.longValue();
}
return getLong(buffer, bufferIndex, typeToken.encoding());
}The above code will output the values as strings to the console.
Note: Constant and optional fields are handled by using the metadata provided in the typeToken.
Enums are encoded on the wire as simple integers or characters. It is necessary to lookup the encoded representation via the metadata tokens to understand the wire encoded value.
public void onEnum(
final Token fieldToken,
final DirectBuffer buffer,
final int bufferIndex,
final List<Token> tokens,
final int beginIndex,
final int endIndex,
final int actingVersion)
{
final Token typeToken = tokens.get(beginIndex + 1);
final long encodedValue = readEncodingAsLong(buffer, bufferIndex, typeToken, actingVersion);
String value = null;
for (int i = beginIndex + 1; i < endIndex; i++)
{
if (encodedValue == tokens.get(i).encoding().constValue().longValue())
{
value = tokens.get(i).name();
break;
}
}
printScope();
out.append(fieldToken.name())
.append('=')
.append(value)
.println();
}BitSets are represented on the wire as an integer with a bit set in the position indicating true or false for the choice value.
public void onBitSet(
final Token fieldToken,
final DirectBuffer buffer,
final int bufferIndex,
final List<Token> tokens,
final int beginIndex,
final int endIndex,
final int actingVersion)
{
final Token typeToken = tokens.get(beginIndex + 1);
final long encodedValue = readEncodingAsLong(buffer, bufferIndex, typeToken, actingVersion);
printScope();
out.append(fieldToken.name()).append(':');
for (int i = beginIndex + 1; i < endIndex; i++)
{
out.append(' ').append(tokens.get(i).name()).append('=');
final long bitPosition = tokens.get(i).encoding().constValue().longValue();
final boolean flag = (encodedValue & (1L << bitPosition)) != 0;
out.append(Boolean.toString(flag));
}
out.println();
}A little bitwise manipulation is required to determine if a each choice is true or false as in the example above.
A composite is a reusable collection of fields to simplify the assembly of messages. The collection of fields usually has a semantic significance. Fields within a composite are decoded just like normal fields. Composites are signalled via callbacks to indicate the beginning and end of the composite. In the example, the begin and end are captured to scope fields by adding the scope to a stack in the example TokenListener.
public void onBeginComposite(final Token fieldToken, final List<Token> tokens, final int fromIndex, final int toIndex)
{
namedScope.push(fieldToken.name() + ".");
}
public void onEndComposite(final Token fieldToken, final List<Token> tokens, final int fromIndex, final int toIndex)
{
namedScope.pop();
}Fields can be semantically bound into a repeating group. On the wire the repeating group has a header that defines the size in bytes of the block of fields and a count of how many times the block will repeat. Repeating groups are signalled by callbacks to indicate the beginning and end of block of fields with counter details for the iteration count and the number of times it will repeat in total.
public void onBeginGroup(final Token token, final int groupIndex, final int numInGroup)
{
namedScope.push(token.name() + ".");
}
public void onEndGroup(final Token token, final int groupIndex, final int numInGroup)
{
namedScope.pop();
}Note: Repeating groups can nest so it is necessary to be prepared to handle this scope recursively.
At the end of a message it is possible to encode variable length strings or binary blobs. Strings are binary data that uses a schema defined character encoding.
public void onVarData(
final Token fieldToken, final DirectBuffer buffer, final int bufferIndex, final int length, final Token typeToken)
{
final String value;
try
{
buffer.getBytes(bufferIndex, tempBuffer, 0, length);
value = new String(tempBuffer, 0, length, typeToken.encoding().characterEncoding());
}
catch (final UnsupportedEncodingException ex)
{
ex.printStackTrace();
return;
}
printScope();
out.append(fieldToken.name())
.append('=')
.append(value)
.println();
}