Best / most efficient way to write a Binary to a OutputStream?

[javafanboy]

I am working on something where I need to "stream" Binary objects (retrieved from BinaryEntry.getBinaryKey & Value) to normal Java OutputStream and at a later time read these binaries back using an InputStream.

I have successfully written and restored Binary objects by wrapping the streams in ObjectStreams but I assume this writes type information etc. that is not needed since I know only Binary objects are written and makes the data to process larger (in particular bad if the Binary objects are quite small as the overhead proportionally becomes large).

I have experimented with using the writeTo(outputstream) method that seem to write data but the question is what to use to read that data back? I tried the Binary.readBinary(inputStream) and Binary,read(inputStream) but neither of them worked (both read zero bytes producing an empty Binary object)when passed the written data :-( What is the difference between them??

Before trying to dive into the source code I am putting the question here to see if I can get some advice....

Also in order to play around with Binary objects what is the simplest way to, without using a Cache, create one by POF serializing some Java object? If I could figure out how do this it would be a lot easier for me to try out serializing/de-serializing etc...

[rlubke]

I have experimented with using the writeTo(outputstream) method that seem to write data but the question is what to use to read that data back? I tried the Binary.readBinary(inputStream) and Binary,read(inputStream) but neither of them worked (both read zero bytes producing an empty Binary object)when passed the written data :-( What is the difference between them??

I'm not able to replicate this with Binary.readBinary(InputStream), perhaps I'm missing something on my end. Also, I don't see any Binary.read(InputStream); was it on another class? Perhaps you can share the code?

Also in order to play around with Binary objects what is the simplest way to, without using a Cache, create one by POF serializing some Java object? If I could figure out how do this it would be a lot easier for me to try out serializing/de-serializing etc...

Here's a simple example that round-trips a single String using POF.

import com.tangosol.io.pof.ConfigurablePofContext;
import com.tangosol.util.Base;
import com.tangosol.util.Binary;
import com.tangosol.util.ExternalizableHelper;

class Scratch1
    {
    public static void main(String[] args)
            throws Exception
        {
        ConfigurablePofContext ctx    = new ConfigurablePofContext("coherence-pof-config.xml");
        String                 sHello = "Hello World!";
        Binary                 b      = ExternalizableHelper.toBinary(sHello, ctx);

        System.out.println(Base.toHexDump(b.toByteArray(), 16));

        String sResult = ExternalizableHelper.fromBinary(b, ctx);

        System.out.println(sResult);
        }
    }

To your original question, your best bet may be rolling a set of stream implementations where the writer, when writing the Binary, will write the length of the binary bytes followed by the binary bytes. And reverse the process on the read side. I have a quick/dirty example I can share if you'd like.

[javafanboy]

Thanks for your reply Ryan! I had all but forgot about the trusty ExternalizableHelper - will make this type of experiments a lot simpler for sure! I worked on a quite deep level with Coherence some 15+ years ago (even had multiple meetings with Gene and Cameron back in the days of Tangosol) but have since done others things and just recently been back working with Coherence again (much thanks to the emergence of CE). I am trying to get back to speed so please share the example you talked about!

…

On Fri, Jul 19, 2024 at 6:02 AM Ryan Lubke ***@***.***> wrote: I have experimented with using the writeTo(outputstream) method that seem to write data but the question is what to use to read that data back? I tried the Binary.readBinary(inputStream) and Binary,read(inputStream) but neither of them worked (both read zero bytes producing an empty Binary object)when passed the written data :-( What is the difference between them?? I'm not able to replicate this with Binary.readBinary(InputStream), perhaps I'm missing something on my end. Also, I don't see any Binary.read(InputStream); was it on another class? Perhaps you can share the code? Also in order to play around with Binary objects what is the simplest way to, without using a Cache, create one by POF serializing some Java object? If I could figure out how do this it would be a lot easier for me to try out serializing/de-serializing etc... Here's a simple example that round-trips a single String using POF. import com.tangosol.io.pof.ConfigurablePofContext;import com.tangosol.util.Base;import com.tangosol.util.Binary;import com.tangosol.util.ExternalizableHelper; class Scratch1 { public static void main(String[] args) throws Exception { ConfigurablePofContext ctx = new ConfigurablePofContext("coherence-pof-config.xml"); String sHello = "Hello World!"; Binary b = ExternalizableHelper.toBinary(sHello, ctx); System.out.println(Base.toHexDump(b.toByteArray(), 16)); String sResult = ExternalizableHelper.fromBinary(b, ctx); System.out.println(sResult); } } To your original question, your best bet may be rolling a set of stream implementations where the writer, when writing the Binary, will write the length of the binary bytes followed by the binary bytes. And reverse the process on the read side. I have a quick/dirty example I can share if you'd like. — Reply to this email directly, view it on GitHub <#124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADXQFZXW7GMQ5H4NA4UCH3ZNCFXBAVCNFSM6AAAAABIFULPCWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBZGEZTIMI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[rlubke]

Here you go. As I said, it's rough, but it does the basics.

import com.tangosol.io.pof.ConfigurablePofContext;

import com.tangosol.util.Base;
import com.tangosol.util.Binary;
import com.tangosol.util.ExternalizableHelper;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Iterator;
import java.util.NoSuchElementException;

class Scratch3
    {
    public static void main(String[] args)
            throws IOException
        {
        ConfigurablePofContext ctx = new ConfigurablePofContext("coherence-pof-config.xml");

        Binary b  = ExternalizableHelper.toBinary("Hello World", ctx);
        Binary b1 = ExternalizableHelper.toBinary(Long.MAX_VALUE, ctx);
        Binary b2 = ExternalizableHelper.toBinary(Float.MAX_VALUE, ctx);

        ByteArrayOutputStream baos      = new ByteArrayOutputStream();
        BinaryOutStream       outStream = new BinaryOutStream(baos);

        outStream.write(b);
        outStream.write(b1);
        outStream.write(b2);

        baos.flush();
        baos.close();

        byte[] written = baos.toByteArray();

        System.out.println(Base.toHexDump(written, 16));

        BinaryInputStreamIterator in = new BinaryInputStreamIterator(new ByteArrayInputStream(written));
        while (in.hasNext())
            {
            System.out.println((Object) ExternalizableHelper.fromBinary(in.next(), ctx));
            }
        }

    public static class BinaryOutStream
        {
        private OutputStream out;

        public BinaryOutStream(OutputStream out)
            {
            this.out = out;
            }

        public void write(Binary binary)
                throws IOException
            {
            encodeLen(binary.length());
            binary.writeTo(out);
            }

        public void encodeLen(int len)
                throws IOException
            {
            out.write(len >> 24);
            out.write(len >> 16);
            out.write(len >> 7);
            out.write(len);
            }
        }

    public static class BinaryInputStreamIterator
            implements Iterator<Binary>
        {
        private InputStream in;

        public BinaryInputStreamIterator(InputStream in)
            {
            this.in = in;
            }

        public Binary next()
                throws NoSuchElementException
            {
            try
                {
                return Binary.readBinary(in, decodeLen());
                }
            catch (IOException ioe)
                {
                throw new NoSuchElementException(ioe);
                }
            }

        public boolean hasNext()
            {
            try
                {
                return in.available() != 0;
                }
            catch (IOException e)
                {
                throw Base.ensureRuntimeException(e);
                }
            }

        protected int decodeLen()
                throws IOException
            {
            return ((in.read() & 0xff) << 24) |
                   ((in.read() & 0xff) << 16) |
                   ((in.read() & 0xff) << 8) |
                   (in.read() & 0xff);
            }
        }
    }

[rlubke]

Also, just wanted to share that you can join our public slack here: https://oraclecoherence.slack.com/archives/CHHPV946N .

[rlubke]

Hey Javafanboy, just wanted to follow up. Was the example along the lines of what you were looking for? Maybe I misunderstood the general ask?

[javafanboy]

Been doing some more urgent stuff for a while but now back to "coherence programming" again. Your example was very helpful for playing around with the various ways to serialize / deserialize Binary objects.

I have not been able to "prove it" with an isolated example (maybe I can figure out a way eventually?) but I have with 100% repeatability in my code for snapshotting back cache partitions to S3 found that "Binary.readBinary(inputStream, decodeLength(inputStream))" don't behave correctly with the Stream I get from the AWS Java SDKv2. I have not fully understood in what way it does not work but it reads to little data my hypothesis is that it throws EOFException (and when this happens possibly also closes the stream?!) even though there is still data left to read in the stream.

What I can think of that is different with the AWS stream compared to your example is that it will block quite a lot (while more data is "streamed" from S3) and I therefore suspects the readBinary method does not handle this correctly. In your example you are for instance using "available" to see if there is more data to read (if I understand the code right?) but according to my understanding this method only return how many bytes that can be read WITHOUT BLOCKING and can return zero way before the stream is empty and if the same logic is used in "readBinary" it could for instance explain the behaviour - I tried looking at the source code but I could not right away see anything wrong.

My "work around" is right now to instead of the problematic method use "new Binary(inputStream.readNBytes(decodeLength(inputStream)))" that works perfectly 100% of the time and is functionally equivalent (maybe less efficient as a temporary byte[] is created).

The corresponding method for writing a Binary to a stream ("binaryKey.writeTo(outputStream)") works flawlessly.

My previous workaround was to use ObjectOutput/ObjectInput streams (making use of the Binary objects externalizable support) but, at least with the small objects I insert in the cache for testing, this produced ~30% larger S3 files (also taking about that much longer to both store and read as this is mostly i/o bound) than the new one described above.

[rlubke]

Would it be possible for you to provide a small reproducer I can setup with S3 and run on my end to diagnose? Or is this as simple as trying to read a large stream from S3 would be sufficient to reproduce?