Out of memory - any way to optimize?

[rexxars]

Hi there!

I'm creating a fairly simple Android-app that simply displays some text based on EventSource messages. It runs on some fairly old and memory limited devices; the one in question has a total of 288MB of memory available.

The app runs for a while, but after a while it runs out of memory. There is rarely any messages published on the EventSource, but every 15 seconds we send a ping to keep the socket alive, which is basically only a :\n\n sequence. So our stream looks something like this:

event: welcome
data: {"listenerName": "XS4itD0DURQaIrsFBz8a40"}

:

:

:

[more ping events here]

event: display
data: Hello there.

I'm curious as to whether there is any way to optimize the code for a lower memory footprint, or if this is a bug somehow.

A potential workaround might be closing and reopening the connection every once in a while, but I'm not sure it solves the issue, and it also feels like a hack.

Any ideas?

Full stack trace follows:

08-04 13:56:48.145 1002-1109/? E/dalvikvm-heap: Out of memory on a 8208-byte allocation.
08-04 13:56:48.145 1002-1109/? I/dalvikvm: "okhttp-eventsource-stream-[]-0" prio=5 tid=60 RUNNABLE
08-04 13:56:48.145 1002-1109/? I/dalvikvm:   | group="main" sCount=0 dsCount=0 obj=0x41679f48 self=0x2cb308
08-04 13:56:48.145 1002-1109/? I/dalvikvm:   | sysTid=1109 nice=0 sched=0/0 cgrp=default handle=2796856
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.Segment.<init>(Segment.java:~63)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.SegmentPool.take(SegmentPool.java:48)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.Segment.split(Segment.java:126)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.Buffer.write(Buffer.java:1243)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.Buffer.read(Buffer.java:1270)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.RealBufferedSource.read(RealBufferedSource.java:51)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:429)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.RealBufferedSource.read(RealBufferedSource.java:46)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at com.launchdarkly.eventsource.EventSource.connect(EventSource.java:195)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at com.launchdarkly.eventsource.EventSource.access$800(EventSource.java:35)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at com.launchdarkly.eventsource.EventSource$1.run(EventSource.java:102)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1088)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:581)
08-04 13:56:48.145 1002-1109/? I/dalvikvm:     at java.lang.Thread.run(Thread.java:1019)

[jkodumal]

@rexxars Our main use case for this library does something very similar-- we use heartbeats sent every 3 minutes. We have not encountered this problem-- but we'd be happy to help investigate. Can you share the version of the library you're using, as well as a heap dump if you can produce one?

[rexxars]

I can't seem to reproduce the problem on the Android emulator, which is annoying. It seems to be garbage collecting memory just fine.

I'm using:

compile 'com.squareup.okhttp3:okhttp:3.8.1'
compile 'com.launchdarkly:okhttp-eventsource:1.4.1'

I'll try updating to okhttp-eventsource 1.5.0, though it shouldn't make any difference judging by the commits.

[jkodumal]

@rexxars keep us posted if you discover more. We have still not encountered this issue in our own usage.

[rexxars]

Out of interest, are you using this over HTTP or HTTPS? HTTP2?

I'm encountering this issue over HTTPS with HTTP2, not sure if that makes any difference.
Not too familiar with Java, but seeing a lot of these, which isn't stopping it from connecting:
09-01 02:58:58.783 3050-23749/codes.espen.digit D/NativeCrypto: returned from sslSelect() with result 1, error code 2

When it crashes, I'm seeing two different OutOfMemoryErrors. One being the one I mentioned in the original post, and the other is:

09-01 02:58:58.342 3050-3070/codes.espen.digit E/AndroidRuntime: FATAL EXCEPTION: Thread-11
 java.lang.OutOfMemoryError: (Heap Size=32775KB, Allocated=29814KB, Bitmap Size=0KB)
     at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:91)
     at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:216)
     at org.bouncycastle.asn1.DEROutputStream.write(DEROutputStream.java:104)
     at org.bouncycastle.asn1.DEROutputStream.writeEncoded(DEROutputStream.java:50)
     at org.bouncycastle.asn1.DERSequence.encode(DERSequence.java:78)
     at org.bouncycastle.asn1.DEROutputStream.writeObject(DEROutputStream.java:127)
     at org.bouncycastle.asn1.ASN1Encodable.getEncoded(ASN1Encodable.java:48)
     at org.bouncycastle.jce.provider.X509CertificateObject.getEncoded(X509CertificateObject.java:531)
     at org.bouncycastle.jce.provider.X509CertificateObject.calculateHashCode(X509CertificateObject.java:585)
     at org.bouncycastle.jce.provider.X509CertificateObject.hashCode(X509CertificateObject.java:574)
     at java.util.HashMap.put(HashMap.java:394)
     at java.util.HashSet.add(HashSet.java:95)
     at okhttp3.internal.tls.TrustRootIndex$BasicTrustRootIndex.<init>(TrustRootIndex.java:115)
     at okhttp3.internal.tls.TrustRootIndex.get(TrustRootIndex.java:48)
     at okhttp3.internal.tls.TrustRootIndex.get(TrustRootIndex.java:43)
     at okhttp3.internal.platform.Platform.buildCertificateChainCleaner(Platform.java:167)
     at okhttp3.internal.platform.AndroidPlatform.buildCertificateChainCleaner(AndroidPlatform.java:173)
     at okhttp3.internal.tls.CertificateChainCleaner.get(CertificateChainCleaner.java:41)
     at okhttp3.OkHttpClient.<init>(OkHttpClient.java:251)
     at okhttp3.OkHttpClient.<init>(OkHttpClient.java:223)
     at com.launchdarkly.eventsource.EventSource$Builder.<init>(EventSource.java:307)
     at codes.espen.digit.MainActivity$SseRunnable.buildAndStart(MainActivity.java:309)
     at codes.espen.digit.MainActivity$SseRunnable.updateStateIfNecessary(MainActivity.java:332)
     at codes.espen.digit.MainActivity$SseRunnable.run(MainActivity.java:342)
     at java.lang.Thread.run(Thread.java:1027)

After the crashing, I'm seeing one of these:

09-01 02:59:03.067 3050-23989/codes.espen.digit W/OkHttp: A connection to https://<url>/ was leaked. Did you forget to close a response body?

It might be interesting to note that this doesn't happen for quite a while, we're talking hours of just receiving ping-replies and reconnecting every 30 minutes (server closes connection, client reconnects). I'm also curious if this could be related to the way the load balancer forcefully terminates the connection. The transfer encoding used is chunked, and it expects the stream to end with a zero-byte chunk. However, the load balancer is not aware of this fact and just forcefully closes after 30 minutes. I'm pondering if somehow this could cause connections to remain open or leak memory.

I'm going to try reproducing the issue with a real device against a local HTTP endpoint, and might also take the time to see if I can create a server replicating the load balancer cut-off behaviour.

[rexxars]

Sorry, this was all on my end. When the eventsource closed, I had written some faulty code that triggered a new EventSource to be instantiated for every disconnect. With a 30 minute timeout, this took quite a while to build up to a out of memory error, but locally with a 15 second timeout the error was much easier to spot.

Sorry about this, thanks for a great library!