Android fetch hangs indefinitely with IPv6 hosts on some devices (Happy Eyeballs)

[andreialecu]

Description

This is a very bizarre issue that has been previously reported a bunch of times, and this is basically a continuation of:

#29608

I initially started running into this on RN 0.66 with AWS Cognito. Bumping to 0.66.3 didn't help.

I'm also pretty sure this used to work before and I'm not sure when it broke. It's on an app that has been shelved for a while.

The problem is very strange because the network request does not seem to be issued, but simply hitting CMD+S to save any file so that a hot-reload is issued will immediately dispatch the network request.

I discovered the promise hanging issue by adding some logs to the fetch calls the cognito library was doing:

Notice how the .then is not executed.

While troubleshooting I came across a mention here of a workaround: #29608 (comment) (courtesy of @danmaas) which seems to completely resolve the issue.

Here's the same .then correctly being executed after applying that patch:

Version

0.66.3

Output of react-native info

System:
    OS: macOS 12.0.1
    CPU: (8) arm64 Apple M1
    Memory: 142.27 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.13.0 - /private/var/folders/9p/k1yqxx0d7rn1nlztg_wm7sbw0000gn/T/xfs-2dcae145/node
    Yarn: 2.4.0-git.20210330.hash-ebcd71d5 - /private/var/folders/9p/k1yqxx0d7rn1nlztg_wm7sbw0000gn/T/xfs-2dcae145/yarn
    npm: 7.20.1 - ~/.nvm/versions/node/v16.13.0/bin/npm
    Watchman: 2021.11.01.00 - /opt/homebrew/bin/watchman
  Managers:
    CocoaPods: 1.11.0 - /Users/andreialecu/.rbenv/shims/pod
  SDKs:
    iOS SDK:
      Platforms: DriverKit 21.0.1, iOS 15.0, macOS 12.0, tvOS 15.0, watchOS 8.0
    Android SDK: Not Found
  IDEs:
    Android Studio: 4.2 AI-202.7660.26.42.7351085
    Xcode: 13.1/13A1030d - /usr/bin/xcodebuild
  Languages:
    Java: 11.0.8 - /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/javac
  npmPackages:
    @react-native-community/cli: Not Found
    react: 17.0.2 => 17.0.2 
    react-native: 0.66.3 => 0.66.3 
    react-native-macos: Not Found
  npmGlobalPackages:
    *react-native*: Not Found

Steps to reproduce

I'm able to reproduce it with this:

  React.useEffect(() => {
    console.log('confirm start', new Date());
    fetch('https://cognito-idp.eu-west-1.amazonaws.com/', {
      method: 'POST',
      mode: 'cors',
    })
      .then(() => console.log('then', new Date()))
      .catch(() => console.log('catch', new Date()));
    setTimeout(() => {
      console.log('5 seconds passed');
    }, 5000);
  }, []);

Output:

After applying #29608 (comment):

Snack, code example, screenshot, or link to a repository

No response

Skip to this comment for the actual cause: #32730 (comment)

[andreialecu]

Note that this doesn't repro on the emulator, this is the output on the emulator:

 LOG  confirm x 2021-12-09T18:44:08.229Z
 LOG  then x 2021-12-09T18:44:08.607Z
 LOG  5 seconds passed

❗ The device where this is happening, and where I initially noticed it on is an Oppo A72 (CPH2067) running Android 11

[andreialecu]

I'm also not able to reproduce it on a Samsung Galaxy Tab A (SM-T290) device.

[danmaas]

In my case, this issue also affected only some devices. I think it has something to do with the network stack or driver on the device. Or how they interact with IPv6 devices out on the network.

[andreialecu]

Interesting. I left it running for a while and it actually seems to time out on the connection:

 LOG  confirm x 2021-12-09T19:00:10.557Z
 LOG  5 seconds passed
 LOG  then x 2021-12-09T19:04:23.177Z

It took 4 minutes to time out.

I just noticed that the device does not have IPv6 connectivity at all based on https://test-ipv6.com/

Might be a router problem on my end, usually I have IPv6. However, I'm not sure why it would even attempt to use an IPv6 address considering it does not have IPv6 connectivity at all.

[andreialecu]

Very interesting, I rebooted the router and now I have IPv6 again.

❗ The Oppo now no longer has the error. It works perfectly:

 LOG  confirm x 2021-12-09T19:28:07.714Z
 LOG  then x 2021-12-09T19:28:09.217Z
 LOG  5 seconds passed

I'm not sure what to make of this. 🤔

[vamper424]

Yo mismo me encargo de resolver lo de sus vidas, ni hay ni habrá problema alguno, temor alguno o alguien que me agreda saludo sigan igual y cada vez mejoren más, ok. Si se puede. Atte: Jesus Francisco Urias García.

[vamper424]

Y nuevamente tu gol, esta vez artificial.

[vamper424]

Jajaja mediocres ellos, sobre ellos.

[andreialecu]

The root cause of this issue seems to be the lack of "Happy Eyeballs" in the underlying okhttp library that react-native is using on Android.

Here's the issue in okhttp that is tracking this:
square/okhttp#506

I think most of the random networking Android slowdowns reported all over the ecosystem are probably related. This is the one I ran into in particular:
aws-amplify/amplify-js#5539

Possibly related issues:
#32467
#29782

This thread also mentions this exact same issue:
#28283 (comment)

I believe a PR to implement Happy Eyeballs into okhttp shouldn't be too hard, if someone would be able to contribute one.

From my understanding a TCP connection to both the IPv6 and IPv4 address needs to be attempted, and the first one to respond is used, while the other one is closed.

This would solve this ubiquitous Android networking problem once and for all.

[marcesengel]

@andreialecu would you be so kind to review my PR?

[andreialecu]

@marcesengel I think this should be addressed in the underlying okhttp library. Perhaps you might want to contribute it there instead?

There is some movement there that indicates this problem might finally get some attention.

See:
square/okhttp#7009

[marcesengel]

@andreialecu looking at the old issue regarding this a change on their end seemed unrealistic without a new major release (due to interceptors needing to be raced, see for example this comment, which reads "It's an invasive feature to add because there's extra complexity. If you need it, don't wait for us."), which in turn would mean that it probably takes some time to adapt this to react-native.

I don't see any issue with implementing it for RN in the meantime. I'd have added it to okhttp if it didn't look like it'd take ages to catch on - in case it ever did.

[andreialecu]

I see. That makes sense.

Not sure who would need to take a look at this. I can't review it myself since I'm not familiar with the RN code dealing with this.

Perhaps @yungsters would be able to take a look?

[cortinico]

@marcesengel I think this should be addressed in the underlying okhttp library. Perhaps you might want to contribute it there instead?

Essentially this. Please take a look at my answer here: #33045 (comment) as I believe that's the most feasible solution we could take at this stage.

[marcesengel]

I've built a patched version for use until OkHTTP 5.x is stable (now that since the first of February the implementation of Happy Eyeballs is confirmed). It's available as marcesengel/react-native#0.66-patched-3.

[cortinico]

Cross-posting here as I realized I accidentally left the comment on the PR instead of the Issue: #33045 (comment)

Would you be up for opening a draft PR with the OkHTTP bump to 5.x alpha? It could help us spot early breaking changes and we could test it against the internal infra to make sure nothing breaks.
(That's also a task that someone else in the community can pickup if they wish).

[marcesengel]

I'd be open to do the transition once major version 5 is stable, as for now there's a fix and I wouldn't want to change something multiple times in case more breaking changes are introduced before v5 hits stable.
If somebody else would like to do it beforehand and take the risk in order to make it available faster that of course would be much appreciated 👍

Do you know if there's an ETA for the stable release?

[cortinico]

Nope but you can subscribe here to get updates

• Happy Eyeballs square/okhttp#506

[hadeskers]

Add below dependencies to android/app/build.gradle to force use okhttp 5.0.0-alpha.11

  dependencies {
        // others dependencies 
  
        implementation 'com.squareup.okhttp3:logging-interceptor:5.0.0-alpha.11'
        implementation 'com.squareup.okhttp3:okhttp:5.0.0-alpha.11'
        implementation 'com.squareup.okhttp3:okhttp-urlconnection:5.0.0-alpha.11'
  }

[lukawolf]

If you are using expo and your project is managed, you can not edit android/app/build.gradle yourself. Since I've not found any expo plugin, which would make the changes for you, I just made one: expo-android-app-gradle-dependencies

[JKKholmatov]

@hadeskers @lukawolf @andreialecu @cortinico I created an Expo app and run expo prebuild, how should I implement okhttp:5.0.0-alpha.11 version? is there any instruction? please help!

[JKKholmatov]

Add below dependencies to android/app/build.gradle to force use okhttp 5.0.0-alpha.11

  dependencies {
        // others dependencies 
  
        implementation 'com.squareup.okhttp3:logging-interceptor:5.0.0-alpha.11'
        implementation 'com.squareup.okhttp3:okhttp:5.0.0-alpha.11'
        implementation 'com.squareup.okhttp3:okhttp-urlconnection:5.0.0-alpha.11'
  }

It did not work for me, is there any other way to implement it?

[irisjae]

Add below dependencies to android/app/build.gradle to force use okhttp 5.0.0-alpha.11

  dependencies {
        // others dependencies 
  
        implementation 'com.squareup.okhttp3:logging-interceptor:5.0.0-alpha.11'
        implementation 'com.squareup.okhttp3:okhttp:5.0.0-alpha.11'
        implementation 'com.squareup.okhttp3:okhttp-urlconnection:5.0.0-alpha.11'
  }

May I ask, having applied this, it is necessary to further modify the OkHttpClientProvider by adding .fastFallback(true) square/okhttp#506 (comment) , and setting React Native to build from source https://reactnative.dev/contributing/how-to-build-from-source#update-your-project-to-build-from-source ?

[yschimke]

fastFallback is the default in alpha.11 already

https://github.com/square/okhttp/blob/2344b87c59456088ed1346221d45648634f7f0c8/CHANGELOG.md?plain=1#L12C1-L12C1

[irisjae]

Strange, I'm still having issues... Fetch with IP works, but not with domain names. Let me look further.

[irisjae]

I've gotten to the bottom of my issue (hopefully), leaving it here in case it helps anyone. As I reported, after applying the Happy Eyeballs patch, I still had issues.

Fetching with IP seemed to be more reliable than with domain names. Still, with more systematic testing, I found that even when fetching via IP, my app exhibited stochastic hanging, just less frequently. Still seeing stochasticity, I thought to adjust the connection pool, so I set my app to build from source for Android, and modified OkHttpClientProvider. I knew that OkHttp kept some kind of connection pool, and assumed it was fixed in size. As my app had some long-running HTTP requests, I thought that OkHttp was waiting for the long-running requests to complete before allowing my new requests to use them from the pool. Assuming 'maxIdleConnections' was the number of connections kept in the pool, I turned the connections up. It was to no avail.

Reading more about the connection pool, it seems like instead of keeping a fixed number of connections, OkHttp tries to keep the number of connections close to Number of connections in use + Number of idle connections. It seems that OkHttp tries to somehow place 'similar' requests on to a shared connection to minimise time reestablishing connection, but on the other hand square/okhttp#4354 (comment), sharing may cause failures.

I thought that perhaps with long-running requests in my app, I would be better served with no connection pool. This can be effectively done by setting 'maxIdleConnections' to 0.

My randomly hanging fetches ceased.

Edit: Attempting to disable the Connection Pool did not eliminate the issue.

As of now, I believe that the culprit is OkHttp's connection sharing somehow breaking things. For now, I removed my patch on OkHttpClientProvider.java, and instead applied a patch on NetworkingModule.java, which violently creates a separate OkHttpClient for each request and destroys the client after use (against all the advice of OkHttp), to eliminate any connection sharing. Nothing seems to hang randomly anymore so far (hopefully thats it...).

i have to say, it frustrates me greatly that the network stack on React Native Android is causing me so much trouble and tinkering with network code, when all I'm using is plain fetches with JSON. I wonder if some of the default configurations should need further improvement for React Native Android - contrasting with the networking on iOS, which has worked flawlessly.

Edit: For the record, both disabling the connection pool and violently preventing connection sharing were crucial to making network connections work reliably on low-end devices. Removing either of the patches reintroduces the issue.

[yschimke]

OkHttp 5 has a connection listener you can configure on the pool. It should expose all this so it's not surprising.

[irisjae]

How might the connection listener help?

[yschimke]

Maybe not solve the problem. But let you see what's going on, when connections are reused, released, fail.

[consuelo-sanna]

I have this sorted out in my projects.

Still, I think it needs to be prioritized because almost all React Native Android apps will run into this problem sooner or later, and it's not an obvious issue.

As I already mentioned above, there is nothing to handle here.

I think something should be done about it to at least make it obvious to the community that their Android apps will not work consistently 100% of the time. See all the threads I linked, and the Reddit story.

Good morning, my project bump on this issue since Android 14 officially come out and it took me one week before being able to connect the dots and realize it was an IPv6 problem and end up here, so I totally agree that this it's not an obvious issue and should be at least pointed out somewhere in the documentation (if there is, I could not find it) or on troubleshooting/common issues.
The app I work on is for US people only, so in order for me to test it I need to go through a VPN and this made me impossible reproduce the issue since the one I chose does not support IPv6 and this made it difficult to debug

Since Android 14, every user of our app based in the USA could not use the app since the first api call instead of taking less of a second it took more than 2 minutes, I have a reproducible repo with latest RN version here if you are interested.

In order to reproduce the issue, the only thing you have to do is use an Android 14 phone with data connection and an ISP provider that supports IPv6, Verizon and AT&T for example

The only thing that solved it, was leveraging okhttp3 okhttp:5.0.0-alpha.12 (alpha 11 did not solved for us) OR patching following danmaas's example here https://gist.github.com/danmaas/c60af5fed9f55d2bc616ce302696540d

I really think this issue should be more visible, especially since Android 14 is out now and I think more and more people/apps will start experiencing this, let me know if there's something I can do to help with that, I was not sure opening a new issue on the repo was a good idea or not since there are already quite a few on the argument, like:

#28022
#28038
#29608

[shubhkhanna]

For me it solved the problem for IPv6 support on android.

dependencies {
      // others dependencies 

      // define a BOM and its version
      implementation(platform("com.squareup.okhttp3:okhttp-bom:5.0.0-alpha.12"))

      // define any required OkHttp artifacts without version
      implementation("com.squareup.okhttp3:okhttp")
      implementation("com.squareup.okhttp3:logging-interceptor")
      implementation("com.squareup.okhttp3:okhttp-urlconnection")
}

Add following dependencies to android/app/build.gradle to force use okhttp 5.0.0-alpha.12

"react": "18.2.0",
"react-native": "0.71.11"

[react-native-bot]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[JKKholmatov]

@shubhkhanna

For me it solved the problem for IPv6 support on android.

How to implement this in Expo project?

[shubhkhanna]

@shubhkhanna

For me it solved the problem for IPv6 support on android.

How to implement this in Expo project?

@JKKholmatov never tried in the expo project. These changes are for bare react-native project.

[whybtech]

@JKKholmatov
I used a custom plugin with expo/config-plugins and withAppBuildGradle, here is an example:

• Create a custom plugin file (e.g. internal/expo-config-plugins/appBuildGradleDependencies.js):

const { withAppBuildGradle } = require('@expo/config-plugins');

module.exports = withAppBuildGradleDependencies = (config, customName) => {
  return withAppBuildGradle(config, (config) => {
    const initialIndex = config.modResults.contents.indexOf("dependencies {");

    config.modResults.contents =
      config.modResults.contents.slice(0, initialIndex) +
      `dependencies {
    implementation 'com.squareup.okhttp3:logging-interceptor:5.0.0-alpha.11'
    implementation 'com.squareup.okhttp3:okhttp:5.0.0-alpha.11'
    implementation 'com.squareup.okhttp3:okhttp-urlconnection:5.0.0-alpha.11'` +
      config.modResults.contents.slice(initialIndex + "dependencies {".length);

    return config;
  });
};

• Declare the plugin in app.json:

"plugins": [
      "./internal/expo-config-plugins/appBuildGradleDependencies.js",

• Re-build your app

It resolved my issue with ipv6 latency on Android

[khakobyan]

Switching to OkHttp version 5 did not resolve the issue. However, I found a solution by creating a custom module for API calls that only functions when running on Android and the required VPN is active. Below is the implementation:

package <com.yourpackage.test>

import android.net.TrafficStats
import com.facebook.react.bridge.Promise
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.bridge.ReactContextBaseJavaModule
import com.facebook.react.bridge.ReactMethod
import okhttp3.OkHttpClient
import okhttp3.Request
import okhttp3.RequestBody
import okhttp3.MediaType.Companion.toMediaTypeOrNull
import okhttp3.Headers.Companion.toHeaders
import okhttp3.Dns
import okhttp3.Response
import java.io.IOException
import java.net.Inet4Address
import com.facebook.react.modules.core.DeviceEventManagerModule
import com.facebook.react.bridge.Arguments

class OkHttpModule(reactContext: ReactApplicationContext) : ReactContextBaseJavaModule(reactContext) {

    override fun getName(): String {
        return "OkHttpModule"
    }

    @ReactMethod
    fun fetchFromApi(url: String, headersString: String, body: String, promise: Promise) {
        // Create OkHttpClient with IPv4-only DNS resolution
        val client = OkHttpClient.Builder()
            .dns { hostname ->
                // Resolve only IPv4 addresses (filter out IPv6 addresses)
                Dns.SYSTEM.lookup(hostname).filter { it is Inet4Address }
            }
            .build()

        // Tag the thread for traffic stats tracking
        TrafficStats.setThreadStatsTag(10042)

        try {
            // Convert headers from a string to OkHttp Headers
            val headers = parseHeaders(headersString).toHeaders()

            // Create the request body
            val requestBody = RequestBody.create("application/x-www-form-urlencoded".toMediaTypeOrNull(), body)

            // Build the request
            val request = Request.Builder()
                .url(url)
                .headers(headers)
                .post(requestBody)
                .build()

            // Execute the request
            val response: Response = client.newCall(request).execute()
            val responseData = response.body?.string() ?: ""

            // Send result back to JS for logging
            sendEventToJS(reactApplicationContext, url, responseData, headers.toMap())

            if (!response.isSuccessful) {
                promise.reject("Error", "Request failed with code: ${response.code}")
            } else {
                promise.resolve(responseData)
            }

        } catch (e: IOException) {
            promise.reject("Error", "Request failed: ${e.message}")
        } finally {
            // Clear the traffic stats tag after the network operation is complete
            TrafficStats.clearThreadStatsTag()
        }
    }

    private fun sendEventToJS(reactContext: ReactApplicationContext, url: String, response: String, headers: Map<String, String>) {
        val params = Arguments.createMap()
        params.putString("url", url)
        params.putString("response", response)
        params.putString("headers", headers.toString())

        reactContext.getJSModule(DeviceEventManagerModule.RCTDeviceEventEmitter::class.java)
            .emit("networkResponse", params)
    }

    // Helper method to parse headers from "Key: Value\nKey: Value" format into a map
    private fun parseHeaders(headersString: String): Map<String, String> {
        val headersMap = mutableMapOf<String, String>()
        val headersArray = headersString.split("\n")
        for (header in headersArray) {
            val keyValue = header.split(": ")
            if (keyValue.size == 2) {
                headersMap[keyValue[0]] = keyValue[1]
            }
        }
        return headersMap
    }}

This module ensures the API calls work specifically with IPv4, mitigating connectivity issues observed when using the VPN.

[yschimke]

I'd be interested to understand why it breaks on VPN.

If you configure an event listener in the client, you should see what it's doing with DNS and connection attempts.

This should work without this workaround, so it would be good to find the root cause.

https://square.github.io/okhttp/features/events/

[khakobyan]

@yschimke

The issue is actually rooted in how Android handles connections with the Happy Eyeballs algorithm, which tries both IPv4 and IPv6 connections to improve speed. On certain devices, though, this approach can cause requests to hang indefinitely when there’s an IPv6 host involved—especially problematic when using a VPN that primarily supports IPv4.

To work around this, I created a custom module that forces OkHttp to resolve only IPv4 addresses, which has been effective in preventing the hang. Ideally, we’d look deeper into how Android handles the Happy Eyeballs fallback in VPN contexts and see if we can prevent this hang without restricting to IPv4.

[yschimke]

OkHttp tries to

• resolve DNS for IPv4+IPv6
• open a socket to the first IPv6 address
• after a delay, open a socket to the first IPv4 address

Which section is hanging with a VPN?

[khakobyan]

@yschimke
The hang isn’t actually with OkHttp itself but with fetch in JavaScript, which stalls on IPv6 with some VPN setups on Android. We’re using OkHttp in a custom module to replace fetch on Android, forcing IPv4 resolution to bypass this issue. This could technically be done with other libraries, but OkHttp makes managing DNS settings straightforward and effective here.