Segmentation fault when connecting to RDS #555
Comments
|
Can you please post the output of the segfault - most notably the ruby and C stack trace? How often does the segfault appear? Is it deterministic? Can you provide a sample script for reproduction? |
|
We started having this issue today with our test automation suite as well (it was working last week). I've also tried setting Details:
I can get this to reproduce 100% with our project whenever we try to connect to the DB. I was able to narrow it down to this function here: The error message we get back: I tried wrapping it in a rescue, but it always bombs out before a more helpful error message is returned. I'm uncomfortable setting breakpoints this deep, but hopefully, this information helps. EDIT: One more thing to add (not sure if applicable) - our test automation suite is still running smoothly in our GitLab CI process (I assume they use Linux for their boxes), it is only having issues locally on my machine |
|
Oh thank god I'm not the only one, it's been a frustrating few days. 😄 I will try to build a small reproduction this morning. |
|
Ok here are details to reproduce. Test script: #!/usr/bin/env ruby
require 'bundler/inline'
PG_VERSION = ENV['PG_GEM_VERSION'] || '1.5.4'
gemfile do
source 'https://rubygems.org'
gem 'pg', PG_VERSION
end
# Expects environment variables PGHOST, PGUSER, PGPASSWORD, PGDATABASE etc.
PG.connect do |conn|
result = conn.exec('select 1 as one')
puts result[0]
endI've used this script with Ruby 3.1, 3.2, and 3.3, and all of them crash 100% reliably when connecting to RDS. The script never crashes when connecting to local Postgres instances (tested against latest releases of Postgres 13 through 15). stderr outputStdout just says Crash Report log file{
"app_name": "ruby",
"timestamp": "2024-01-03 11:18:39.00 +0100",
"app_version": "",
"slice_uuid": "f8a4348a-d09a-36e8-a156-529550909f41",
"build_version": "",
"platform": 1,
"share_with_app_devs": 0,
"is_first_party": 1,
"bug_type": "309",
"os_version": "macOS 14.2.1 (23C71)",
"roots_installed": 0,
"incident_id": "6B9B252C-8C52-488D-84CD-9B96D4AFA835",
"name": "ruby"
}
{
"uptime": 190000,
"procRole": "Unspecified",
"version": 2,
"userID": 501,
"deployVersion": 210,
"modelCode": "MacBookPro18,2",
"coalitionID": 5613,
"osVersion": {
"train": "macOS 14.2.1",
"build": "23C71",
"releaseType": "User"
},
"captureTime": "2024-01-03 11:18:39.1286 +0100",
"codeSigningMonitor": 1,
"incident": "6B9B252C-8C52-488D-84CD-9B96D4AFA835",
"pid": 13654,
"translated": false,
"cpuType": "ARM-64",
"roots_installed": 0,
"bug_type": "309",
"procLaunch": "2024-01-03 11:18:38.8530 +0100",
"procStartAbsTime": 4726840308643,
"procExitAbsTime": 4726846859910,
"procName": "ruby",
"procPath": "/Users/USER/*/ruby",
"parentProc": "bash",
"parentPid": 71784,
"coalitionName": "net.kovidgoyal.kitty",
"crashReporterKey": "FCA26E46-4B29-BF3B-0CCB-C240CEBE320B",
"responsiblePid": 4489,
"responsibleProc": "kitty",
"codeSigningID": "ruby",
"codeSigningTeamID": "",
"codeSigningFlags": 570556929,
"codeSigningValidationCategory": 10,
"codeSigningTrustLevel": 4294967295,
"instructionByteStream": {
"beforePC": "fyMD1f17v6n9AwCRXOD/l78DAJH9e8Go/w9f1sADX9YQKYDSARAA1A==",
"atPC": "AwEAVH8jA9X9e7+p/QMAkVHg/5e/AwCR/XvBqP8PX9bAA1/WcAqA0g=="
},
"wakeTime": 855,
"sleepWakeUUID": "12DFE54A-0B08-40B7-8907-C0C93521267E",
"sip": "enabled",
"vmRegionInfo": "0 is not in any region. Bytes before following region: 4374773760\n REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL\n UNUSED SPACE AT START\n---> \n __TEXT 104c1c000-104c20000 [ 16K] r-x/r-x SM=COW ...s/USER/*/ruby",
"exception": {
"codes": "0x0000000000000001, 0x0000000000000000",
"rawCodes": [
1,
0
],
"type": "EXC_BAD_ACCESS",
"signal": "SIGSEGV",
"subtype": "KERN_INVALID_ADDRESS at 0x0000000000000000"
},
"termination": {
"flags": 0,
"code": 11,
"namespace": "SIGNAL",
"indicator": "Segmentation fault: 11",
"byProc": "ruby",
"byPid": 13654
},
"vmregioninfo": "0 is not in any region. Bytes before following region: 4374773760\n REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL\n UNUSED SPACE AT START\n---> \n __TEXT 104c1c000-104c20000 [ 16K] r-x/r-x SM=COW ...s/USER/*/ruby",
"extMods": {
"caller": {
"thread_create": 0,
"thread_set_state": 0,
"task_for_pid": 0
},
"system": {
"thread_create": 0,
"thread_set_state": 0,
"task_for_pid": 0
},
"targeted": {
"thread_create": 0,
"thread_set_state": 0,
"task_for_pid": 0
},
"warnings": 0
},
"faultingThread": 0,
"threads": [
{
"triggered": true,
"id": 2595603,
"threadState": {
"x": [
{
"value": 0
},
{
"value": 0
},
{
"value": 0
},
{
"value": 7994836232,
"symbolLocation": 304,
"symbol": "__sF"
},
{
"value": 20680267530240
},
{
"value": 7726646165504
},
{
"value": 144
},
{
"value": 0
},
{
"value": 881320986018656300
},
{
"value": 881320991311675400
},
{
"value": 1
},
{
"value": 4294967293
},
{
"value": 1099511627776
},
{
"value": 0
},
{
"value": 0
},
{
"value": 0
},
{
"value": 328
},
{
"value": 8146268992
},
{
"value": 0
},
{
"value": 11
},
{
"value": 7994806272,
"symbolLocation": 0,
"symbol": "_main_thread"
},
{
"value": 259
},
{
"value": 7994806496,
"symbolLocation": 224,
"symbol": "_main_thread"
},
{
"value": 4390359040,
"symbolLocation": 5280,
"symbol": "global_enc_table"
},
{
"value": 4966220112
},
{
"value": 4838380880
},
{
"value": 1431634051
},
{
"value": 105553181185232
},
{
"value": 0
}
],
"flavor": "ARM_THREAD_STATE64",
"lr": {
"value": 6543711424
},
"cpsr": {
"value": 1073745920
},
"fp": {
"value": 4924389744
},
"sp": {
"value": 4924389712
},
"esr": {
"value": 1442840704,
"description": " Address size fault"
},
"pc": {
"value": 6543483100,
"matchesCrashFrame": 1
},
"far": {
"value": 0
}
},
"queue": "com.apple.main-thread",
"frames": [
{
"imageOffset": 41180,
"symbol": "__pthread_kill",
"symbolLocation": 8,
"imageIndex": 11
},
{
"imageOffset": 27840,
"symbol": "pthread_kill",
"symbolLocation": 288,
"imageIndex": 12
},
{
"imageOffset": 259392,
"symbol": "raise",
"symbolLocation": 32,
"imageIndex": 13
},
{
"imageOffset": 1835484,
"symbol": "ruby_default_signal",
"symbolLocation": 124,
"imageIndex": 9
},
{
"imageOffset": 645420,
"symbol": "rb_bug_for_fatal_signal",
"symbolLocation": 428,
"imageIndex": 9
},
{
"imageOffset": 1843716,
"symbol": "sigsegv",
"symbolLocation": 96,
"imageIndex": 9
},
{
"imageOffset": 14884,
"symbol": "_sigtramp",
"symbolLocation": 56,
"imageIndex": 14
},
{
"imageOffset": 22536,
"symbol": "closePGconn",
"symbolLocation": 172,
"imageIndex": 2
},
{
"imageOffset": 22536,
"symbol": "closePGconn",
"symbolLocation": 172,
"imageIndex": 2
},
{
"imageOffset": 9296,
"symbol": "PQfinish",
"symbolLocation": 24,
"imageIndex": 2
},
{
"imageOffset": 28540,
"symbol": "pgconn_finish",
"symbolLocation": 152,
"imageIndex": 1
},
{
"imageOffset": 2413280,
"symbol": "vm_call_cfunc_with_frame_",
"symbolLocation": 240,
"imageIndex": 9
},
{
"imageOffset": 2306104,
"symbol": "vm_exec_core",
"symbolLocation": 8264,
"imageIndex": 9
},
{
"imageOffset": 2293864,
"symbol": "rb_vm_exec",
"symbolLocation": 984,
"imageIndex": 9
},
{
"imageOffset": 692232,
"symbol": "rb_ec_exec_node",
"symbolLocation": 160,
"imageIndex": 9
},
{
"imageOffset": 691968,
"symbol": "ruby_run_node",
"symbolLocation": 96,
"imageIndex": 9
},
{
"imageOffset": 16164,
"symbol": "main",
"symbolLocation": 104,
"imageIndex": 10
},
{
"imageOffset": 24800,
"symbol": "start",
"symbolLocation": 2360,
"imageIndex": 15
}
]
},
{
"id": 2595605,
"frames": [
{
"imageOffset": 31168,
"symbol": "kevent",
"symbolLocation": 8,
"imageIndex": 11
},
{
"imageOffset": 2122136,
"symbol": "timer_thread_func",
"symbolLocation": 656,
"imageIndex": 9
},
{
"imageOffset": 28724,
"symbol": "_pthread_start",
"symbolLocation": 136,
"imageIndex": 12
},
{
"imageOffset": 7740,
"symbol": "thread_start",
"symbolLocation": 8,
"imageIndex": 12
}
],
"threadState": {
"x": [
{
"value": 4
},
{
"value": 0
},
{
"value": 0
},
{
"value": 4390376748,
"symbolLocation": 172,
"symbol": "_MergedGlobals.371"
},
{
"value": 16
},
{
"value": 0
},
{
"value": 0
},
{
"value": 0
},
{
"value": 4390377312,
"symbolLocation": 736,
"symbol": "_MergedGlobals.371"
},
{
"value": 2
},
{
"value": 0
},
{
"value": 2
},
{
"value": 0
},
{
"value": 0
},
{
"value": 0
},
{
"value": 0
},
{
"value": 363
},
{
"value": 8146269992
},
{
"value": 0
},
{
"value": 4294967295
},
{
"value": 4390309040,
"symbolLocation": 16,
"symbol": "_MergedGlobals"
},
{
"value": 4924172040
},
{
"value": 4390377264,
"symbolLocation": 688,
"symbol": "_MergedGlobals.371"
},
{
"value": 4390377344,
"symbolLocation": 768,
"symbol": "_MergedGlobals.371"
},
{
"value": 4390377264,
"symbolLocation": 688,
"symbol": "_MergedGlobals.371"
},
{
"value": 419018983969000
},
{
"value": 1000000
},
{
"value": 4924171960
},
{
"value": 4924172144
}
],
"flavor": "ARM_THREAD_STATE64",
"lr": {
"value": 4387742104
},
"cpsr": {
"value": 1610616832
},
"fp": {
"value": 6092664768
},
"sp": {
"value": 6092664576
},
"esr": {
"value": 1442840704,
"description": " Address size fault"
},
"pc": {
"value": 6543473088
},
"far": {
"value": 0
}
}
}
],
"usedImages": [
{
"source": "P",
"arch": "arm64",
"base": 4865998848,
"size": 147456,
"uuid": "48142769-413e-3460-8fbb-403ed1059277",
"path": "/Users/USER/*/socket.bundle",
"name": "socket.bundle"
},
{
"source": "P",
"arch": "arm64",
"base": 4865392640,
"size": 163840,
"uuid": "e7c54df0-e82f-32a6-bda8-0dbab44928c9",
"path": "/Users/USER/*/pg_ext.bundle",
"name": "pg_ext.bundle"
},
{
"source": "P",
"arch": "arm64",
"base": 4865687552,
"size": 229376,
"uuid": "58daecdc-5784-37c0-9468-ae9517b60aaa",
"path": "/Users/USER/*/libpq.5.14.dylib",
"name": "libpq.5.14.dylib"
},
{
"source": "P",
"arch": "arm64",
"base": 4866818048,
"size": 540672,
"uuid": "71ada2bf-c666-3669-9489-d354d1f0f06c",
"path": "/opt/homebrew/*/libssl.3.dylib",
"name": "libssl.3.dylib"
},
{
"source": "P",
"arch": "arm64",
"base": 4871831552,
"size": 3014656,
"uuid": "26b09b12-09c0-36ca-abe9-a78b49d334a0",
"path": "/opt/homebrew/*/libcrypto.3.dylib",
"name": "libcrypto.3.dylib"
},
{
"source": "P",
"arch": "arm64",
"base": 4381687808,
"size": 32768,
"uuid": "c74629a4-e20b-3ee2-81c4-8d9233ccbcb4",
"path": "/Users/USER/*/pathname.bundle",
"name": "pathname.bundle"
},
{
"source": "P",
"arch": "arm64",
"base": 4375314432,
"size": 16384,
"uuid": "b729ba6c-0315-34e3-b329-de3d01ead4bb",
"path": "/Users/USER/*/monitor.bundle",
"name": "monitor.bundle"
},
{
"source": "P",
"arch": "arm64",
"base": 4375183360,
"size": 16384,
"uuid": "7bbb0883-de9c-372c-908d-9af5254a2d19",
"path": "/Users/USER/*/transdb.bundle",
"name": "transdb.bundle"
},
{
"source": "P",
"arch": "arm64",
"base": 4375052288,
"size": 16384,
"uuid": "32c4a410-6a6e-3932-9f70-ba259090c963",
"path": "/Users/USER/*/encdb.bundle",
"name": "encdb.bundle"
},
{
"source": "P",
"arch": "arm64",
"base": 4385619968,
"size": 4587520,
"uuid": "5da0da22-fce6-394c-b9c5-568069961dfc",
"path": "/Users/USER/*/libruby.3.3.dylib",
"name": "libruby.3.3.dylib"
},
{
"source": "P",
"arch": "arm64",
"base": 4374773760,
"size": 16384,
"uuid": "f8a4348a-d09a-36e8-a156-529550909f41",
"path": "/Users/USER/*/ruby",
"name": "ruby"
},
{
"source": "P",
"arch": "arm64e",
"base": 6543441920,
"size": 241664,
"uuid": "ca94fc21-bc40-3b43-b65d-b87ece9e1d48",
"path": "/usr/lib/system/libsystem_kernel.dylib",
"name": "libsystem_kernel.dylib"
},
{
"source": "P",
"arch": "arm64e",
"base": 6543683584,
"size": 53236,
"uuid": "a7d94c96-7b1f-3229-9bea-048d037c3292",
"path": "/usr/lib/system/libsystem_pthread.dylib",
"name": "libsystem_pthread.dylib"
},
{
"source": "P",
"arch": "arm64e",
"base": 6542225408,
"size": 520188,
"uuid": "ab8a0a0b-9a1e-3284-9243-5a0334468ae5",
"path": "/usr/lib/system/libsystem_c.dylib",
"name": "libsystem_c.dylib"
},
{
"source": "P",
"arch": "arm64e",
"base": 6543892480,
"size": 28668,
"uuid": "e1840daa-9c10-393d-a3a3-94dde6fe0011",
"path": "/usr/lib/system/libsystem_platform.dylib",
"name": "libsystem_platform.dylib"
},
{
"source": "P",
"arch": "arm64e",
"base": 6540013568,
"size": 607048,
"uuid": "324e4ad9-e01f-3183-b09f-3e20b326643a",
"path": "/usr/lib/dyld",
"name": "dyld"
},
{
"size": 0,
"source": "A",
"base": 0,
"uuid": "00000000-0000-0000-0000-000000000000"
}
],
"sharedCache": {
"base": 6539264000,
"size": 4061216768,
"uuid": "f9ddd844-7f3f-34bd-be29-f0c72d5e5449"
},
"vmSummary": "ReadOnly portion of Libraries: Total=1.0G resident=0K(0%) swapped_out_or_unallocated=1.0G(100%)\nWritable regions: Total=1.6G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.6G(100%)\n\n VIRTUAL REGION \nREGION TYPE SIZE COUNT (non-coalesced) \n=========== ======= ======= \nActivity Tracing 256K 1 \nKernel Alloc Once 32K 1 \nMALLOC 1.2G 26 \nMALLOC guard page 96K 6 \nSTACK GUARD 16K 1 \nStack 8720K 2 \nStack Guard 56.0M 1 \nVM_ALLOCATE 172.9M 128 \nVM_ALLOCATE (reserved) 256.0M 1 reserved VM address space (unallocated)\n__AUTH 339K 65 \n__AUTH_CONST 4056K 146 \n__DATA 1967K 149 \n__DATA_CONST 4753K 158 \n__DATA_DIRTY 365K 58 \n__LINKEDIT 894.1M 12 \n__OBJC_RO 71.1M 1 \n__OBJC_RW 2168K 1 \n__TEXT 128.2M 165 \ndyld private memory 272K 2 \nmapped file 7296K 5 \nshared memory 32K 2 \n=========== ======= ======= \nTOTAL 2.7G 931 \nTOTAL, minus reserved VM space 2.5G 931 \n",
"legacyInfo": {
"threadTriggered": {
"queue": "com.apple.main-thread"
}
},
"logWritingSignature": "64f1906ac9a8c822e3990ee026b4c5ab13dba069",
"trialInfo": {
"rollouts": [
{
"rolloutId": "60186475825c62000ccf5450",
"factorPackIds": {},
"deploymentId": 240000068
},
{
"rolloutId": "63508950b3714d3622fc77f7",
"factorPackIds": {
"SIRI_MEMORY_SYNC_CONFIG": "63508beab3714d3622fc7808"
},
"deploymentId": 240000004
}
],
"experiments": []
}
}
|
|
This only seems to hit and affect my mac (running Sonoma 14.2.1 currently). The same ruby code that crashes on my mac, when running on windows server and windows workstation runs fine. By contrast with the OP of this issue, I am connecting to an Aurora PG cluster. All other (non-pg gem) means of connecting to this same cluster connect fine from the same mac where the pg gem crashes on connect. The crash occurs with every version of the pg gem i've tried, going all the way back to 1.2.3. |
|
Hey @ohTHATaaronbrown thanks for adding that. So far all of those affected are running Sonoma and connecting to RDS (both aurora and non-aurora Postgres). I realize now that the crash report doesn't really include a C-stack, just some symbols. The Ruby stack points at the call to Lines 693 to 697 in 5c105f2 PQFinish which is being called here Line 537 in 5c105f2 It seems that something goes wrong before the |
|
Another update from a colleague, also using an M1 macbook, also on Sonoma. His connections to RDS were still working until today, when he installed neovim using After encountering segfaults, he ran |
|
Ahhh. I already had |
|
Looks like upgrading |
|
Ok, I have also resolved the issue locally by installing I believe my problem is that the copy of postgres (and Not really a |
|
Thanks for all the hints. I was able to resolve it just by -- brew upgrade postgresql in my mac Somona 14.2.1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've recently started to experience segfaults when connecting to an RDS instance from my MacBook.
Some details:
ruby 3.1.4p2231.2.3and1.5.4.In my testing, I have been unable to reproduce the issue when running different Postgres versions locally. That is, everything works with any combination of pg gem and server version.
However, when I connect to an RDS instance (standard Postgres, not Aurora, version 13.10) I receive a segfault due to a double-free ("pointer being freed was not allocated"). Setting
PGGSSENCMODE="disable"does not fix the issue. Connecting viapsqldoes work. In trying to narrow things down, the only thing I can see that might be relevant is that the connection to RDS uses TLS, whereas my local connections do not.The text was updated successfully, but these errors were encountered: