[YSQL] Calling fclose when fopen failed while trying to set oom_score_adj when a postgres backend starts #12843
Assignees
Labels
2.14 Backport Required
area/ysql
Yugabyte SQL (YSQL)
kind/bug
This issue is a bug
priority/medium
Medium priority issue
Comments
ramsrivatsa
added
area/ysql
yugabyte-ci
added
kind/bug
ramsrivatsa
assigned ramsrivatsa, sushantrmishra, kmuthukk, kapil-yb, sonalsagarwal and tedyu and unassigned sushantrmishra, sonalsagarwal, kmuthukk, kapil-yb and tedyu
ramsrivatsa
added a commit
that referenced
this issue
Jun 9, 2022
…ing oom_score_adj Summary: When a `ysqlsh` shell is opened, the postmaster tries to set the oom_score_adj to a specific value. This is accomplished by modifying the oom value in the following file `/proc/[pid]/oom_score_adj`. However, when trying to set this sometimes the postmaster crashes with the following error. ``` ysqlsh: could not connect to server: Connection refused\n\tIs the server running on host "172.151.31.53" and accepting\n\tTCP/IP connections on port 5433? ``` ``` warning: File "/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". To enable execution of this file add add-auto-load-safe-path /home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1 line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. warning: File "/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by `/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/postgres/bin/post'. Program terminated with signal 11, Segmentation fault. #0 _IO_new_fclose (fp=0x0) at iofclose.c:53 53 iofclose.c: No such file or directory. (gdb) bt #0 _IO_new_fclose (fp=0x0) at iofclose.c:53 #1 0x00000000008ab237 in BackendStartup (port=0x1b7c1e0) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4255 #2 ServerLoop () at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1767 #3 0x00000000008a7d01 in PostmasterMain (argc=<optimized out>, argv=0x1b92780) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1423 #4 0x00000000007c85c3 in PostgresServerProcessMain (argc=23, argv=0x1b92780) at ../../../../../../src/postgres/src/backend/main/main.c:234 #5 0x00000000004f5ae2 in main () ``` This is due to the fact that fclose tries to close a file descriptor when fopen fails. The diff that caused this regression is as follows --> https://phabricator.dev.yugabyte.com/D14099 Log message looks like this ``` 2022-06-09 19:12:18.685 UTC [17266] LOG: error 2: No such file or directory, unable to open file /proc/17390/oom_score_adj123 ``` Test Plan: Force fclose to fail by providing a garbage path to the oom_score_adj. Make sure that after the fix postmaster does not restart or segfault does not happen. Reviewers: sagarwal, zyu, mihnea, smishra Reviewed By: zyu, smishra Subscribers: rthallam, kannan, yql Differential Revision: https://phabricator.dev.yugabyte.com/D17567
|
Closed with the following commit -> cae0227 |
ramsrivatsa
added a commit
that referenced
this issue
Jun 10, 2022
…eed while changing oom_score_adj Summary: Original Commit: [[ cae0227 | cae0227 ]] / [[ https://phabricator.dev.yugabyte.com/D17567 | D17578 ]] When a `ysqlsh` shell is opened, the postmaster tries to set the oom_score_adj to a specific value. This is accomplished by modifying the oom value in the following file `/proc/[pid]/oom_score_adj`. However, when trying to set this sometimes the postmaster crashes with the following error. ``` ysqlsh: could not connect to server: Connection refused\n\tIs the server running on host "172.151.31.53" and accepting\n\tTCP/IP connections on port 5433? ``` ``` warning: File "/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". To enable execution of this file add add-auto-load-safe-path /home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1 line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. warning: File "/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by `/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/postgres/bin/post'. Program terminated with signal 11, Segmentation fault. #0 _IO_new_fclose (fp=0x0) at iofclose.c:53 53 iofclose.c: No such file or directory. (gdb) bt #0 _IO_new_fclose (fp=0x0) at iofclose.c:53 #1 0x00000000008ab237 in BackendStartup (port=0x1b7c1e0) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4255 #2 ServerLoop () at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1767 #3 0x00000000008a7d01 in PostmasterMain (argc=<optimized out>, argv=0x1b92780) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1423 #4 0x00000000007c85c3 in PostgresServerProcessMain (argc=23, argv=0x1b92780) at ../../../../../../src/postgres/src/backend/main/main.c:234 #5 0x00000000004f5ae2 in main () ``` This is due to the fact that fclose tries to close a file descriptor when fopen fails. The diff that caused this regression is as follows --> https://phabricator.dev.yugabyte.com/D14099 Log message looks like this ``` 2022-06-09 19:12:18.685 UTC [17266] LOG: error 2: No such file or directory, unable to open file /proc/17390/oom_score_adj123 ``` Test Plan: Jenkins: rebase: 2.14 Force fclose to fail by providing a garbage path to the oom_score_adj. Make sure that after the fix postmaster does not restart or segfault does not happen. Reviewers: sagarwal, mihnea, smishra, zyu, rthallam Reviewed By: rthallam Subscribers: yql, kannan, rthallam Differential Revision: https://phabricator.dev.yugabyte.com/D17578
ramsrivatsa
added a commit
that referenced
this issue
Jun 13, 2022
…cceed while changing oom_score_adj Summary: Original Commit: [[ cae0227 | cae0227 ]] / [[ https://phabricator.dev.yugabyte.com/D17567 | D17578 ]] When a `ysqlsh` shell is opened, the postmaster tries to set the oom_score_adj to a specific value. This is accomplished by modifying the oom value in the following file `/proc/[pid]/oom_score_adj`. However, when trying to set this sometimes the postmaster crashes with the following error. ``` ysqlsh: could not connect to server: Connection refused\n\tIs the server running on host "172.151.31.53" and accepting\n\tTCP/IP connections on port 5433? ``` ``` warning: File "/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". To enable execution of this file add add-auto-load-safe-path /home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1 line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. warning: File "/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/linuxbrew/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by `/home/yugabyte/yb-software/yugabyte-2.15.1.0-b3-centos-x86_64/postgres/bin/post'. Program terminated with signal 11, Segmentation fault. #0 _IO_new_fclose (fp=0x0) at iofclose.c:53 53 iofclose.c: No such file or directory. (gdb) bt #0 _IO_new_fclose (fp=0x0) at iofclose.c:53 #1 0x00000000008ab237 in BackendStartup (port=0x1b7c1e0) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4255 #2 ServerLoop () at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1767 #3 0x00000000008a7d01 in PostmasterMain (argc=<optimized out>, argv=0x1b92780) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1423 #4 0x00000000007c85c3 in PostgresServerProcessMain (argc=23, argv=0x1b92780) at ../../../../../../src/postgres/src/backend/main/main.c:234 #5 0x00000000004f5ae2 in main () ``` This is due to the fact that fclose tries to close a file descriptor when fopen fails. The diff that caused this regression is as follows --> https://phabricator.dev.yugabyte.com/D14099 Log message looks like this ``` 2022-06-09 19:12:18.685 UTC [17266] LOG: error 2: No such file or directory, unable to open file /proc/17390/oom_score_adj123 ``` Test Plan: Jenkins: rebase: 2.15.0 Force fclose to fail by providing a garbage path to the oom_score_adj. Make sure that after the fix postmaster does not restart or segfault does not happen. Reviewers: sagarwal, mihnea, smishra, rthallam, zyu Reviewed By: rthallam Subscribers: yql, kannan, rthallam Differential Revision: https://phabricator.dev.yugabyte.com/D17582
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-2607
Description
When a
ysqlshshell is opened, the postmaster tries to set the oom_score_adj to a specific value. This is accomplished by modifying the oom value in the following file/proc/[pid]/oom_score_adj. However, when trying to set this sometimes the postmaster crashes with the following error.The text was updated successfully, but these errors were encountered: