Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creation an object with ignore_on_error flag set causes an infinite recursion #8824

Closed
yhabteab opened this issue Jun 4, 2021 · 6 comments · Fixed by #9406
Closed

Creation an object with ignore_on_error flag set causes an infinite recursion #8824

yhabteab opened this issue Jun 4, 2021 · 6 comments · Fixed by #9406
Labels
area/configuration DSL, parser, compiler, error handling bug Something isn't working
Milestone

Comments

@yhabteab
Copy link
Member

yhabteab commented Jun 4, 2021

Describe the bug

When an error is encountered while creating an object, for example due to forgetting the check_command attribute for a host object and having ignore_on_error flag set, it unfortunately causes an endless recursion in ConfigItem::CommitNewItems() and finally leads Icinga2 to crash. It is normally the case that if an error occurs in ConfigItem::Commit() when evaluating or validating the config and the ignore_on_error flag is not set, immediately an error is raised but if this flag is set, a nullptr is returned instead. Since this returned value is not checked in CommitNewItems() it will always continue even though the object could not be created.

And as CommitNewItems() calls itself recursively and the config item which could not be created has not been committed and hence is not yet a config object, it will be considered as a new object over and over again. This procedure will be repeated until it most likely ends up hitting a corrupt memory and crashing icinga2 because none of the conditions will break this infinite recursion call unless an exception is raised in the WorkQueue.

However, I'm still not sure where exactly this crash occurs, as it only happens occasionally and the nullptr value being returned is nowhere direferenced.

@yhabteab
Copy link
Member Author

yhabteab commented Jun 7, 2021

ping @Al2Klimov!

@Al2Klimov
Copy link
Member

If ignore_on_error crashes API objects, how did regular config objects ever work?

@yhabteab
Copy link
Member Author

yhabteab commented Jun 8, 2021

If ignore_on_error crashes API objects, how did regular config objects ever work?

As I already said, the problem appears only after this fix.

You can only reproduce the error with this fix #8819.

@yhabteab
Copy link
Member Author

yhabteab commented Jun 21, 2021

Indeed, I don't think anyone sets the ignore_on_error flag even on manually created objects, but Icinga2 will still not continue anyway, even if the flag is set, when error occurs in the configs.

[2021-06-21 12:33:39 +0200] notice/ConfigObject: Ignoring config object 'icinga2-host' of type 'Host' due to errors: Error: Validation failed for object 'icinga2-host' of type 'Host'; Attribute 'check_command': Attribute must not be empty.
Location: in /Users/yhabteab/Workspace/icinga2/prefix/etc/icinga2/conf.d/hosts.conf: 18:1-18:42
/Users/yhabteab/Workspace/icinga2/prefix/etc/icinga2/conf.d/hosts.conf(16):  */
/Users/yhabteab/Workspace/icinga2/prefix/etc/icinga2/conf.d/hosts.conf(17): 
/Users/yhabteab/Workspace/icinga2/prefix/etc/icinga2/conf.d/hosts.conf(18): object Host "icinga2-host" ignore_on_error {
                                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/Users/yhabteab/Workspace/icinga2/prefix/etc/icinga2/conf.d/hosts.conf(19):     address = "localhost"
/Users/yhabteab/Workspace/icinga2/prefix/etc/icinga2/conf.d/hosts.conf(20):     vars.os = "Linux"
[2021-06-21 12:33:39 +0200] notice/cli: Worker process couldn't load its config

@Al2Klimov
Copy link
Member

As I already said, the problem appears only after this fix.

You can only reproduce the error with this fix #8819.

Then I'd say it's a problem of the fix.

@yhabteab
Copy link
Member Author

Then I'd say it's a problem of the fix.

This can be also reproduced without the referenced PR.

Test Config:

object Host "dummy-check" ignore_on_error {
    address = "localhost"
}

Result: With a snapshot package.

root@parallels-Parallels-Virtual-Platform:/etc/icinga2/conf.d# icinga2 daemon -C
[2021-10-22 17:39:51 +0200] information/cli: Icinga application loader (version: v2.13.0-69-ga7cbf5067)
[2021-10-22 17:39:51 +0200] information/cli: Loading configuration file(s).
[2021-10-22 17:39:51 +0200] information/ConfigItem: Committing config item(s).
[2021-10-22 17:40:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 0, rate: 5536.83/s (332210/min 332210/5min 332210/15min);
[2021-10-22 17:40:51 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 39035.2/s (2342114/min 2343932/5min 2343932/15min);
[2021-10-22 17:41:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 39048.5/s (2342912/min 2678602/5min 2678602/15min);
[2021-10-22 17:41:11 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 38995.3/s (2339716/min 3017034/5min 3017034/15min);
[2021-10-22 17:42:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 37891.5/s (2273492/min 4955786/5min 4955786/15min);
[2021-10-22 17:43:31 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 37716.2/s (2262970/min 8234992/5min 8234992/15min);
[2021-10-22 17:43:51 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 38034.2/s (2282054/min 9155972/5min 9155972/15min);
[2021-10-22 17:44:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 37781.5/s (2266892/min 9480282/5min 9480282/15min);
[2021-10-22 17:44:11 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 37844.7/s (2270684/min 9823834/5min 9823834/15min);
[2021-10-22 17:44:31 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 36534.2/s (2192050/min 10430618/5min 10430618/15min);
[2021-10-22 17:44:41 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 33200.7/s (1992044/min 10637584/5min 10637584/15min);
[2021-10-22 17:45:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 30276.6/s (1816598/min 10964782/5min 11300472/15min);
[2021-10-22 17:45:21 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 26802.7/s (1608160/min 10746082/5min 11757114/15min);
[2021-10-22 17:45:31 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 1, rate: 25424.8/s (1525488/min 10436396/5min 11959482/15min);
[2021-10-22 17:46:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 26408.2/s (1584494/min 10205058/5min 12887352/15min);
[2021-10-22 17:46:31 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 31871.8/s (1912308/min 10177978/5min 13876364/15min);
[2021-10-22 17:46:41 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 33051/s (1983062/min 9889370/5min 14175858/15min);
[2021-10-22 17:46:51 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 33108.3/s (1986498/min 9749160/5min 14375690/15min);
[2021-10-22 17:47:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 30964.4/s (1857862/min 9789582/5min 14749206/15min);
[2021-10-22 17:47:11 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 31199.8/s (1871988/min 9809034/5min 15102504/15min);
[2021-10-22 17:47:21 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 29398.8/s (1763930/min 9673336/5min 15299920/15min);
[2021-10-22 17:47:41 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 26426.4/s (1585584/min 9222678/5min 15763862/15min);
[2021-10-22 17:47:51 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 27109.8/s (1626590/min 9128866/5min 16002784/15min);
[2021-10-22 17:48:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 24296.1/s (1457768/min 9001086/5min 16214476/15min);
[2021-10-22 17:48:31 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 1, rate: 28753.9/s (1725234/min 8988316/5min 17226884/15min);
[2021-10-22 17:48:41 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 1, rate: 29249.7/s (1754982/min 8877144/5min 17522684/15min);
[2021-10-22 17:48:51 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 30826.6/s (1849598/min 8696332/5min 17855690/15min);
[2021-10-22 17:49:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 32940.8/s (1976450/min 8707962/5min 18191836/15min);
[2021-10-22 17:49:11 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 30905.8/s (1854346/min 8661190/5min 18488520/15min);
[2021-10-22 17:49:21 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 35104.9/s (2106295/min 8882866/5min 19031820/15min);
[2021-10-22 17:49:31 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 1, rate: 33906.3/s (2034380/min 8830958/5min 19264952/15min);
[2021-10-22 17:49:51 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 32233/s (1933978/min 8826760/5min 19793098/15min);
[2021-10-22 17:50:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 0, rate: 30715.5/s (1842928/min 8735738/5min 20038596/15min);
[2021-10-22 17:50:41 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 34665.6/s (2079934/min 9383034/5min 21575830/15min);
[2021-10-22 17:50:51 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 35242.3/s (2114540/min 9522066/5min 21911258/15min);
[2021-10-22 17:51:11 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 36714.7/s (2202884/min 9347452/5min 22577968/15min);
[2021-10-22 17:51:21 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 1, rate: 33985.4/s (2039124/min 9336494/5min 22872484/15min);
[2021-10-22 17:51:41 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 34608.9/s (2076532/min 9477126/5min 23655404/15min);
[2021-10-22 17:52:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 32402.1/s (1944128/min 9429152/5min 24185860/15min);
[2021-10-22 17:52:11 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 31397/s (1883820/min 9360828/5min 24465504/15min);
[2021-10-22 17:52:41 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 1, rate: 33462.2/s (2007734/min 9898982/5min 25666684/15min);
[2021-10-22 17:52:51 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 34170.2/s (2050214/min 9980046/5min 25986138/15min);
[2021-10-22 17:53:01 +0200] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 2, rate: 34515.6/s (2070938/min 10044438/5min 26259824/15min);

AND SO ON ........

@yhabteab yhabteab reopened this Oct 22, 2021
@yhabteab yhabteab changed the title Creation an object via API with ignore_on_error flag set causes an infinite recursion Creation an object with ignore_on_error flag set causes an infinite recursion Oct 22, 2021
@Al2Klimov Al2Klimov added the bug Something isn't working label Oct 26, 2021
@Al2Klimov Al2Klimov added the area/configuration DSL, parser, compiler, error handling label Nov 30, 2021
@Al2Klimov Al2Klimov added this to the 2.13.6 milestone Jul 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/configuration DSL, parser, compiler, error handling bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants