Cacti 1.3.0 - Dev b94bf9c3 - Graph Unknown RRDtool Error and Segmentation fault (core dumped) after enabling Thold Threshold #5915

morganfw · 2024-11-12T17:52:02Z

Hi,
not sure if RRDtool, Cacti or Thold plugin related.
After create a new graph:

then enabling a Threshold in 'Console - Management - Graphs' by selecting Graph and Create Aggregate from Template, at next poller run, Graph going into error then in segmentation fault

If delete the Threshold, Graph works again.

TheWitness · 2024-11-12T18:55:27Z

Click on the Cog and then the wrench and post the RRDtool graph syntax that you see. Run that from the command line and let us know the error.

TheWitness · 2024-11-12T18:57:07Z

Example below. Also, what's your RRDtool version from the GUI under General Settings and then the command line?

morganfw · 2024-11-12T19:19:03Z

Click on the Cog and then the wrench and post the RRDtool graph syntax that you see. Run that from the command line and let us know the error.

Example below. Also, what's your RRDtool version from the GUI under General Settings and then the command line?

Hi @TheWitness,
below the requested informations.

RRDtool version:

RRDtool path:

RRDtool CLI version:

# /opt/rrdtool-1.9.0/bin/rrdtool
RRDtool 1.9.0  Copyright by Tobias Oetiker <tobi@oetiker.ch>
               Compiled Nov  4 2024 22:40:11

Usage: rrdtool [options] command command_options
Valid commands: create, update, updatev, graph, graphv,  dump, restore,
                last, lastupdate, first, info, list, fetch, tune,
                resize, xport, flushcached

RRDtool is distributed under the Terms of the GNU General
Public License Version 2. (www.gnu.org/copyleft/gpl.html)

For more information read the RRD manpages

RRDtool graph syntax:

/opt/rrdtool-1.9.0/bin/rrdtool graph - \
--imgformat=SVG \
--start='-86400' \
--end='-70' \
--pango-markup  \
--title='Local Linux Machine - CPU Utilization - CPUTotal' \
--vertical-label='percent' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--tabwidth '40' \
--rigid \
--upper-limit='110' \
--lower-limit='0' \
COMMENT:"From 2024-11-11 20\:01\:11 To 2024-11-12 20\:00\:01\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 \
--font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/var/www/html/cacti/rra/local_linux_machine_cpu_204.rrd':'cpu':AVERAGE \
DEF:b='/var/www/html/cacti/rra/local_linux_machine_cpu_204.rrd':'cpu':MAX \
AREA:a#FF00007F:'CPU Utilization'  \
GPRINT:a:LAST:'Current\:%8.0lf'  \
GPRINT:a:AVERAGE:'Average\:%8.0lf'  \
GPRINT:a:MAX:'Maximum\:%8.0lf\n'  \
LINE1:b#FF0000FF: \
COMMENT:' \n' \
COMMENT:'<u><b>Threshold Alert/Warning Values</b></u>\n' \
LINE1:95#800517:'Alert Hi for Local Linux Machine - CPU Utilization - CPUTotal (95)' \
LINE1:90#0000FF:'Warning Hi for Local Linux Machine - CPU Utilization - CPUTotal (90)' \
RRDtool Command lengths = 1320 characters.
								RRDtool Says:

SVG/XML Output OK

Graph syntax executed from CLI in attachment.

Cacti_RRDtool_Graph_Syntax_CLI.txt

TheWitness · 2024-11-12T19:47:36Z

That should be rendering then.

morganfw · 2024-11-12T20:35:47Z

That should be rendering then.

Yes it's strange, because if try to enable Real-time, the Graph works:

even converting the Graph Templates to PNG, same behaviour.
Other suggestions?

TheWitness · 2024-11-12T23:45:51Z

Downgrade to 1.8 and see if that does anything better.

TheWitness · 2024-11-12T23:46:38Z

Or another thing you can do is change the version on the settings general to version 1.7+ and it will disable a feature that we're experimenting with in rrdtool.

TheWitness · 2024-11-12T23:46:51Z

Make sure you log off and then back into cacti before you test graphs after you make that change.

morganfw · 2024-11-13T06:20:02Z

Downgrade to 1.8 and see if that does anything better.

Downgraded to 1.8+ nothing changed.

morganfw · 2024-11-13T06:31:24Z

By downgrade to 1.7+ on General tab, with binary path /opt/rrdtool-1.8.0/bin/rrdtool the Graph still works again.

TheWitness · 2024-11-13T13:17:02Z

Okay you should go back to the 1.9.0 binary and then attempt to do a backtrace of the rrd tool and then report that to Tobi Oetikers GitHub page for rrdtool. @netniV and @browniebraun, problems with the new feature hover. FYI.

* This may correct the issue with RRDtool. Not sure yet.

TheWitness · 2024-11-13T15:03:35Z

I just made a change to lib/rrd.php. Do a fresh pull and test again, easiest way is to put this in your Web Root (/var/www/html), mark it executable and run it.

#!/bin/sh
/bin/rm -rf cacti-develop
git clone -b develop https://github.com/cacti/cacti.git cacti-develop
/bin/cp -rpf cacti-develop/* cacti
chown -R apache:apache cacti
/bin/rm -rf cacti-develop

TheWitness · 2024-11-13T15:04:08Z

Unless you are just doing a clean pull of course.

morganfw · 2024-11-13T19:27:29Z

I'm just doing a git pull from my Cacti 1.3 installation, then switching back to RRDtool 1.9+ under General tab and 1.9.0 executable path under Path tab, but the problem persists.

# cd /var/www/html/cacti

# git pull
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 8 (delta 6), reused 5 (delta 3), pack-reused 0 (from 0)
Unpacking objects: 100% (8/8), 909 bytes | 34.00 KiB/s, done.
From https://github.com/Cacti/cacti
   b94bf9c3a..b91946646  develop    -> origin/develop
Updating b94bf9c3a..b91946646
Fast-forward
 lib/rrd.php | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)

# chown -R www-data:www-data /var/www/html/cacti

Even by executing the script under my Ubuntu 24.04 LTS Server (the Apache user and group is www-data):

# pwd
/var/www/html

# cat cacti-dev-download.sh
#!/bin/sh
/bin/rm -rf cacti-develop
git clone -b develop https://github.com/cacti/cacti.git cacti-develop
/bin/cp -rpf cacti-develop/* cacti
chown -R www-data:www-data cacti
/bin/rm -rf cacti-develop

If try to going back to RRDtool 1.7.2.+

with 1.9.0 path

The graph is displayed again

/opt/rrdtool-1.9.0/bin/rrdtool graph - \
--imgformat=PNG \
--start='1731439200' \
--end='1731525395' \
--pango-markup  \
--title='Local Linux Machine - CPU Utilization - CPUTotal' \
--vertical-label='percent' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--tabwidth '40' \
--rigid \
--upper-limit='110' \
--lower-limit='0' \
COMMENT:"From 2024-11-12 20\:20\:00 To 2024-11-13 20\:16\:35\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 \
--font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':AVERAGE \
DEF:b='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':MAX \
AREA:a#FF00007F:'CPU Utilization'  \
GPRINT:a:LAST:'Current\:%8.0lf'  \
GPRINT:a:AVERAGE:'Average\:%8.0lf'  \
GPRINT:a:MAX:'Maximum\:%8.0lf\n'  \
LINE1:b#FF0000FF: \
COMMENT:' \n' \
COMMENT:'<u><b>Threshold Alert/Warning Values</b></u>\n' \
LINE1:95#800517:'Alert Hi for Local Linux Machine - CPU Utilization - CPUTotal (95)' \
LINE1:90#0000FF:'Warning Hi for Local Linux Machine - CPU Utilization - CPUTotal (90)' \
RRDtool Command lengths = 1331 characters.
								RRDtool Says:

OK

morganfw · 2024-11-13T20:59:35Z

Opened an issue on RRDTool Github too.

TheWitness · 2024-11-13T22:07:52Z

Change graph to graphv on the command line and see if it crashes.

TheWitness · 2024-11-13T22:08:49Z

It's the graphv. Option that is crashing using a new option to include the data points on the command line output.

TheWitness · 2024-11-13T22:10:49Z

Tobi will be asking for a lot more detail. He will for sure ask for a backtrace.

morganfw · 2024-11-13T22:37:41Z

Change graph to graphv on the command line and see if it crashes.

It's the graphv. Option that is crashing using a new option to include the data points on the command line output.

Command executed with graphv, results attached

Cacti_RRDtool_graphv_command_CLI.txt

TheWitness · 2024-11-13T23:05:47Z

There is an option missing from the command. Let me look it up.

TheWitness · 2024-11-13T23:10:14Z

The option is --add-jsontime. Add that to the rrdtool options.

morganfw · 2024-11-14T06:16:56Z

The option is --add-jsontime. Add that to the rrdtool options.

Ok, after adding option to graphv, it crashes:

/opt/rrdtool-1.9.0/bin/rrdtool graphv - \
--add-jsontime \
--imgformat=PNG \
--start='1731477900' \
--end='1731564277' \
--pango-markup  \
--title='Local Linux Machine - CPU Utilization - CPUTotal' \
--vertical-label='percent' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--tabwidth '40' \
--rigid \
--upper-limit='110' \
--lower-limit='0' \
COMMENT:"From 2024-11-13 07\:05\:00 To 2024-11-14 07\:04\:37\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 \
--font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
LINE1:90#0000FF:'Warning Hi for Local Linux Machine - CPU Utilization - CPUTotal (90)' \
>
Segmentation fault (core dumped)

TheWitness · 2024-11-14T11:26:53Z

It's that last line. I think the buffer for the legend is too small. Let Tobi know. @netniV, can you patch rrdtool?

If you shorten that text format to 20 characters or so, I bet it won't crash any longer.

TheWitness · 2024-11-14T11:31:21Z

Show the graph syntax with tholds enabled. I thought those tholds lines were hrules not lines...

morganfw · 2024-11-14T13:22:18Z

It's that last line. I think the buffer for the legend is too small. Let Tobi know. @netniV, can you patch rrdtool?

If you shorten that text format to 20 characters or so, I bet it won't crash any longer.

New Thold for Memory Usage created and same issue. If shorten the name under 20 characters, the issu persists:

below Graph syntax:

/opt/rrdtool-1.9.0/bin/rrdtool graph - \
--imgformat=PNG \
--start='-86400' \
--end='-177' \
--pango-markup  \
--title='Local Linux Machine - Memory Usage Real' \
--vertical-label='bytes' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--rigid \
--alt-autoscale-max \
--lower-limit='0' \
COMMENT:"From 2024-11-13 14\:12\:57 To 2024-11-14 14\:10\:00\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 \
--font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/var/www/html/cacti/rra/local_linux_machine_mem_total_real_207.rrd':'mem_total_real':AVERAGE \
DEF:b='/var/www/html/cacti/rra/local_linux_machine_mem_avail_real_206.rrd':'mem_avail_real':AVERAGE \
CDEF:cdefa='a,1024,*' \
CDEF:cdefe='b,1024,*' \
AREA:cdefa#00CF00FF:'Total    '  \
GPRINT:cdefa:LAST:'Current\:%8.2lf %s'  \
GPRINT:cdefa:AVERAGE:'Average\:%8.2lf %s'  \
GPRINT:cdefa:MAX:'Maximum\:%8.2lf %s\n'  \
AREA:cdefe#DE0056FF:'Available'  \
GPRINT:cdefe:LAST:'Current\:%8.2lf %s'  \
GPRINT:cdefe:AVERAGE:'Average\:%8.2lf %s'  \
GPRINT:cdefe:MAX:'Maximum\:%8.2lf %s\n'  \
COMMENT:' \n' \
COMMENT:'<u><b>Threshold Alert/Warning Values</b></u>\n' \
LINE1:512000000#800517:'Alert Low for A (512 M)' \
LINE1:1024000000#0000FF:'Warning Low for A (1.02 G)' \
RRDtool Command lengths = 1478 characters.
								RRDtool Says:

OK

Trying to execute from CLI:

/opt/rrdtool-1.9.0/bin/rrdtool graphv - \
--add-jsontime \
--imgformat=PNG \
--start='-86400' \
--end='-14' \
--pango-markup  \
--title='Local Linux Machine - Memory Usage Real' \
--vertical-label='bytes' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--rigid \
--alt-autoscale-max \
--lower-limit='0' \
COMMENT:"From 2024-11-13 14\:15\:14 To 2024-11-14 14\:15\:00\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 \
--font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/var/www/html/cacti/rra/local_linux_machine_mem_total_real_207.rrd':'mem_total_real':AVERAGE \
LINE1:1024000000#0000FF:'Warning Low for A (1.02 G)' \n' \il_real_206.rrd':'mem_avail_real':AVERAGE \
>
Segmentation fault (core dumped)

Show the graph syntax with tholds enabled. I thought those tholds lines were hrules not lines...

Thold config:

Graph syntax with Thold enabled:

/opt/rrdtool-1.9.0/bin/rrdtool graph - \
--imgformat=PNG \
--start='-86400' \
--end='-204' \
--pango-markup  \
--title='Local Linux Machine - CPU Utilization - CPUTotal' \
--vertical-label='percent' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--tabwidth '40' \
--rigid \
--upper-limit='110' \
--lower-limit='0' \
COMMENT:"From 2024-11-13 14\:03\:24 To 2024-11-14 14\:00\:00\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 \
--font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':AVERAGE \
DEF:b='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':MAX \
AREA:a#FF00007F:'CPU Utilization'  \
GPRINT:a:LAST:'Current\:%8.0lf'  \
GPRINT:a:AVERAGE:'Average\:%8.0lf'  \
GPRINT:a:MAX:'Maximum\:%8.0lf\n'  \
LINE1:b#FF0000FF: \
COMMENT:' \n' \
COMMENT:'<u><b>Threshold Alert/Warning Values</b></u>\n' \
LINE1:95#800517:'Alert Hi for Local Linux Machine - CPU Utilization - CPUTotal (95)' \
LINE1:90#0000FF:'Warning Hi for Local Linux Machine - CPU Utilization - CPUTotal (90)' \
RRDtool Command lengths = 1321 characters.
								RRDtool Says:

OK

TheWitness · 2024-11-14T22:12:44Z

I've put a temp fix in for the moment so others don't experience this while we figure out what is broken in RRDtool.

TheWitness · 2024-11-15T13:22:00Z

I've created the following pull request to address this issue: oetiker/rrdtool-1.x#1269

morganfw · 2024-11-15T16:36:52Z

I've created the following pull request to address this issue: oetiker/rrdtool-1.x#1269

Thank you for the support @TheWitness, I've sent the backtrace to Tobi.

TheWitness · 2024-11-15T19:27:42Z

There is a real issue with the export logic. I've been 'playing' with it today, and though I can keep the tool from locking up, some of the data that comes back on the export is not right.

Additionally, it get's worse when you use redirected standard in vs. just calling the tool with the command line. This is indicative of a major data corruption somewhere in rrdtool.

I've prepared a new lib/rrd.php that will change the method by which RRDtool is called to make it at least a bit more reliable based upon my testing.

…-jsontime * This does not fix the bug in RRDtool, but with the patch delivered to Tobi, the RRDtool binary is not crashing any more for me. * In this fix, we will use shell_exec() in place of proc_open() for now and attempt up-to 5 times. If RRDtool crashes, the Cacti admin will get an Email stating as much as well.

This add's some additional logging and addresses one bogus return variable.

morganfw · 2024-11-15T20:36:44Z

Hi @TheWitness I've issued a git pull with your latest patches, now the following warning is written on cacti.log

2024-11-15 21:30:52 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command graphv - --imgformat=PNG --start='1731616200' --end='1731702600' --pango-markup --add-jsontime --title='Local Linux Machine - Memory Usage Real' --vertical-label='bytes' --slope-mode --base=1000 --height=200 --width=700 --rigid --alt-autoscale-max --lower-limit='0' COMMENT:"From 2024-11-14 21\:30\:00 To 2024-11-15 21\:30\:00\c" COMMENT:" \n" --color BACK#F3F3F3 --color CANVAS#FDFDFD --color SHADEA#CBCBCB --color SHADEB#999999 --color FONT#000000 --color AXIS#2C4D43 --color ARROW#2C4D43 --color FRAME#2C4D43 --border 1 --font TITLE:11:'Arial' --font AXIS:8:'Arial' --font LEGEND:8:'Courier' --font UNIT:8:'Arial' --font WATERMARK:6:'Arial' --slope-mode --watermark 'Generated by Cacti®' DEF:a='/var/www/html/cacti/rra/local_linux_machine_mem_total_real_207.rrd':'mem_total_real':AVERAGE DEF:b='/var/www/html/cacti/rra/local_linux_machine_mem_avail_real_206.rrd':'mem_avail_real':AVERAGE CDEF:cdefa='a,1024,*' CDEF:cdeff='b,1024,*' AREA:cdefa#00CF00FF:'Total ' GPRINT:cdefa:LAST:'Current\:%8.2lf %s' GPRINT:cdefa:MIN:'Minimum\:%8.2lf %s' GPRINT:cdefa:AVERAGE:'Average\:%8.2lf %s' GPRINT:cdefa:MAX:'Maximum\:%8.2lf %s\n' AREA:cdeff#F5F800FF:'Available' GPRINT:cdeff:LAST:'Current\:%8.2lf %s' GPRINT:cdeff:MIN:'Minimum\:%8.2lf %s' GPRINT:cdeff:AVERAGE:'Average\:%8.2lf %s' GPRINT:cdeff:MAX:'Maximum\:%8.2lf %s\n' COMMENT:' \n' COMMENT:'<u><b>Threshold Alert/Warning Values</b></u>\n' LINE1:512000000#800517:'Alert Low for Linux - Memory - Available Real (512 M)' LINE1:1024000000#0000FF:'Warning Low for Linux - Memory - Available Real (1.02 G)'

How I can compile your patched rrdtool here: https://github.com/TheWitness/rrdtool-1.x ?

morganfw · 2024-11-15T21:13:55Z

Ok @TheWitness I've compiled your patched RRDtool:

git clone -b master https://github.com/TheWitness/rrdtool-1.x.git

cd rrdtool-1.x/

./bootstrap

./configure --prefix=/opt/rrdtool-1.9.0-new

make clean

make -j4

make install

time to test now!

TheWitness · 2024-11-15T21:14:00Z

Yes. It's just a workaround for now though.

TheWitness · 2024-11-15T21:18:49Z

I think I see the issue now although I'm not the guy that needs to be fixing this there's some better people to deal with it.

It's a feature interaction issue. They recently introduced a feature to allow a line to be drawn as a constant.

Unfortunately this export function thinks that it's a real line and it's offsetting into memory as if there's values to be returned. Unfortunately that line is a constant and not containing any data from the Rd file structure so it needs to be eliminated from the array of returned objects.

Once that done offsetting into the data should be correct.

I guess I need to explain this to Tobi.

morganfw · 2024-11-15T21:44:04Z

@TheWitness with compiled RRDtool patched by you, I see segfault in GUI

but trying to execute from CLI, it works

/opt/rrdtool-1.9.0-patched/bin/rrdtool graphv - \
--add-jsontime \
--imgformat=PNG \
--start='-86400' \
--end='-70' \
--pango-markup  \
--title='Local Linux Machine - CPU Utilization - CPUTotal' \
--vertical-label='percent' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--tabwidth '40' \
--rigid \
--upper-limit='110' \
--lower-limit='0' \
COMMENT:"From 2024-11-14 22\:36\:10 To 2024-11-15 22\:35\:00\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 \
--font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':AVERAGE \
DEF:b='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':MAX \
AREA:a#FF00007F:'CPU Utilization'  \
GPRINT:a:LAST:'Current\:%8.0lf'  \
GPRINT:a:AVERAGE:'Average\:%8.0lf'  \
GPRINT:a:MAX:'Maximum\:%8.0lf\n'  \
LINE1:b#FF0000FF: \
COMMENT:' \n' \
COMMENT:'<u><b>Threshold Alert/Warning Values</b></u>\n' \
LINE1:95#800517:'Alert Hi for Local Linux Machine - CPU Utilization - CPUTotal (95)' \
LINE1:90#0000FF:'Warning Hi for Local Linux Machine - CPU Utilization - CPUTotal (90)' \
>
graph_left = 73
graph_top = 35
graph_width = 700
graph_height = 200
image_width = 803
image_height = 374
graph_start = 1731620293
graph_end = 1731706623
value_min = 0.0000000000e+00
value_max = 1.1000000000e+02
legend[0] = "From 2024-11-14 22:36:10 To 2024-11-15 22:35:00"
coords[0] = "237,257,566,271"
legend[1] = "  "
coords[1] = "16,271,30,285"
legend[2] = "  CPU Utilization"
coords[2] = "16,285,135,299"
legend[3] = "Current:       1"
coords[3] = "151,285,263,299"
legend[4] = "Average:       1"
coords[4] = "279,285,391,299"
legend[5] = "Maximum:      12"
coords[5] = "407,285,519,299"
legend[6] = " "
coords[6] = "16,299,23,313"
legend[7] = "<u><b>Threshold Alert/Warning Values</b></u>"
coords[7] = "16,313,226,327"
legend[8] = "  Alert Hi for Local Linux Machine - CPU Utilization - CPUTotal (95)"
coords[8] = "16,327,492,341"
legend[9] = "  Warning Hi for Local Linux Machine - CPU Utilization - CPUTotal (90)"
coords[9] = "16,341,506,355"
image = BLOB_SIZE:30814
[...]

same warn in cacti.log

2024-11-15 22:45:51 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command graphv - --imgformat=PNG --start='1731620700' --end='1731706851' --pango-markup --add-jsontime --title='Local Linux Machine - CPU Utilization - CPUTotal' --vertical-label='percent' --slope-mode --base=1000 --height=200 --width=700 --tabwidth '40' --rigid --upper-limit='110' --lower-limit='0' COMMENT:"From 2024-11-14 22\:45\:00 To 2024-11-15 22\:40\:51\c" COMMENT:" \n" --color BACK#F3F3F3 --color CANVAS#FDFDFD --color SHADEA#CBCBCB --color SHADEB#999999 --color FONT#000000 --color AXIS#2C4D43 --color ARROW#2C4D43 --color FRAME#2C4D43 --border 1 --font TITLE:11:'Arial' --font AXIS:8:'Arial' --font LEGEND:8:'Courier' --font UNIT:8:'Arial' --font WATERMARK:6:'Arial' --slope-mode --watermark 'Generated by Cacti®' DEF:a='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':AVERAGE DEF:b='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':MAX AREA:a#FF00007F:'CPU Utilization' GPRINT:a:LAST:'Current\:%8.0lf' GPRINT:a:AVERAGE:'Average\:%8.0lf' GPRINT:a:MAX:'Maximum\:%8.0lf\n' LINE1:b#FF0000FF: COMMENT:' \n' COMMENT:'<u><b>Threshold Alert/Warning Values</b></u>\n' LINE1:95#800517:'Alert Hi for Local Linux Machine - CPU Utilization - CPUTotal (95)' LINE1:90#0000FF:'Warning Hi for Local Linux Machine - CPU Utilization - CPUTotal (90)'
2024-11-15 22:45:48 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command graphv - --imgformat=PNG --start='1731620701' --end='1731707101' --pango-markup --add-jsontime --title='Local Linux Machine - CPU Utilization - CPUTotal' --vertical-label='percent' --slope-mode --base=1000 --height=200 --width=700 --tabwidth '40' --rigid --upper-limit='110' --lower-limit='0' COMMENT:"From 2024-11-14 22\:45\:01 To 2024-11-15 22\:45\:01\c" COMMENT:" \n" --color BACK#F3F3F3 --color CANVAS#FDFDFD --color SHADEA#CBCBCB --color SHADEB#999999 --color FONT#000000 --color AXIS#2C4D43 --color ARROW#2C4D43 --color FRAME#2C4D43 --border 1 --font TITLE:11:'Arial' --font AXIS:8:'Arial' --font LEGEND:8:'Courier' --font UNIT:8:'Arial' --font WATERMARK:6:'Arial' --slope-mode --watermark 'Generated by Cacti®' DEF:a='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':AVERAGE DEF:b='/var/www/html/cacti/rra/local_linux_machine_cpu_205.rrd':'cpu':MAX AREA:a#FF00007F:'CPU Utilization' GPRINT:a:LAST:'Current\:%8.0lf' GPRINT:a:AVERAGE:'Average\:%8.0lf' GPRINT:a:MAX:'Maximum\:%8.0lf\n' LINE1:b#FF0000FF: COMMENT:' \n' COMMENT:'<u><b>Threshold Alert/Warning Values</b></u>\n' LINE1:95#800517:'Alert Hi for Local Linux Machine - CPU Utilization - CPUTotal (95)' LINE1:90#0000FF:'Warning Hi for Local Linux Machine - CPU Utilization - CPUTotal (90)'

the difference from before is that I also had the memory graph in segfault, while now with your patched RRDtool, it works!

TheWitness · 2024-11-15T22:23:22Z

@morganfw, I just updated my rrdtool-1.x repo with the final solution. Data is good now. Going to let Tobi know.

morganfw · 2024-11-16T06:31:48Z

@morganfw, I just updated my rrdtool-1.x repo with the final solution. Data is good now. Going to let Tobi know.

@TheWitness I've done a git pull again and recompiled, now it works like a charm.

You are the man!

morganfw · 2024-11-16T07:23:46Z

After creating a new DS, then a new Graph from scratch, the following Warn are written to log:

2024-11-16 08:02:23 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command graphv - --imgformat=PNG --start='1731654000' --end='1731740400' --pango-markup --add-jsontime --title='Local Fail2Ban - Fail2Ban Current SSH Statistics' --vertical-label='Current' --slope-mode --base=1000 --height=200 --width=700 --tabwidth '30' --alt-autoscale-max --lower-limit='0' COMMENT:"From 2024-11-15 08\:00\:00 To 2024-11-16 08\:00\:00\c" COMMENT:" \n" --color BACK#F3F3F3 --color CANVAS#FDFDFD --color SHADEA#CBCBCB --color SHADEB#999999 --color FONT#000000 --color AXIS#2C4D43 --color ARROW#2C4D43 --color FRAME#2C4D43 --border 1 --font TITLE:11:'Arial' --font AXIS:8:'Arial' --font LEGEND:8:'Courier' --font UNIT:8:'Arial' --font WATERMARK:6:'Arial' --slope-mode --watermark 'Generated by Cacti®' DEF:a='/var/www/html/cacti/rra/local_fail2ban_cur_banned_208.rrd':'cur_failed':AVERAGE DEF:b='/var/www/html/cacti/rra/local_fail2ban_cur_banned_208.rrd':'cur_banned':AVERAGE AREA:a#00CF00FF:'Failed Current' GPRINT:a:LAST:'Current\:%8.2lf %s' GPRINT:a:AVERAGE:'Average\:%8.2lf %s' GPRINT:a:MAX:'Maximum\:%8.2lf %s\n' AREA:b#FF00007F:'Banned Current' GPRINT:b:LAST:'Current\:%8.2lf %s' GPRINT:b:AVERAGE:'Average\:%8.2lf %s' GPRINT:b:MAX:'Maximum\:%8.2lf %s\n'
2024-11-16 08:02:23 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command info /var/www/html/cacti/rra/local_fail2ban_cur_banned_208.rrd
2024-11-16 08:02:23 - MAILER INFO: Mail successfully sent via SMTP from 'Cacti 1.3 Dev Monitoring <cacti.monitoring@acme.com>', to 'Administrator <jdoe@acme.com>', cc '', bcc '', and took 1.17 seconds, Subject 'Poller in Heartbeat Mode'
2024-11-16 08:02:21 - RRDTOOL WARNING: RRDtool Cashed executing the following command info /var/www/html/cacti/rra/local_fail2ban_cur_banned_208.rrd
2024-11-16 08:02:11 - AUTOM8 NOTE: Graph Added - Device[[Local Fail2Ban]], Graph[[Local Fail2Ban - Fail2Ban Current SSH Statistics]], DS[[Local Fail2Ban - Fail2Ban Current SSH Statistics]], Graphs[[Local Fail2Ban - Fail2Ban Current SSH Statistics]]
2024-11-16 08:02:11 - AUTOM8 NOTE: Data Check Succeeded for - Device[[Local Fail2Ban]], GT[[Fail2Ban - SSH Stats Current]]

If try to go to graph and enable Real-time, a pop-up with error 500 appear, and following error is written to log:

2024-11-16 08:15:56 - PHP ERROR Backtrace: (CactiShutdownHandler())
2024-11-16 08:15:56 - ERROR PHP ERROR: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, string given in /var/www/html/cacti/lib/rrd.php:458 Stack trace: #0 /var/www/html/cacti/lib/rrd.php(458): fwrite() #1 /var/www/html/cacti/lib/rrd.php(284): __rrd_execute() #2 /var/www/html/cacti/lib/rrd.php(2728): rrdtool_execute() #3 /var/www/html/cacti/graph_realtime.php(229): rrdtool_function_graph() #4 {main} thrown in file: /var/www/html/cacti/lib/rrd.php on line: 458

TheWitness · 2024-11-16T12:20:59Z

I knew something would break. I'll try to get to it this week, but I'm going to be out of time with no access to my lab.

TheWitness · 2024-11-16T12:22:29Z

Did those files exist? I imagining that this is the problem.

TheWitness · 2024-11-16T12:43:37Z

Okay, a few hours before heading to the airport. So, I've fixed this one:

2024-11-16 08:15:56 - PHP ERROR Backtrace: (CactiShutdownHandler())
2024-11-16 08:15:56 - ERROR PHP ERROR: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, string given in /var/www/html/cacti/lib/rrd.php:458 Stack trace: #0 /var/www/html/cacti/lib/rrd.php(458): fwrite() #1 /var/www/html/cacti/lib/rrd.php(284): __rrd_execute() #2 /var/www/html/cacti/lib/rrd.php(2728): rrdtool_execute() #3 /var/www/html/cacti/graph_realtime.php(229): rrdtool_function_graph() #4 {main} thrown in file: /var/www/html/cacti/lib/rrd.php on line: 458

TheWitness · 2024-11-16T12:47:45Z

Just about to commit something to correct this one:

2024-11-16 08:02:23 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command info /var/www/html/cacti/rra/local_fail2ban_cur_banned_208.rrd

This should catch the failed INFO calls.

TheWitness · 2024-11-16T12:50:03Z

I might want to do a backtrace on these failures too.

This will help us figure out the source of the issues.

TheWitness · 2024-11-16T12:54:45Z

Okay, the backtrace is there now. Keep me posted.

morganfw · 2024-11-16T19:30:58Z

Okay, the backtrace is there now. Keep me posted.

Deleted and recreated graph, clicked on real-time, below the logs:

2024-11-16 20:26:45 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command info /var/www/html/cacti/rra/local_fail2ban_cur_banned_209.rrd.
2024-11-16 20:26:45 - RRDTOOL Error Backtrace: (/graph_realtime.php[229]:rrdtool_function_graph(), /lib/rrd.php[1747]:generate_graph_best_cf(), /lib/functions.php[3463]:get_rrd_cfs(), /lib/functions.php[3508]:rrdtool_execute(), /lib/rrd.php[284]:__rrd_execute(), /lib/rrd.php[444]:cacti_debug_backtrace())
2024-11-16 20:26:41 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command graphv - --imgformat=PNG --start='1731698700' --end='1731785100' --pango-markup --add-jsontime --title='Local Fail2Ban - Fail2Ban Current SSH Statistics' --vertical-label='Current' --slope-mode --base=1000 --height=200 --width=700 --tabwidth '30' --alt-autoscale-max --lower-limit='0' COMMENT:"From 2024-11-15 20\:25\:00 To 2024-11-16 20\:25\:00\c" COMMENT:" \n" --color BACK#F3F3F3 --color CANVAS#FDFDFD --color SHADEA#CBCBCB --color SHADEB#999999 --color FONT#000000 --color AXIS#2C4D43 --color ARROW#2C4D43 --color FRAME#2C4D43 --border 1 --font TITLE:11:'Arial' --font AXIS:8:'Arial' --font LEGEND:8:'Courier' --font UNIT:8:'Arial' --font WATERMARK:6:'Arial' --slope-mode --watermark 'Generated by Cacti®' DEF:a='/var/www/html/cacti/rra/local_fail2ban_cur_banned_209.rrd':'cur_failed':AVERAGE DEF:b='/var/www/html/cacti/rra/local_fail2ban_cur_banned_209.rrd':'cur_banned':AVERAGE AREA:a#00CF00FF:'Failed Current' GPRINT:a:LAST:'Current\:%8.2lf %s' GPRINT:a:AVERAGE:'Average\:%8.2lf %s' GPRINT:a:MAX:'Maximum\:%8.2lf %s\n' AREA:b#FF00007F:'Banned Current' GPRINT:b:LAST:'Current\:%8.2lf %s' GPRINT:b:AVERAGE:'Average\:%8.2lf %s' GPRINT:b:MAX:'Maximum\:%8.2lf %s\n' .
2024-11-16 20:26:41 - RRDTOOL Error Backtrace: (/graph_json.php[168]:rrdtool_function_graph(), /lib/rrd.php[2750]:rrdtool_execute(), /lib/rrd.php[284]:__rrd_execute(), /lib/rrd.php[444]:cacti_debug_backtrace())
2024-11-16 20:26:41 - RRDTOOL WARNING: RRDtool failed after 5 attempts for the following command info /var/www/html/cacti/rra/local_fail2ban_cur_banned_209.rrd.
2024-11-16 20:26:41 - RRDTOOL Error Backtrace: (/graph_json.php[168]:rrdtool_function_graph(), /lib/rrd.php[1747]:generate_graph_best_cf(), /lib/functions.php[3463]:get_rrd_cfs(), /lib/functions.php[3508]:rrdtool_execute(), /lib/rrd.php[284]:__rrd_execute(), /lib/rrd.php[444]:cacti_debug_backtrace())
2024-11-16 20:26:41 - MAILER INFO: Mail successfully sent via SMTP from 'Cacti 1.3 Dev Monitoring <cacti.monitoring@acme.com>', to 'Administrator <jdoe@acme.com>', cc '', bcc '', and took 2.26 seconds, Subject 'Poller in Heartbeat Mode'
2024-11-16 20:26:39 - RRDTOOL WARNING: RRDtool Cashed executing the following command info /var/www/html/cacti/rra/local_fail2ban_cur_banned_209.rrd.
2024-11-16 20:26:39 - RRDTOOL Error Backtrace: (/graph_json.php[168]:rrdtool_function_graph(), /lib/rrd.php[1747]:generate_graph_best_cf(), /lib/functions.php[3463]:get_rrd_cfs(), /lib/functions.php[3508]:rrdtool_execute(), /lib/rrd.php[284]:__rrd_execute(), /lib/rrd.php[396]:cacti_debug_backtrace())

morganfw added bug Undesired behaviour unverified Some days we don't have a clue labels Nov 12, 2024

TheWitness added a commit that referenced this issue Nov 13, 2024

QA: Error correction after testing related to #5915

b919466

* This may correct the issue with RRDtool. Not sure yet.

TheWitness removed the unverified Some days we don't have a clue label Nov 14, 2024

TheWitness added 3rd Party Bug 3rd party bug confirmed Bug is confirm by dev team graphview Issue related to the Graphs page and removed bug Undesired behaviour labels Nov 14, 2024

TheWitness closed this as completed in 5182c23 Nov 15, 2024

TheWitness added a commit that referenced this issue Nov 15, 2024

QA: Like rework on #5915 and #5923

bcf048d

This add's some additional logging and addresses one bogus return variable.

TheWitness added a commit that referenced this issue Nov 16, 2024

QA: #5915 - Minor logic change to assist polling

988921c

TheWitness added a commit that referenced this issue Nov 16, 2024

QA: Another fix for #5915

955bec5

This should catch the failed INFO calls.

TheWitness added a commit that referenced this issue Nov 16, 2024

QA: Add backtrace to #5915

bdad29d

This will help us figure out the source of the issues.

Cacti 1.3.0 - Dev b94bf9c3 - Graph Unknown RRDtool Error and Segmentation fault (core dumped) after enabling Thold Threshold #5915

Cacti 1.3.0 - Dev b94bf9c3 - Graph Unknown RRDtool Error and Segmentation fault (core dumped) after enabling Thold Threshold #5915

Comments

morganfw commented Nov 12, 2024 • edited Loading

TheWitness commented Nov 12, 2024

TheWitness commented Nov 12, 2024

morganfw commented Nov 12, 2024

TheWitness commented Nov 12, 2024

morganfw commented Nov 12, 2024

TheWitness commented Nov 12, 2024

TheWitness commented Nov 12, 2024

TheWitness commented Nov 12, 2024

morganfw commented Nov 13, 2024

morganfw commented Nov 13, 2024

TheWitness commented Nov 13, 2024

TheWitness commented Nov 13, 2024

TheWitness commented Nov 13, 2024

morganfw commented Nov 13, 2024

morganfw commented Nov 13, 2024

TheWitness commented Nov 13, 2024

TheWitness commented Nov 13, 2024

TheWitness commented Nov 13, 2024

morganfw commented Nov 13, 2024 • edited Loading

TheWitness commented Nov 13, 2024

TheWitness commented Nov 13, 2024

morganfw commented Nov 14, 2024

TheWitness commented Nov 14, 2024

TheWitness commented Nov 14, 2024

morganfw commented Nov 14, 2024

TheWitness commented Nov 14, 2024

TheWitness commented Nov 15, 2024

morganfw commented Nov 15, 2024

TheWitness commented Nov 15, 2024

morganfw commented Nov 15, 2024

morganfw commented Nov 15, 2024 • edited Loading

TheWitness commented Nov 15, 2024 • edited Loading

TheWitness commented Nov 15, 2024

morganfw commented Nov 15, 2024 • edited Loading

TheWitness commented Nov 15, 2024

morganfw commented Nov 16, 2024

morganfw commented Nov 16, 2024

TheWitness commented Nov 16, 2024

TheWitness commented Nov 16, 2024

TheWitness commented Nov 16, 2024

TheWitness commented Nov 16, 2024

TheWitness commented Nov 16, 2024

TheWitness commented Nov 16, 2024

morganfw commented Nov 16, 2024

morganfw commented Nov 12, 2024 •

edited

Loading

morganfw commented Nov 13, 2024 •

edited

Loading

morganfw commented Nov 15, 2024 •

edited

Loading

TheWitness commented Nov 15, 2024 •

edited

Loading

morganfw commented Nov 15, 2024 •

edited

Loading