Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Endpoint#pending_outgoing_messages #10271

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Al2Klimov
Copy link
Member

@Al2Klimov Al2Klimov commented Dec 11, 2024

It works! 👍

> GET /v1/objects/endpoints?pretty=1 HTTP/1.1
> Host: 10.27.2.128:5665
> Authorization: Basic cm9vdDpmNjBjNTM5NzNkYmU1ZmJm
> User-Agent: curl/8.7.1
> Accept: application/json
> Content-Length: 39
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 39 bytes
< HTTP/1.1 200 OK
< Server: Icinga/r2.14.0-1
< Content-Type: application/json
< Content-Length: 461
<
{
    "results": [
        {
            "attrs": {
                "pending_outgoing_messages": 23
            },
            "joins": {},
            "meta": {},
            "name": "master1",
            "type": "Endpoint"
        },
        {
            "attrs": {
                "pending_outgoing_messages": 0
            },
            "joins": {},
            "meta": {},
            "name": "master2",
            "type": "Endpoint"
        }
    ]
}

The API request was fired during a present JSON RPC connection.

@Al2Klimov Al2Klimov added the area/distributed Distributed monitoring (master, satellites, clients) label Dec 11, 2024
@cla-bot cla-bot bot added the cla/signed label Dec 11, 2024
@Al2Klimov Al2Klimov linked an issue Dec 11, 2024 that may be closed by this pull request
@Al2Klimov
Copy link
Member Author

Colleagues, if you have enough dummy checks (mine: 320,000) with an IDO master and an Icinga DB one, the IDO one will accumulate pending writes:

[root@aklimov820479-1 ~]# while curl -fsSku root:f60c53973dbe11:21:01 [692/710] cept: application/json' -d '{"attrs":["pending_outgoing_messages"]}' 'https://1 0.27.2.188:5665/v1/objects/endpoints'; do sleep 5m; done {"results":[{"attrs":{"pending_outgoing_messages":0},"joins":{},"meta":{},"name ":"master1","type":"Endpoint"},{"attrs":{"pending_outgoing_messages":3033671}," joins":{},"meta":{},"name":"master2","type":"Endpoint"}]}{"results":[{"attrs":{ "pending_outgoing_messages":0},"joins":{},"meta":{},"name":"master1","type":"En dpoint"},{"attrs":{"pending_outgoing_messages":3331043},"joins":{},"meta":{},"n ame":"master2","type":"Endpoint"}]}{"results":[{"attrs":{"pending_outgoing_mess ages":0},"joins":{},"meta":{},"name":"master1","type":"Endpoint"},{"attrs":{"pe nding_outgoing_messages":3588325},"joins":{},"meta":{},"name":"master2","type": "Endpoint"}]}{"results":[{"attrs":{"pending_outgoing_messages":0},"joins":{},"m eta":{},"name":"master1","type":"Endpoint"},{"attrs":{"pending_outgoing_message s":3818835},"joins":{},"meta":{},"name":"master2","type":"Endpoint"}]}{"results ":[{"attrs":{"pending_outgoing_messages":0},"joins":{},"meta":{},"name":"master 1","type":"Endpoint"},{"attrs":{"pending_outgoing_messages":4083728},"joins":{} ,"meta":{},"name":"master2","type":"Endpoint"}]}{"results":[{"attrs":{"pending_ outgoing_messages":0},"joins":{},"meta":{},"name":"master1","type":"Endpoint"}, {"attrs":{"pending_outgoing_messages":4389195},"joins":{},"meta":{},"name":"mas ter2","type":"Endpoint"}]}{"results":[{"attrs":{"pending_outgoing_messages":0}, "joins":{},"meta":{},"name":"master1","type":"Endpoint"},{"attrs":{"pending_out going_messages":4627411},"joins":{},"meta":{},"name":"master2","type":"Endpoint "}]}{"results":[{"attrs":{"pending_outgoing_messages":0},"joins":{},"meta":{}," name":"master1","type":"Endpoint"},{"attrs":{"pending_outgoing_messages":482323 4},"joins":{},"meta":{},"name":"master2","type":"Endpoint"}]}{"results":[{"attr s":{"pending_outgoing_messages":0},"joins":{},"meta":{},"name":"master1","type" :"Endpoint"},{"attrs":{"pending_outgoing_messages":5001852},"joins":{},"meta":{ },"name":"master2","type":"Endpoint"}]}{"results":[{"attrs":{"pending_outgoing_ messages":0},"joins":{},"meta":{},"name":"master1","type":"Endpoint"},{"attrs": {"pending_outgoing_messages":5219258},"joins":{},"meta":{},"name":"master2","ty pe":"Endpoint"}]}{"results":[{"attrs":{"pending_outgoing_messages":0},"joins":{ },"meta":{},"name":"master1","type":"Endpoint"},{"attrs":{"pending_outgoing_mes sages":5501339},"joins":{},"meta":{},"name":"master2","type":"Endpoint"}]}{"res ults":[{"attrs":{"pending_outgoing_messages":0},"joins":{},"meta":{},"name":"ma ster1","type":"Endpoint"},{"attrs":{"pending_outgoing_messages":5722573},"joins ":{},"meta":{},"name":"master2","type":"Endpoint"}]}{"results":[{"attrs":{"pend ing_outgoing_messages":0},"joins":{},"meta":{},"name":"master1","type":"Endpoin t"},{"attrs":{"pending_outgoing_messages":5912916},"joins":{},"meta":{},"name": "master2","type":"Endpoint"}]}{"results":[{"attrs":{"pending_outgoing_messages" :0},"joins":{},"meta":{},"name":"master1","type":"Endpoint"},{"attrs":{"pending _outgoing_messages":6117871},"joins":{},"meta":{},"name":"master2","type":"Endp oint"}]}{"results":[{"attrs":{"pending_outgoing_messages":0},"joins":{},"meta": {},"name":"master1","type":"Endpoint"},{"attrs":{"pending_outgoing_messages":63 82224},"joins":{},"meta":{},"name":"master2","type":"Endpoint"}]}{"results":[{" attrs":{"pending_outgoing_messages":0},"joins":{},"meta":{},"name":"master1","t ype":"Endpoint"},{"attrs":{"pending_outgoing_messages":6697645},"joins":{},"met a":{},"name":"master2","type":"Endpoint"}]}{"results":[{"attrs":{"pending_outgo

Or pending reads on Icinga DB site. Idk so far. Don't forget to test with a non-default maxmemory policy for Redis and the Blackhole engine for the IDO schema (ex. schema table), or you'll run OOM.

@julianbrost
Copy link
Contributor

Please try to format this in a more readable way. It also contains copy & paste errors, for example -fsSku root:f60c53973dbe11:21:01 [692/710] cept: application/json'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/distributed Distributed monitoring (master, satellites, clients) cla/signed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

JSON-RPC outgoing messages queue metric
2 participants