Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Icinga 2.9.0 / 2.10.0 incompatibility #6702

Closed
gagaha opened this issue Oct 18, 2018 · 4 comments
Closed

Icinga 2.9.0 / 2.10.0 incompatibility #6702

gagaha opened this issue Oct 18, 2018 · 4 comments
Labels
area/distributed Distributed monitoring (master, satellites, clients)

Comments

@gagaha
Copy link

gagaha commented Oct 18, 2018

Expected Behavior

Icinga agent (v 2.10.0) can successfully connect to its master (v 2.9.0)

Current Behavior

Icinga agent (v 2.10.0) can't connect to its master (v 2.9.0)

Possible Solution

Steps to Reproduce (for bugs)

  1. Setup a icinga master node (v2.9.0)
  2. Setup a icinga agent node (v2.9.2)
  3. Connect them with each other via PKI
  4. Update icinga agent node to v2.10.0

Context

Customer updated his icinga agent, since then the agent node will not connect with the master.
After downgrading back to 2.9.0 the connection could be established.

Your Environment

  • Version used (icinga2 --version): Master v2.9.0, (Customer) Agent v2.10.0
  • Operating System and version:
    Master: Ubuntu 14.04.5 LTS
    Agent: Ubuntu 16.04.5 LTS
  • Enabled features (icinga2 feature list):
    Master: api checker command graphite ido-mysql mainlog notification
    Agent: checker mainlog notification
  • Icinga Web 2 version and modules (System - About):
  • Config validation (icinga2 daemon -C):
[2018-10-18 12:05:59 +0000] information/cli: Icinga application loader (version: r2.9.0-1)
[2018-10-18 12:05:59 +0000] information/cli: Loading configuration file(s).
[2018-10-18 12:05:59 +0000] information/ConfigItem: Committing config item(s).
[2018-10-18 12:05:59 +0000] information/ApiListener: My API identity: $icinga-master-node
[2018-10-18 12:05:59 +0000] warning/ApplyRule: Apply rule 'swap' (in /var/lib/icinga2/api/packages/director/0c946858-a33e-42e3-8d59-fa5535441c1e/zones.d/director-global/service_apply.conf: 59:1-59:20) for type 'Service' does not match anywhere!
[2018-10-18 12:05:59 +0000] warning/ApplyRule: Apply rule 'yum' (in /var/lib/icinga2/api/packages/director/0c946858-a33e-42e3-8d59-fa5535441c1e/zones.d/director-global/service_apply.conf: 75:1-75:19) for type 'Service' does not match anywhere!
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 132 Services.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 ServiceGroup.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 6 HostGroups.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 8 NotificationCommands.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 456 Notifications.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 20 Hosts.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 Comment.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 18 Zones.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 17 Endpoints.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 2 ApiUsers.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 6 UserGroups.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 216 CheckCommands.
[2018-10-18 12:05:59 +0000] information/ConfigItem: Instantiated 2 Users.
[2018-10-18 12:05:59 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2018-10-18 12:05:59 +0000] information/cli: Finished validating the configuration file(s).

Agent Logs:

[2018-10-17 21:22:30 +0000] debug/ThreadPool: Spawning worker thread.
[2018-10-17 21:22:30 +0000] debug/ApiListener: Not connecting to Endpoint '$icinga-agent-node' because that's us.
[2018-10-17 21:22:30 +0000] notice/ApiListener: Current zone master: $icinga-agent-node
[2018-10-17 21:22:30 +0000] notice/ApiListener: Connected endpoints: 
[2018-10-17 21:22:30 +0000] information/ApiListener: Reconnecting to endpoint '$icinga-master-node' via host '$icinga-master-node' and port '5665'
[2018-10-17 21:22:30 +0000] warning/TlsStream: TLS stream was disconnected.
[2018-10-17 21:22:30 +0000] critical/ApiListener: Client TLS handshake failed (to [$icinga-master-node-ip]:5665): Error: Socket was closed during TLS handshake.

	(0) Handling new API client connection

Context:
	(0) Handling new API client connection

[2018-10-17 21:22:30 +0000] information/ApiListener: Finished reconnecting to endpoint '$icinga-master-node' via host '$icinga-master-node' and port '5665'
[2018-10-17 21:22:30 +0000] notice/ThreadPool: Thread pool; current: 2; adjustment: -1
[2018-10-17 21:22:30 +0000] debug/ThreadPool: Killing worker thread.
[2018-10-17 21:22:40 +0000] debug/ThreadPool: Spawning worker thread.
[2018-10-17 21:22:40 +0000] debug/ApiListener: Not connecting to Endpoint '$icinga-agent-node' because that's us.
[2018-10-17 21:22:40 +0000] notice/ApiListener: Current zone master: $icinga-agent-node
[2018-10-17 21:22:40 +0000] notice/ApiListener: Connected endpoints: 
[2018-10-17 21:22:40 +0000] warning/TlsStream: TLS stream was disconnected.
[2018-10-17 21:22:40 +0000] critical/ApiListener: Client TLS handshake failed (to [$icinga-master-node-ip]:5665): Error: Socket was closed during TLS handshake.

	(0) Handling new API client connection

Context:
	(0) Handling new API client connection

[2018-10-17 21:22:40 +0000] information/ApiListener: Finished reconnecting to endpoint '$icinga-master-node' via host '$icinga-master-node' and port '5665'
[2018-10-17 21:22:40 +0000] notice/ThreadPool: Thread pool; current: 2; adjustment: -1
[2018-10-17 21:22:40 +0000] debug/ThreadPool: Killing worker thread.
[2018-10-17 21:22:40 +0000] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.00499832%
[2018-10-17 21:22:40 +0000] notice/ThreadPool: Pool #2: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.218846%

Master Logs:

Context:
        (0) Handling new API client connection

[2018-10-17 12:37:27 +0000] warning/JsonRpcConnection: API client disconnected for identity '$icinga-agent-node'
[2018-10-17 12:37:27 +0000] warning/ApiListener: Removing API client for endpoint '$icinga-agent-node'. 0 API clients left.
[2018-10-17 12:37:31 +0000] warning/TlsStream: TLS stream was disconnected.
[2018-10-17 12:37:31 +0000] critical/ApiListener: Client TLS handshake failed (from [$icinga-master-web]:55064)
@dnsmichi
Copy link
Contributor

Seems the master does something weird here. The master is Ubuntu 14.04, and 2.9.0 which contains a problem with daemonize/fork where no further socket events are processed. I'd recommend to upgrade the master to at least 2.9.2, yet better 2.10.1.

See #6445 for details.

@dnsmichi dnsmichi added area/distributed Distributed monitoring (master, satellites, clients) needs feedback We'll only proceed once we hear from you again labels Oct 18, 2018
@dnsmichi
Copy link
Contributor

Might also be related to #6738.

@gagaha
Copy link
Author

gagaha commented Oct 31, 2018

Upgrading the Master to 2.10.1 has solved the problem. Thanks!

@dnsmichi
Copy link
Contributor

Arschgeil :-)

@dnsmichi dnsmichi removed the needs feedback We'll only proceed once we hear from you again label Oct 31, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/distributed Distributed monitoring (master, satellites, clients)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dnsmichi @gagaha