v2.0.7 Improvements

Improvements

1. Stability and accuracy of network change detection
2. Logstash pipeline changed accordingly to new watcher output
3. Documentation
4. Tools (client.py)

v2.0 Architecture changes

Changes

OSPF Watcher doesn't keep any OSPF debug messages anymore, but tracks OSPF changes via listening raw packets on FRR's interface. As a result it brings the following benefits:

1. doesn't create extra load on FRR
2. there is no dependency on FRR version
3. we shouldn't bother about disk space because of OSPF dump file size

Current architecture:

FRR establishes OSPF adjacency, Watcher passively listen to packets coming to FRR

Additionally, OSPF Watcher stability increased drastically. The following Watcher's output on network changes in containerlab

router4(config-if)# shutdown
2024-07-22T20:20:04Z,watcher-local,host,10.10.10.4,down,10.10.10.5,,
2024-07-22T20:20:04Z,watcher-local,metric,10.10.10.4,changed,old_cost:10,new_cost:-1,10.10.10.5,,

router4(config-if)# no shutdown
2024-07-22T21:54:49Z,watcher-local,host,10.10.10.4,up,10.10.10.5,,
2024-07-22T21:54:59Z,watcher-local,host,10.10.10.5,up,10.10.10.4,,
2024-07-22T21:54:59Z,watcher-local,metric,10.10.10.5,changed,old_cost:-1,new_cost:10,10.10.10.4,,

router4(config-if)# ip ospf cost 4444
2024-07-22T20:19:05Z,watcher-local,metric,10.10.10.5,changed,old_cost:10,new_cost:4444,10.10.10.4,,

router5(config)# ip route 8.8.0.64/30 Null0
2024-07-22T20:24:08Z,watcher-local,network,8.8.0.60/30,changed,old_cost:-1,new_cost:12,10.10.10.5,,,external,1
2024-07-22T20:24:08Z,watcher-local,network,8.8.0.60/30,up,10.10.10.5,,

router4(config-router)# redistribute static metric 444 metric-type 2
2024-07-22T21:04:23Z,watcher-local,network,4.4.4.10/32,changed,old_cost:-1,new_cost:444,10.10.10.4,,,external,2
2024-07-22T21:04:23Z,watcher-local,network,4.4.4.10/32,up,10.10.10.4,,

router2(config-if)# shutdown
2024-07-20T13:42:43Z,watcher-local,host,10.10.10.2,down,10.10.10.3,,
2024-07-22T20:28:06Z,watcher-local,metric,10.10.10.2,changed,old_cost:10,new_cost:-1,10.10.10.3,,
2024-07-22T20:28:06Z,watcher-local,network,192.168.23.0/24,down,10.10.10.3,,
2024-07-22T20:28:06Z,watcher-local,network,192.168.23.0/24,changed,old_cost:10,new_cost:-1,10.10.10.3,,,internal,0

router2(config-if)# no shutdown
2024-07-22T20:29:33Z,watcher-local,network,192.168.23.0/24,up,10.10.10.3,,
2024-07-22T20:29:33Z,watcher-local,network,192.168.23.0/24,changed,old_cost:-1,new_cost:10,10.10.10.3,,,internal,0
2024-07-22T20:29:43Z,watcher-local,host,10.10.10.2,up,10.10.10.3,,
2024-07-22T20:29:43Z,watcher-local,metric,10.10.10.2,changed,old_cost:-1,new_cost:10,10.10.10.3,,
2024-07-22T20:29:53Z,watcher-local,host,10.10.10.3,up,10.10.10.2,,
2024-07-22T20:29:53Z,watcher-local,metric,10.10.10.3,changed,old_cost:-1,new_cost:10,10.10.10.2,,

router2(config-if)# ip ospf cost 222
2024-07-22T21:55:32Z,watcher-local,metric,10.10.10.3,changed,old_cost:10,new_cost:222,10.10.10.2,,
2024-07-22T21:55:32Z,watcher-local,network,192.168.23.0/24,changed,old_cost:10,new_cost:222,10.10.10.2,,,internal,0

v1.7 Multi-Area Monitoring from a Single Node

Multi-Area Monitoring from a Single Node! 🌐🔍

🔹 What’s New? OSPF and IS-IS Watcher now supports multi-area monitoring directly from a single node. This powerful feature allows network administrators to keep an eye on multiple OSPF/IS-IS areas without the need for complex setups.

🔹 How It Works:

Each Watcher instance maintains all routes and updates within an isolated network namespace. This isolation ensures safety and efficient monitoring without interference.
Thanks to containerlab, managing Watcher instances has never been easier. Containerlab handles the heavy lifting behind the scenes, making it a seamless experience for network operators.
🔹 Benefits:

1. Simplified monitoring: No need to set up separate monitoring nodes for each area.
2. Centralized view: Monitor all OSPF/IS-IS areas from a single point.
3. Safety first: Isolated namespaces prevent route leaks.

AS number is added

Optional ASN env variable is added. Logstash index templates are needed to be updated.

v1.6

Logstash pipeline bugfix

Based on feedback #6 and #7 it seems that pipeline-to-pipeline communication doesn't work as expected. Based on tests - it was discovered a lot of duplicated entities, so pipeline-to-pipeline communication was replaced by keeping all output plugins in a single file.

docker exec -it mongo /bin/bash
mongo mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@mongodb:27017/admin?gssapiServiceName=mongodb
use admins
> db.adj_change.find({"graph_time" : "28Oct2023_01h10m00s_7_hosts_ospfwatcher", "event_object" : "10.88.0.208/29"}).count()
0

so there is no event about 10.88.0.208/29. Emulate that network is up.

docker exec -it watcher /bin/bash
echo "2023-10-27T07:50:24Z,demo-watcher,network,10.88.0.208/29,up,10.17.1.4,28Oct2023_01h10m00s_7_hosts_ospfwatcher" >> /home/watcher/watcher/logs/watcher.log

> db.adj_change.find({"graph_time" : "28Oct2023_01h10m00s_7_hosts_ospfwatcher", "event_object" : "10.88.0.208/29"}).count()
17

17 duplicated records. With all-in-one file we get a single record per each event.

v1.5

Instance notification is available in OSPF Watcher

Slack notification

HTTP POST messages can be easily accepted by messengers, which allows to get instant notifications of OSPF topology changes:

WebHook setting

1. Create a Slack app
2. Enable Incoming Webhooks
3. Create an Incoming Webhook (generates URL)
4. Uncomment EXPORT_TO_WEBHOOK_URL_BOOL in .env, set the URL to WEBHOOK_URL

v1.4

OSPF Watcher is integrated with Zabbix

Monitoring/alerting based on topology changes in OSPF domain

OSPF topology changes are exported to Zabbix.

Zabbix OSPF neighbor up/down alarm

This alarm tracks all new OSPF adjacencies or when device loses its OSPF neighbor

Zabbix OSPF Cost changed on transit links

Transit links are all links between active OSPF neighbors. If cost on a link was changed it might affect all actual/shortest paths traffic follows

Zabbix alert if OSPF network was stopped announcing from node

If a subnet was removed from OSPF node (the node withdrew it from the announcement) it means the network from this node became unavailable for others, this event will be logged too.

Zabbix settings

Zabbix settings are available here /docs/zabbix-ui. There are 4 hosts and items (host and item inside each host has the same names) are required:

• ospf_neighbor_up_down
• ospf_network_up_down
• ospf_link_cost_change
• ospf_stub_network_cost_change

v1.3

Reusage watcher container by logstash-index-creator. #4

v1.1

OSPF Watcher is integrated into Topolograph

OSPF Monitoring page

The solution is available only on docker-based Topolograph. It requires launching topolograph-docker and ospfwatcher on your on-premise host. Right after it OSPF topology changes will be logged into Topolograph's DB and ELK. Topolograph's OSPFWatcher page provides the following options:

• filter logs by start and end time
• filter network event by new/old subnets, up/down links, cost changes events

Demo logs

OSPF Monitoring page is prefilled by demo logs.
Event on timeline dashboard is clickable - the historical event is shown on the graph too. For example if we choose the link down event between 10.1.1.2 and 10.1.1.4 on the timeline dashboard, we see an appropriate link on the graph. It means that this link went down at 8:26 AM UTC.
All logs are grouped by the detected device.

Filter any subnet-related events

Left only up/down link events

Enhancements

Define all variables in single env file

1.0

The initial release