[feature request] Add reconciliation loop to check route entries and vpn connections periodically.

[DrmagicE]

Currently, If failure happens while setting route table or IPsec connection, we will requeue the event and retry later.

raven/pkg/k8s/engine_controller.go

Lines 115 to 129 in a4ee18f

	func (c *EngineController) handleEventErr(err error, event interface{}) {
	if err == nil {
	c.queue.Forget(event)
	return
	}
	if c.queue.NumRequeues(event) < maxRetries {
	klog.Infof("error syncing event %v: %v", event, err)
	c.queue.AddRateLimited(event)
	return
	}
	utilruntime.HandleError(err)
	klog.Infof("dropping event %q out of the queue: %v", event, err)
	c.queue.Forget(event)
	}

This approach has some problems:

1. The re-entered event may break correctness.
2. If max retries exceeded, the event get lost forever.
3. If rules are deleted by the user accidentally, there is no way to recover. (unless to restart the agent)

To improve this, we need a reconciliation mechanism to compare the current state of the node and the desired state (sync from the informer) periodically and make any required changes if they are mismatched.