Route updates don't work in when two routes have same value for one of the route fields

[nnbu]

Describe the bug
When we change the routes (e.g. delete some of the routes) from the route table CR, the CR status goes to out of sync for the remaining routes.

Steps to reproduce
This happens in specific case where route comparison logic does not work. E.g. when we have two routes with say, same vpcPeeringConnectionID.

1. Create following route table CR. Note that there are 2 routes with different destinationCIDRBlock , but same vpcPeeringConnectionID.

apiVersion: ec2.services.k8s.aws/v1alpha1
kind: RouteTable
metadata:
  name: rt1
spec:
  routes:
  - destinationCIDRBlock: 172.29.0.0/16
    gatewayID: igw-<xxx>
  - destinationCIDRBlock: 172.30.0.0/16
    vpcPeeringConnectionID: pcx-05f31992c7ca70be5
  - destinationCIDRBlock: 172.28.0.0/16
     vpcPeeringConnectionID: pcx-05f31992c7ca70be5
  tags:
  - key: Name
    value: abc
  vpcID: vpc-<xxxx>

2. All 3 routes would get created correctly in aws and CR status will be sync
3. Delete route 172.29.0.0/16 (without changing two routes with same vpcPeeringConnectionID ). This triggers update and controller tries to find what needs to be deleted/added.
4. CR goes to out of sync state with error something like

RouteAlreadyExists: The route identified by 172.30.0.0/16 already exists.\n\tstatus
      code: 400, request id: a471eeab-5aa3-43c7-a146-ceb380c6b50f

This suggests that update tried to create route 172.30.0.0/16 and failed with RouteAlreadyExists error.

On prima facie, there appears to be two issues

1. getMatchingRoute logic has issues when there is one field (vpcPeeringConnectionID) having same value for two routes. Due to this same value, it thinks, the route is getting updated from 172.28.0.0/16 to 172.30.0.0/16. So, it tries add 172.30.0.0/16 and fails with the error.

Ideally, this logic should not have determined any update for these two routes with vpcPeeringConnectionID.

2. When update determines that there is a route to be deleted and added, it creates toAdd and toDelete lists.
   Ideally, delete should happen first and then add should happen later. But current logic is reverse.

Expected outcome
CR should not go into out of sync state.

Environment

• Kubernetes version
• Using EKS (yes/no), if so version?
• AWS service targeted (S3, RDS, etc.)

[nnbu]

In fact, this happens when any kind of update is performed on the CR.
e.g. If we add a new tag (keeping routes section untouched), that also triggers the update. and getMatchingRoute logic fails to understand routes have not been modified. Result is that CR goes out of sync with the above mentioned error.

So, as long as we have two routes with same value for one of the fields (in this case, vpcPeeringConnectionID field), we hit this issue

[a-hilaly]

@nnbu I guess this implementation is not correct https://github.com/aws-controllers-k8s/ec2-controller/blob/main/pkg/resource/route_table/sdk.go#L613