externalTrafficPolicy of tidb service should default to local in terraform deployment

[aylei]

Bug Report

When the externalTrafficPolicy is set to cluster (the default value), every Kubernetes node is a valid backend for the load balancer, a single machine failure will impact part of new connections. More seriously, for immature clients that has a 0 connect-timeout (no timeout), the new connections that routed to the failure machine hangs forever.

With externalTrafficPolicy set to local, the nodes without backend pods route the traffic to blackhole, therefore, LB can mark these nodes as unhealthy when performs the initial check. local mode may cause unbalanced traffic, but not for our terraform deployment which uses dedicated node.

What did you do?

1. Deploy TiDB cluster via the AWS terraform scripts
2. Simulate inserts using mysqlcli
3. Terminate a PD instance from the AWS console

What did you expect to see?
No impact on the workload

What did you see instead?
mysqlcli hangs forever

cc @tennix @weekface

ref:
https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/
https://www.asykim.com/blog/deep-dive-into-kubernetes-external-traffic-policies

[sokada1221]

I'll quickly fix this one as it's related to our use case.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days

[sokada1221]

@aylei i can't close the ticket - can you?

[aylei]

Yes.
closed via #947