gc_thresh values update to support higher scale for arp and nd entries

[sumanbrcm]

- Why I did it
When sending a lot of ARP/ND requests in a burst, ARP entries are getting purged from the kernel while the later set of ARP entries was still getting added.
The sequence of add/remove is in such a way that we were never able to cross ~2400 entries .

- How I did it
In the kernel ARP module the following attributes govern how many ARP entries are key in the Kernel ARP cache

gc_thresh1
The minimum number of entries to keep in the ARP cache. The garbage collector will not run if there are fewer than this number of entries in the cache. Defaults to 128.

gc_thresh2
The soft maximum number of entries to keep in the ARP cache. The garbage collector will allow the number of entries to exceed this for 5 seconds before collection will be performed. Defaults to 512.

gc_thresh3
The hard maximum number of entries to keep in the ARP cache. The garbage collector will always run if there are more than this number of entries in the cache. Defaults to 1024.

To increase the number of ARP/ND entries these attributes will be changed to following values for IPv4 and IPv6

net.ipv4.neigh.default.gc_thresh1=16000
net.ipv4.neigh.default.gc_thresh2=32000
net.ipv4.neigh.default.gc_thresh3=48000

net.ipv6.neigh.default.gc_thresh1=8000
net.ipv6.neigh.default.gc_thresh2=16000
net.ipv6.neigh.default.gc_thresh3=32000

- How to verify it
Verification Tests done :

1. Arp
   root@sonic:# sudo cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
   32000
   root@sonic:# sudo cat /proc/sys/net/ipv4/neigh/default/gc_thresh2
   48000
   root@sonic:# sudo cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
   64000
   root@sonic:#
   root@sonic:# ip -4 neigh show | wc -l
   32011
   root@sonic:#
   root@sonic:# show arp | grep Total
   Total number of entries 32000
   root@sonic:#

2. ND
   root@sonic:# sudo cat /proc/sys/net/ipv6/neigh/default/gc_thresh1
   16000
   root@sonic:# sudo cat /proc/sys/net/ipv6/neigh/default/gc_thresh2
   32000
   root@sonic:# sudo cat /proc/sys/net/ipv6/neigh/default/gc_thresh3
   48000
   root@sonic:#
   root@sonic:# ip -6 neigh show | wc -l
   16076
   root@sonic:#
   root@sonic:# show ndp | grep Total
   Total number of entries 16007
   root@sonic:#

- Description for the changelog
Update of gc_thresh value to support 32k arp and 16k ND entries

[ben-gale]

@lguohan - can we get a reviewer for this pls?

[lguohan]

gc_thresh3 is the really the hard limit, and i remember the asic does not support those many entries in the hardware, then we will have table full problem. @prsunny, any more comments?

[ben-gale]

@lguohan, @prsunny - are you satisfied with the response on this? Do you need anything else? Thx

[lguohan]

does not look like the concern is addressed. we could still have table full issue. to fully address the concern we need a mechanism to configure on platform basis

[ben-gale]

@lguohan, @prsunny - I'm not sure how to proceed here. Generally SONiC does not impose HW table scale limits - today it is the job of the deployer to ensure that the hardware has the required capacity for their network. The intent of this PR is not to change this (e.g. fix this problem by having the kernel enforce a HW limit) - the intent is to get kernel garbage collection out of the way so that the HW can be used. For this reason a high value has been chosen that we believe is better for all silicon. Can the HW table overflow? Of course - it always could - we're talking about garbage collection here, and fundamental design point of SONiC is unchanged.

The more fundamental problem of table limit enforcement in SONiC is a much bigger problem that goes way beyond the scope of this PR. Broadcom tried to address this with the error reporting framework (to give the application feedback of a table insertion error), but this got hung up on Synchronous SAI and was rejected. Either way, many application changes are needed for a robust solution to this problem, and I don't think we should tie this PR to that.

Thoughts?