docs: include tuning of free backend slots in performance suggestions

[ssanders1449]

I was unable to complete a Proof of Concept demonstration of haproxy-ingress (due to poor performance) until I increased backend-server-slots-increment for my active workloads

As soon as I made that change, the POC passed beautifully

Add this to the 'How to improve performance' section.

[jcmoraisjr]

Hi thanks for sharing this info! What was the behavior (what means poor performance) with the default config? What was the final configuration that improves the performance? I'm asking this to suggest a few more details in the doc and also to verify if it worth to tune the defaults a bit more. If you consider an easily reproducible scenario and can share the details, I can also try to reproduce the behavior myself and help to suggest something.

[ssanders1449]

I was doing spike testing. I have 150 different backends, most are fairly stable in the number of pods, but some of the backends were scaling up very rapidly. I didn't want to change –reload-interval because I needed to start sending requests to the new backend pods as soon as possible in order to handle the load.

What I saw was that haproxy-ingress was reloading haproxy much too often. Eventually I saw that haproxy could not complete a connection to some of the backend within 1.5 seconds (which is my SLA). I didn't investigate fully why the excessive amount of reloads caused this connection timeout, but it is fairly obvious that reducing the #reloads helps performance.

I was about to give up and try increasing –reload-interval, but then I re-read the documentation one more time and saw the backend-server-slots-increment parameter. By increasing it to 100 for my active backends, the connection timeouts went away. Based on your response to me in slack, I probably could also have increased slots-min-free (https://kubernetes.slack.com/archives/C01D8GKKT5H/p1641428990000500?thread_ts=1641395608.182500&cid=C01D8GKKT5H)

I think that the default behavior of adding 6 empty slots per reload is probably fine. I just missed the option to change it when I was reading the FAQ, so I thought it would be nice to add.

[jcmoraisjr]

Thanks for the clarification, it seems good, merging now. Doc page will be updated in the next v0.13 tag.