fix: list identities tuning

[harnash]

This change should improve the /identities endpoint performance. Right now it uses Eager() method to load identity model associations n times (where n is the number of identities fetched per request, default is 100). This causes 100+1 SQL queries. It is a tradeoff between memory and speed/queries. I found out that on our setup (yes, our DEV DB have significant lag due to some infrastructure setup but it is not crazy high ~100ms which can happen in some low tier setups as well).

Currently queries looks like this:

SELECT * FROM identities LIMIT, OFFSET;
SELECT * FROM identity_verifiable_addresses WHERE identity_id = id1;
SELECT * FROM identity_verifiable_addresses WHERE identity_id = id2;
SELECT * FROM identity_verifiable_addresses WHERE identity_id = id3;
...
SELECT * FROM identity_verifiable_addresses WHERE identity_id = id100;

We propose to use EagerPreload() which will do only one query to fetch associations.
Now the queries would look like this:

SELECT * FROM identities LIMIT, OFFSET;
SELECT * FROM identity_verifiable_addresses WHERE identity_id IN (id1, id2, id3,..., id100);

Performance on master:

Requests      [total, rate, throughput]         1500, 50.03, 0.19
Duration      [total, attack, wait]             42.616s, 29.98s, 12.636s
Latencies     [min, mean, 50, 90, 95, 99, max]  75.896µs, 5.553s, 3.831ms, 21.809s, 26.786s, 30.004s, 30.013s
Bytes In      [total, mean]                     34317, 22.88
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           0.53%
Status Codes  [code:count]                      0:1133  200:8  502:359

performance after the change:

Requests      [total, rate, throughput]         1500, 50.04, 42.72
Duration      [total, attack, wait]             34.269s, 29.978s, 4.291s
Latencies     [min, mean, 50, 90, 95, 99, max]  1.223ms, 3.693s, 2.874s, 6.784s, 8.782s, 16.303s, 23.191s
Bytes In      [total, mean]                     5557366, 3704.91
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           97.60%
Status Codes  [code:count]                      0:34  200:1464  502:2

Memory usage: https://ibb.co/2YS8sgP (those two bumps are tests on the branch with changes so I guess it does require more memory).

docs: https://gobuffalo.io/en/docs/db/relations#loading-associations

That said I'm not 100% sure this is a good change but it should help with high latency DBs and require more memory to operate (still using Argon2Id could mean Kratos will need more memory anyway). Your call!

Related issue(s)

I don't think there is an issue but I'm pretty sure someone mentioned this on Slack one day.

Checklist

• I have read the contributing guidelines.
• I am following the
  contributing code guidelines.
• I have read the security policy.
• I confirm that this pull request does not address a security
  vulnerability. If this pull request addresses a security. vulnerability, I
  confirm that I got green light (please contact
  security@ory.sh) from the maintainers to push
  the changes.
• I have added tests that prove my fix is effective or that my feature
  works.
• I have added or changed the documentation.

[codecov]

Codecov Report

Merging #1588 (f63ea11) into master (ca15200) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1588   +/-   ##
=======================================
  Coverage   74.17%   74.17%           
=======================================
  Files         257      257           
  Lines       12556    12556           
=======================================
  Hits         9313     9313           
  Misses       2624     2624           
  Partials      619      619           

Impacted Files	Coverage Δ
persistence/sql/persister_identity.go	63.29% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca15200...f63ea11. Read the comment docs.

[aeneasr]

Awesome! I am a pop maintainer and did not know about this functionality!