-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Reduce replication overhead #8283
Comments
The main entry points being discussed here are:
Overall a decent amount of time is spent in
Unfortunately the above callbacks generally spawn background processes which makes it a bit hard to see the amount of processing in each. This was from the profile at profile.svg.zip (and similar related profiles) captured with: |
@erikjohnston Curious if you have any thoughts on this? A few avenues that might be worth investigating:
|
I did some quick benchmarking of this and it seems that my assumption here is wrong. The list comprehension is faster than using a Benchmark scriptimport timeit
# number of runs over the data for each repeat
N_RUNS = 20
# number of times to repeat the runs. We'll take the fastest run.
N_REPEATS = 5
NO_DATA = []
SM_DATA = range(10)
LG_DATA = range(10000)
def func(it):
result = 0
for d in it:
result += d
return result
def lister(data):
func([d for d in data])
def mapper(data):
func(map(lambda d: d, data))
for f in [lister, mapper]:
print('Running {} benchmarks...'.format(f.__name__))
for d in [NO_DATA, SM_DATA, LG_DATA]:
times = timeit.repeat(lambda: f(d), number=N_RUNS, repeat=N_REPEATS)
best = min(times)
print(' For %d items: %i loops, best of %i: %f sec per loop' % (
len(d), N_RUNS, len(times), best / N_RUNS,
)) Benchmark results
|
I wonder how much of this is still relevant. Much of the replication handling code has been refactored in the last few months. /cc @erikjohnston |
This was based on looking at py-spy traces where we were spending a lot of time in replication. I don't think any of the recent changes have affected the behavior I saw above (it was pretty baked into our architecture), but would love to be wrong! 🎉 I believe I capture the above by profiling the dummy worker. |
I think we shuffled things around in this area a bit, but I imagine its still an issue. The test worker is idling at ~25% CPU. |
Possibly we can just close this in favour of #12461? |
That seems reasonable to me -- the other issue is more actionable. |
From some profiling of matrix.org, it seems that the overhead of replication (using Redis) is ~20 - 25% per process. This figure was gather via profiling (via py-spy) a worker that had no traffic going to it.
We should investigate what's going on here!
Spun out of #7670
The text was updated successfully, but these errors were encountered: