Performance issue with many groups, too many SQL queries #4461
Assignees
Labels
Comments
This was referenced Aug 12, 2019
|
That linked issue should not apply for groups. Can you run |
|
Ok then, I ran this command in the awx_task container: But here it states, that it enables "slowlog" for queries that take longer than 2 seconds. What I observed, were ten thousand queries fired in rapid succession. So this output is useless I'm afraid. I'll paste the query log from Postgres itself below. |
|
Postgres log: Note this is 11 thousand queries from one job only ... |
|
|
Thanks I would like to reproduce this on my machine. Are you running AWX 6.1.0 or an earlier version? What are the general metrics of your inventory? Was it imported from a cloud source? Is it a smart inventory? What is the approximate number of groups? Ballpark, what fraction of them are populated with hosts, and are a lot of groups children of other groups? |
|
6.1.0, latest greatest! Groups: Hosts: |
|
We hit the same issue, when there is 236 groups for the inventory update. |
|
I'm able to reproduce this:
#! /usr/bin/env python
import json
data = {
"_meta": {
"hostvars": {}
},
}
for i in range(10000):
data['Group %d' % i] = {'hosts': ['localhost']}
print(json.dumps(data))
|
|
@ryanpetrello I've tried to test but I couldn't make it work. Pulled master and then used --- a/installer/inventory
+++ b/installer/inventory
@@ -6,8 +6,8 @@
# Remove these lines if you want to run a local image build
# Otherwise the setup playbook will install the official Ansible images. Versions may
# be selected based on: latest, 1, 1.0, 1.0.0, 1.0.0.123
# by default the base will be used to search for ansible/awx_web and ansible/awx_task
-dockerhub_base=ansible
-dockerhub_version=6.1.0
+#dockerhub_base=ansible
+#dockerhub_version=6.1.0The result was this strange error in the Job output:
|
|
Do you have an easier way to test this change? |
|
@towolf that failure looks unrelated to this issue/PR. You might have better luck just installing 6.1.0 and applying this diff manually: https://github.com/ansible/awx/pull/4487/files It's just a single line and you should see what you need in |
|
@ryanpetrello I got it working. I thought the awx_task container would reset if I edit the files in it, but they are persistent, in fact, so a restart allowed me to test. With query logging enabled the wait was reduced from 60 seconds to 15-20 seconds, before any text is printed in the output pane. So that looks decent, thank you. I could still log 900 DB queries in between starting the job and when the fitst output is printed. But those |
I would expect less than 200 from numbers I had from #3883... or at least less than 300 including the sync. If this is due to the cancel callback, then there's nothing we can do about that. It is expected that a job will regularly check the database for a signal from the user to cancel. If there are repetitive queries being shown other than that, then I would like to have a look. |
|
Tried this out....there is still definitely a delay, but it is under 30 seconds. Would like to reproduce original bug (tried out on current devel that has fix). Also not quite sure how to get the postgres logs in the development environment. Will catch up with you about this @ryanpetrello |
|
Tried out again with #4461 (comment) , still works for me, still seeing the more reasonable delay. Going to close. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ISSUE TYPE
SUMMARY
Same Issue as #3214, only this time please handle groups also
AWX is crunching through all groups with 10,000 queries for us. Before the job even starts for real. For 60 seconds.
The text was updated successfully, but these errors were encountered: