Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvements #115

Merged
merged 188 commits into from
Jun 9, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
188 commits
Select commit Hold shift + click to select a range
6d9c979
prefetch solves
Apr 6, 2021
f41d64e
actually prefetch solves
Apr 6, 2021
7aa9698
reorder should_deny_admin
Apr 6, 2021
d478254
add cachalot to poetry
Bentechy66 Apr 6, 2021
6f14d2f
add cachalot to INSTALLED_APPS
Bentechy66 Apr 6, 2021
532e23f
Merge remote-tracking branch 'origin/performace/zoom' into performace…
Bentechy66 Apr 6, 2021
7196ae2
spelling is hard
Bentechy66 Apr 6, 2021
47b70a7
2 hour wait -> 15 minute wait
Bentechy66 Apr 7, 2021
c8cd21a
log how long stuff took
Bentechy66 Apr 7, 2021
2bcfba3
remove auto_unlock, move some query logic into python
Bentechy66 Apr 8, 2021
a1ff372
stonking optimisation to challenges
Bentechy66 Apr 8, 2021
768f64a
haha locked challenges exist i totally didnt forget those exist 🤪
Bentechy66 Apr 8, 2021
852e299
remove one query from every pageload lol
Bentechy66 Apr 10, 2021
b59bd4d
proof of concept custom serialization
Apr 12, 2021
1ed429b
revert to serpy with solve count optimisation
Apr 12, 2021
220bb7f
solve counts go brrrrr
Apr 12, 2021
87ff950
ZOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOM
Apr 12, 2021
8893f34
more zoom
Apr 12, 2021
8d3a80e
simplify queryset
Apr 12, 2021
a8051ed
disable rate limits for testing
Apr 12, 2021
11e10a3
cache leaderboard graph
Apr 12, 2021
ee0f4a9
~100x speedup on stats full
Apr 12, 2021
a9f81d5
significant stats speedup
Apr 12, 2021
33e1e78
move raw sql out of serializers.py
Apr 12, 2021
b5c6c1d
make more tests pass
Apr 12, 2021
afdbfda
add config value to disable caching
Apr 12, 2021
e187687
Reformat config.py imports so they no longer break everything
jerbob Apr 13, 2021
9b9d60e
Add autoflake to dev dependencies
jerbob Apr 13, 2021
2d508e0
Remove unused imports, since apparently PyCharm doesn't support linters
jerbob Apr 13, 2021
bb46c83
Re-add the {silly, dullard, person with Down Syndrome, person with de…
jerbob Apr 13, 2021
6be9b19
Add serpy to lockfile
jerbob Apr 13, 2021
cf2f96a
Co-authored-by: Jeremiah Boby <mail@jeremiahboby.me>
May 28, 2021
7bbb18b
tests pass pog
Jun 2, 2021
8b0d644
thank you pycharm
Jun 2, 2021
8f2c127
Disable Poetry preview
thebeanogamer Jun 2, 2021
0300ed3
Add config LazyLoader
jerbob Jun 3, 2021
6734ace
Remove deprecated 'assertEquals' calls
jerbob Jun 3, 2021
2111f4b
Remove deprecated 'assertNotEquals' calls
jerbob Jun 3, 2021
f75deac
Update pyproject.toml
jerbob Jun 3, 2021
ca97f03
Modularize tests
jerbob Jun 3, 2021
05ffc49
Update references to challenge.tests.mixins
jerbob Jun 3, 2021
2a46d0d
Update mixins.py
jerbob Jun 3, 2021
073c369
Bump django-zxcvbn-password-validator
jerbob Jun 3, 2021
07d54ee
Start work on removing challenge test order requirement
jerbob Jun 3, 2021
8d751b5
load config defaults properly
Jun 4, 2021
ba069f9
remove debug thing
Jun 4, 2021
6e27b8a
extract common code out of serializers
Jun 4, 2021
dbbd7fc
cache invalidation
Jun 4, 2021
db67d55
delete user if email sending fails
Jun 4, 2021
40d66c4
make silk an environment variable
Jun 4, 2021
701d80d
move config defaults
Jun 4, 2021
a2344cf
Merge branch 'master' into performace/zoom
jerbob Jun 4, 2021
35c921b
Re-lock poetry lockfile
jerbob Jun 6, 2021
ba12b7b
Update decay.py
jerbob Jun 6, 2021
77b846a
Merge diverging migrations
jerbob Jun 6, 2021
4dc17b2
Add newline at end of apps.py
jerbob Jun 6, 2021
73566ac
Remove re-definition of prometheus Gauges
jerbob Jun 6, 2021
cad9085
Pull Gauge counts from cache directly after set
jerbob Jun 6, 2021
024c3cb
Update decay plugin to no longer import from model namespace
jerbob Jun 6, 2021
ba9de78
Remove faulty migration operation
jerbob Jun 6, 2021
ba6cbae
Remove old challenge/tests.py
jerbob Jun 6, 2021
f97789b
run black
Jun 7, 2021
b3de044
cleanup some merge conflict stuff
Jun 7, 2021
664fcb6
refactor config
Jun 7, 2021
3052d26
refactor config
Jun 7, 2021
735c647
Add isort to dev dependencies
jerbob Jun 7, 2021
5f5c2c9
Run isort to remove unused imports
jerbob Jun 7, 2021
9e6b00e
Add flake8 to dev dependencies
jerbob Jun 7, 2021
b3a6d2c
Add pre-commit to dev dependencies
jerbob Jun 7, 2021
66861b6
Remove explicit 'Twisted' pin
jerbob Jun 7, 2021
edb8cb5
Add .pre-commit-config.yaml
jerbob Jun 7, 2021
8a0165b
Add backend/settings/lint.py
jerbob Jun 7, 2021
df69694
Run black
jerbob Jun 7, 2021
0f00f9e
Use 'black' profile for isort
jerbob Jun 7, 2021
51b1f74
Add isort configuration to pyproject.toml
jerbob Jun 7, 2021
c0fbdef
Add Makefile
jerbob Jun 7, 2021
2c9f1f2
Replace flake8 with flake9
jerbob Jun 7, 2021
6650e27
Add flake9 configuration to pyproject.toml
jerbob Jun 7, 2021
831cf1e
Update pyproject.toml
jerbob Jun 7, 2021
0853530
Use importlib for weird imports
jerbob Jun 7, 2021
4570e99
Remove redundant assignment
jerbob Jun 7, 2021
3aeeff3
Remove redundant assignment
jerbob Jun 7, 2021
b9a888d
Remove import at bottom of file in stats/apps.py
jerbob Jun 7, 2021
7592d10
Remove sockets/models.py
jerbob Jun 7, 2021
d4dd3f4
Remove redundant re-imports of config.config
jerbob Jun 7, 2021
20b52cf
Remove redundant assignments of 'response'
jerbob Jun 7, 2021
db478eb
Remove admin/admin.py
jerbob Jun 7, 2021
6125027
Remove unused 'render' import
jerbob Jun 7, 2021
a49bb38
Remove redundant assignments of 'response'
jerbob Jun 7, 2021
132c0fb
Remove redundant assignments of 'response'
jerbob Jun 7, 2021
6221709
Replace bare except with 'except SMTPException'
jerbob Jun 7, 2021
49ab464
Use file-based cache for linting
jerbob Jun 7, 2021
c305d66
Update cache file name
jerbob Jun 7, 2021
24392a5
Update lint.py
jerbob Jun 7, 2021
46dab0d
Add better-exceptions to dev dependencies
jerbob Jun 7, 2021
87c7560
Update Makefile
jerbob Jun 7, 2021
2dc5ffe
Use CachedBackend for local linting
jerbob Jun 7, 2021
f3a5414
Update exceptions to catch for migration checking
jerbob Jun 7, 2021
7f3da83
Update exceptions to catch for migration checking
jerbob Jun 7, 2021
efc13d2
Replace test checker bodge in migrations with database check
jerbob Jun 7, 2021
4cafe14
Replace test checker bodge in migrations with database check
jerbob Jun 7, 2021
2860c76
export DJANGO_SETTINGS_MODULE in Makefile
jerbob Jun 7, 2021
7653818
Use importlib to import stats.signals
jerbob Jun 7, 2021
fdc126a
Remove test requirement on precommit :(
jerbob Jun 7, 2021
28094b7
Update pytest config in pyproject.toml
jerbob Jun 7, 2021
d9f1b8d
Update CHANNEL_LAYERS in lint settings
jerbob Jun 7, 2021
8efab7c
Add 'make format' precommit hook
jerbob Jun 7, 2021
d8042bb
Update Makefile
jerbob Jun 7, 2021
b96ba4a
Add DEFAULT_AUTO_FIELD
jerbob Jun 7, 2021
39a5997
Update warnings filter in pyproject.toml
jerbob Jun 7, 2021
d3ddfb4
Replace deprecated assertEquals with assertEqual
jerbob Jun 7, 2021
5564a57
Amend typo in Challenge mixin docstring
jerbob Jun 8, 2021
da7756e
Run format pre-commit hook before lint
jerbob Jun 8, 2021
925e82c
Bubble through all errors outside custom Exceptions
jerbob Jun 8, 2021
a3c9ac0
Add generic server error handler to backend/exception_handler.py
jerbob Jun 8, 2021
a62e14b
Add patch for serpy context
jerbob Jun 8, 2021
b6834c7
Default to the settings file with no dependencies
jerbob Jun 8, 2021
b739606
Don't add BrinIndex if we're not using PostgreSQL
jerbob Jun 8, 2021
db65f70
Update assertFalse statement in test_category_list_challenge_unlocked…
jerbob Jun 8, 2021
8bdd153
Refactor find_challenge_entry method
jerbob Jun 8, 2021
fd3ee92
Update mixins.py
jerbob Jun 8, 2021
d51b1f0
Specify custom generic error view
jerbob Jun 8, 2021
bc15595
fix some tests
Jun 8, 2021
f8f357f
Fix a failing test
Bentechy66 Jun 8, 2021
befa1cc
fix challenge tests
Bentechy66 Jun 8, 2021
1f4e63e
send metadata
Bentechy66 Jun 8, 2021
b3b9e97
shut beano up
Bentechy66 Jun 8, 2021
688bb7a
Add pytest testmon to project dependencies
jerbob Jun 8, 2021
29c15a8
Add testmon argument to pytest run
jerbob Jun 8, 2021
18e63d7
Reconfigure coverage.py fail-under
jerbob Jun 8, 2021
29450d6
Add coverage arguments to Makefile
jerbob Jun 8, 2021
09a6e5b
update .gitignore
jerbob Jun 8, 2021
37f4b78
Use --testmon in Makefile test command
jerbob Jun 8, 2021
22365eb
Add 'make test' to pre-commit hooks
jerbob Jun 8, 2021
1f054ca
Update Makefile
jerbob Jun 8, 2021
4439e5b
fix first bloods
Bentechy66 Jun 8, 2021
101fd1a
Merge remote-tracking branch 'origin/feature/optimisations' into feat…
Bentechy66 Jun 8, 2021
223217a
Add test case for leaving a team as non-owner.
jchristgit Jun 8, 2021
a61a5da
Exclude migrations from coverage calculation.
jchristgit Jun 8, 2021
90ccd79
Fix typo for gunicorn configuration path.
jchristgit Jun 8, 2021
47d1a1c
https://www.youtube.com/watch?v=LDU_Txk06tM
Jun 8, 2021
791f0c5
Add 'coverage' step to Makefile
jerbob Jun 8, 2021
11593f8
Lower fail_under percentage for coverage
jerbob Jun 8, 2021
19a9a9f
add html coverage to the makefile
Jun 8, 2021
9d93b9a
Update Makefile
jerbob Jun 8, 2021
2a9fb29
Kill unused module.
jchristgit Jun 8, 2021
92fe59e
Add nplusone and django-querycount to dev dependencies
jerbob Jun 8, 2021
28781d0
Update Dockerfile
jerbob Jun 8, 2021
80d3a65
Raise ValidationError for already verified email.
jchristgit Jun 8, 2021
74450c9
Remove double verification.
jchristgit Jun 8, 2021
f55adf1
Update WORKDIR in Dockerfile
jerbob Jun 8, 2021
333fa97
Do not cover branch.
jchristgit Jun 8, 2021
efef222
Update docker-compose.yml
jerbob Jun 8, 2021
806b097
Remove coverage configuration from pyproject.toml
jerbob Jun 8, 2021
a6e3ff2
Add test for uncached categories.
jchristgit Jun 8, 2021
d35b878
Revert "Add test for uncached categories."
jchristgit Jun 8, 2021
75c0dab
test token and fix config persisting between tests
Jun 8, 2021
e99af51
make targets
Jun 8, 2021
54780bc
remove polaris app, its not compatible with the current api and we sh…
Jun 8, 2021
2d6d977
100% coverage on auth views
Jun 8, 2021
1431f7b
100% coverage on leaderboard views and challenge mixin
Jun 8, 2021
09f7d8f
Add querycount middleware
jerbob Jun 9, 2021
8214f2d
no cover plugin loading
Jun 8, 2021
9402285
fix vote counts
Jun 9, 2021
7ca30da
more flag submit tests
Jun 9, 2021
21f5962
Flag submit tests
Jun 9, 2021
6f59d27
sql tests and bugfix
Jun 9, 2021
096ff89
remove iscompetitionopen - unused
Jun 9, 2021
8de3ef4
test permissions
Jun 9, 2021
222b571
test printable_name
Jun 9, 2021
3424bcf
fix accepted_renderer terminal spam
Jun 9, 2021
6957a81
config bugfixes and tests
Jun 9, 2021
2d6efc9
config post test coverage
Jun 9, 2021
8a429f6
remove unused password_reset_token
Jun 9, 2021
dfce5f3
remove databasebackend - unused since we converted the old RedisBacke…
Jun 9, 2021
780debe
clean up test_password_reset_issues_token
Jun 9, 2021
fe9d693
remove a couple instances of is_superuser, replacing with is_staff
Jun 9, 2021
32c3d76
test username changing
Jun 9, 2021
0a8ae63
test username changing
Jun 9, 2021
e5bb1e0
test userip
Jun 9, 2021
9f2bd92
no cover stats migration
Jun 9, 2021
b3588d2
Remove logic for no team users from the base points plugin
Jun 9, 2021
bb0e300
test incorrect solve tracking being disabled
Jun 9, 2021
3d5e62d
Make config post tests actually test post
Jun 9, 2021
a6069d3
fix config
Jun 9, 2021
9cbda2e
fix config
Jun 9, 2021
4d42397
test flag check view
Jun 9, 2021
a6cac86
remove deprecated and implict cli flag from isort
Jun 9, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Created by https://www.toptal.com/developers/gitignore/api/python,jetbrains,linux,windows,code
# Edit at https://www.toptal.com/developers/gitignore?templates=python,jetbrains,linux,windows,code
# Persistent test caching
.testmondata

### Code ###
.vscode/*
Expand Down
33 changes: 33 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.5.0
hooks:
- id: check-merge-conflict
- id: end-of-file-fixer
- id: mixed-line-ending
args: [--fix=lf]
- id: trailing-whitespace
args: [--markdown-linebreak-ext=md]

- repo: local
hooks:
- id: poetry-format
name: Poetry Format
description: This hook runs `make format` inside a relevant poetry environment.
entry: poetry run make format
language: system
types: [python]

- id: poetry-lint
name: Poetry Lint
description: This hook runs `make lint` inside a relevant poetry environment.
entry: poetry run make lint
language: system
types: [python]

- id: poetry-test
name: Poetry Test
description: This hook runs `make test` inside a relevant poetry environment.
entry: poetry run make test
language: system
types: [python]
3 changes: 1 addition & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ FROM python:3.9-slim
ARG BUILD_DEPS="build-essential curl"

RUN set -ex \
&& apt-get update && apt-get -y --no-install-recommends install $BUILD_DEPS libpq-dev netcat git \
&& apt-get update && apt-get -y --no-install-recommends install $BUILD_DEPS libpq-dev netcat make git \
&& rm -rf /var/lib/apt/lists/* \
&& curl -sSL "https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py" | python \
&& . $HOME/.poetry/env \
Expand All @@ -23,4 +23,3 @@ RUN poetry install --no-root --no-interaction \
COPY . /app/

EXPOSE 8000
WORKDIR /app/src/
35 changes: 35 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
test:
export DJANGO_SETTINGS_MODULE='backend.settings.lint' && \
cd src && \
BETTER_EXCEPTIONS=1 \
python manage.py migrate && \
pytest --testmon || \
if [ $$? = 5 ]; \
then exit 0; \
else exit $$?; \
fi

coverage:
export DJANGO_SETTINGS_MODULE='backend.settings.lint' && \
cd src && \
BETTER_EXCEPTIONS=1 \
python manage.py migrate && \
pytest --cov=. --cov-report=xml && \
coverage html && \
xdg-open htmlcov/index.html

format:
isort src && \
black src

lint:
flake8 && \
isort --check-only src

dev-server:
docker-compose build && \
docker-compose up -d

dev-test:
make dev-server && \
docker-compose exec backend pytest --cov=src src
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ services:
backend: &backend
build: .
entrypoint: /app/entrypoints/backend.sh
command: ./manage.py runserver 0.0.0.0:8000
command: python src/manage.py runserver 0.0.0.0:8000
volumes:
- .:/app
environment:
Expand Down
Loading