Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockerize #1365

Merged
merged 70 commits into from
Jul 20, 2020
Merged
Show file tree
Hide file tree
Changes from 53 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
7e28b23
WIP: All processes in a single container
Jul 8, 2020
afb4594
Use one container per process
Jul 8, 2020
08e99c6
Ignore venv and env directories
Jul 8, 2020
e7f9e3f
Skip flake8 checks on virtualenv dirs
Jul 8, 2020
bd1d793
Revert build_tools to Vagrant versions
Jul 9, 2020
58ca166
WIP on base image
Jul 9, 2020
c31ea12
Pass the master hostname to tests
Jul 9, 2020
a1e5a76
Fix flake8 errors
Jul 9, 2020
e0460b2
Use the existing --redis-url param to get master host
Jul 10, 2020
b5c2f2e
Don't shut down containers when building
Jul 10, 2020
61ece12
Add guide to contributing
Jul 13, 2020
6e31155
Merge branch 'dockerize' of github.com:abrookins/redis-py into dockerize
Jul 13, 2020
c5c9e81
Remove references to maling lists and IRC channels
Jul 13, 2020
af3771b
Attempt to use docker for travis testing
Jul 13, 2020
b4b1d97
WIP on travis config
Jul 13, 2020
867b28d
Install the version of compose that we need
Jul 13, 2020
e01cc2f
WIP
Jul 13, 2020
e4954ba
No need for Python in the test build anymore
Jul 13, 2020
27c4af4
Fix numbering
Jul 13, 2020
266e206
Format CLI commands correctly for RST
Jul 13, 2020
639a8dc
Update PHONY targets
Jul 13, 2020
a10751d
Improvements based on review feedback:
Jul 13, 2020
73b694d
Second try to add pypy and pypy3 to test runs
Jul 14, 2020
59bb043
Clean up cache files; delete root-owned .tox files...
Jul 14, 2020
7dfcd3b
Attempt to fix a timing bug
Jul 14, 2020
8fb3f97
Remove a rm command that would not work anyway
Jul 14, 2020
b2ef59b
Remove .tox from container side
Jul 14, 2020
df50cf6
Test all testenvs
Jul 14, 2020
aef6d4b
Move test Dockerfile into root, use COPY
Jul 14, 2020
55326ca
Include wait-for-it.sh copyright & license
Jul 15, 2020
273c8fe
remove unnecessary tox environments
andymccurdy Jul 15, 2020
a5508b1
force removal of docker images rather than ask the user if it's ok
andymccurdy Jul 15, 2020
0fe33ba
update PHONY targets
andymccurdy Jul 15, 2020
694f405
Clean up the wait-for-it.sh license
Jul 15, 2020
462fd34
master/slave configs for docker
andymccurdy Jul 15, 2020
26b611e
update configs to bind to explicit addresses. configure sentinel
andymccurdy Jul 15, 2020
fd32878
Merge branch 'dockerize' of github.com:abrookins/redis-py into dockerize
Jul 15, 2020
8c7a816
Check that we're subscribed to the right channels
Jul 15, 2020
54a8656
rename sentinel configs to sentinel.conf for clarity
andymccurdy Jul 15, 2020
1f5b2a6
no longer need to `make build`. `make test` will do everything
andymccurdy Jul 15, 2020
c8dc6a6
rename absurdly long test name
andymccurdy Jul 15, 2020
07abda7
Remove Vagrant files
Jul 15, 2020
6feed6e
Flesh out the docker env docs
Jul 15, 2020
adebdc3
Editing the contrib guide
Jul 15, 2020
61af925
Editing
Jul 15, 2020
344f589
Editing
Jul 15, 2020
f67ab19
Editing
Jul 15, 2020
5c32b23
Editing
Jul 15, 2020
11ef996
Editing docs
Jul 15, 2020
4cf1d83
restore codecov
andymccurdy Jul 15, 2020
5ab3a8d
Try running codecov in the test container
Jul 16, 2020
841d5ce
Try to pass the codecov token...
Jul 16, 2020
8c7d2da
testing docker-entry
andymccurdy Jul 16, 2020
59b460a
pass travis/codecov env vars to docker
andymccurdy Jul 16, 2020
07dd8e3
debug
andymccurdy Jul 16, 2020
d582494
debug
andymccurdy Jul 16, 2020
e5c7bfd
debug
andymccurdy Jul 16, 2020
cbf7b27
debug
andymccurdy Jul 16, 2020
2bff161
debug
andymccurdy Jul 16, 2020
fe89d31
debug
andymccurdy Jul 16, 2020
d878df7
debug
andymccurdy Jul 16, 2020
7da4b55
Try running codecov from tox
Jul 17, 2020
884c3d3
Attempt to combine coverage files
Jul 17, 2020
07c2675
Use the -a flag instead of "combine"
Jul 17, 2020
4c6cd16
Go back to "merge" -- -a failed
Jul 17, 2020
4565195
Remove unnecessary pytest-cov dep
Jul 17, 2020
a4ccbe4
add the covreport env to the list of default envs tox runs
andymccurdy Jul 20, 2020
f24b7f2
make the slowlog_get test more resilient to multiple clients being co…
andymccurdy Jul 20, 2020
458ded5
run the codecov env by default and disable when running outside Travis
andymccurdy Jul 20, 2020
1f988d8
changelog
andymccurdy Jul 20, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
**/__pycache__
**/*.pyc
.tox
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@ vagrant/.vagrant
.eggs
.idea
.coverage
env
venv
52 changes: 12 additions & 40 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,42 +1,14 @@
language: python
cache: pip
matrix:
include:
- env: TOXENV=flake8
- python: 2.7
env: TOXENV=py27-plain
- python: 2.7
env: TOXENV=py27-hiredis
- python: 3.5
env: TOXENV=py35-plain
- python: 3.5
env: TOXENV=py35-hiredis
- python: 3.6
env: TOXENV=py36-plain
- python: 3.6
env: TOXENV=py36-hiredis
- python: 3.7
env: TOXENV=py37-plain
- python: 3.7
env: TOXENV=py37-hiredis
- python: 3.8
env: TOXENV=py38-plain
- python: 3.8
env: TOXENV=py38-hiredis
- python: pypy
env: TOXENV=pypy-plain
- python: pypy
env: TOXENV=pypy-hiredis
- python: pypy3
env: TOXENV=pypy3-plain
- python: pypy3
env: TOXENV=pypy3-hiredis
env:
- DOCKER_COMPOSE_VERSION=1.26.2

before_install:
- wget https://github.com/antirez/redis/archive/6.0.5.tar.gz && mkdir redis_install && tar -xvzf 6.0.5.tar.gz -C redis_install && cd redis_install/redis-6.0.5 && make && src/redis-server --daemonize yes && cd ../..
- redis-cli info
install:
- pip install codecov tox
- sudo rm /usr/local/bin/docker-compose
- curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
- chmod +x docker-compose
- sudo mv docker-compose /usr/local/bin

services:
- docker

script:
- tox
after_success:
- "if [[ $TOXENV != 'flake8' ]]; then codecov; fi"
- make test
128 changes: 128 additions & 0 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
Contributing
============

Introduction
------------

First off, thank you for considering contributing to redis-py. We value community contributions!

Contributions We Need
----------------------

You may already know what you want to contribute -- a fix for a bug you encountered, or a new feature your team wants to use.

If you don't know what to contribute, keep an open mind! Improving documentation, bug triaging, and writing tutorials are all examples of helpful contributions that mean less work for you.

Your First Contribution
-----------------------
Unsure where to begin contributing? You can start by looking through `help-wanted issues <https://github.com/andymccurdy/redis-py/issues?q=is%3Aopen+is%3Aissue+label%3ahelp-wanted>`_.

Never contributed to open source before? Here are a couple of friendly tutorials:

- http://makeapullrequest.com/
- http://www.firsttimersonly.com/

Getting Started
---------------

Here's how to get started with your code contribution:

1. Create your own fork of redis-py
2. Do the changes in your fork
3. If you need a development environment, run ``make dev``
4. While developing, make sure the tests pass by running ``make test``
5. If you like the change and think the project could use it, send a pull request

The Development Environment
---------------------------

Running ``make dev`` will create a Docker-based development environment that starts the following containers:

* A master Redis node
* A slave Redis node
* Three sentinel Redis nodes
* A test container

The slave is a replica of the master node, using the `leader-follower replication <https://redis.io/topics/replication>`_ feature.

The sentinels monitor the master node in a `sentinel high-availability configuration <https://redis.io/topics/sentinel>`_.

Meanwhile, the `test` container hosts the code from your checkout of ``redis-py`` and allows running tests against many Python versions.

Docker Tips
^^^^^^^^^^^

Following are a few tips that can help you work with the Docker-based development environment.

To get a bash shell inside of a container:

``$ docker-compose run <service> /bin/bash``

**Note**: The term "service" refers to the "services" defined in the ``docker-compose.yml`` file: "master", "slave", "sentinel_1", "sentinel_2", "sentinel_3", "test".

Containers run a minimal Debian image that probably lacks tools you want to use. To install packages, first get a bash session (see previous tip) and then run:

``$ apt update && apt install <package>``

You can see the combined logging output of all containers like this:

``$ docker-compose logs``

The command `make test` runs all tests in all tested Python environments. To run the tests in a single environment, like Python 3.6, use a command like this:

``$ docker-compose run test tox -e py36 -- --redis-url=redis://master:6379/9``

Here, the flag ``-e py36`` runs tests against the Python 3.6 tox environment. And note from the example that whenever you run tests like this, instead of using `make test`, you need to pass ``-- --redis-url=redis://master:6379/9``. This points the tests at the "master" container.

Our test suite uses ``pytest``. You can run a specific test suite against a specific Python version like this:

``$ docker-compose run test tox -e py36 -- --redis-url=redis://master:6379/9 tests/test_commands.py``

Troubleshooting
^^^^^^^^^^^^^^^
If you get any errors when running ``make dev`` or ``make test``, make sure that you
are using supported versions of Docker and docker-compose.

The included Dockerfiles and docker-compose.yml file work with the following
versions of Docker and docker-compose:

* Docker 19.03.12
* docker-compose 1.26.2

How to Report a Bug
-------------------

Security Vulnerabilities
^^^^^^^^^^^^^^^^^^^^^^^^

**NOTE**: If you find a security vulnerability, do NOT open an issue. Email Andy McCurdy (sedrik@gmail.com) instead.

In order to determine whether you are dealing with a security issue, ask yourself these two questions:

* Can I access something that's not mine, or something I shouldn't have access to?
* Can I disable something for other people?

If the answer to either of those two questions are "yes", then you're probably dealing with a security issue. Note that even if you answer "no" to both questions, you may still be dealing with a security issue, so if you're unsure, just email Andy at sedrik@gmail.com.

Everything Else
^^^^^^^^^^^^^^^

When filing an issue, make sure to answer these five questions:

1. What version of redis-py are you using?
2. What version of redis are you using?
3. What did you do?
4. What did you expect to see?
5. What did you see instead?

How to Suggest a Feature or Enhancement
---------------------------------------

If you'd like to contribute a new feature, make sure you check our issue list to see if someone has already proposed it. Work may already be under way on the feature you want -- or we may have rejected a feature like it already.

If you don't see anything, open a new issue that describes the feature you would like and how it should work.

Code Review Process
-------------------

The core team looks at Pull Requests on a regular basis. We will give feedback as as soon as possible. After feedback, we expect a response within two weeks. After that time, we may close your PR if it isn't showing any activity.
9 changes: 9 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM fkrull/multi-python:latest

RUN apt update && apt install -y pypy pypy-dev pypy3-dev curl
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python3.8 get-pip.py
RUN pip install codecov

WORKDIR /redis-py

COPY . /redis-py
14 changes: 14 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.PHONY: base clean dev test

base:
docker build -t redis-py-base docker/base

dev: base
docker-compose up -d --build

test: dev
docker-compose run --rm -e TRAVIS test /redis-py/docker-entry.sh

clean:
docker-compose stop
docker-compose rm -f
6 changes: 6 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,12 @@ or from source:

$ python setup.py install

Contributing
------------

Want to contribute a feature, bug report, or report an issue? Check out our `guide to
contributing <https://github.com/andymccurdy/redis-py/blob/master/CONTRIBUTING.rst>`_.


Getting Started
---------------
Expand Down
1 change: 0 additions & 1 deletion build_tools/.bash_profile

This file was deleted.

4 changes: 0 additions & 4 deletions build_tools/bootstrap.sh

This file was deleted.

29 changes: 0 additions & 29 deletions build_tools/build_redis.sh

This file was deleted.

37 changes: 0 additions & 37 deletions build_tools/install_redis.sh

This file was deleted.

37 changes: 0 additions & 37 deletions build_tools/install_sentinel.sh

This file was deleted.

8 changes: 0 additions & 8 deletions build_tools/redis-configs/001-master

This file was deleted.

10 changes: 0 additions & 10 deletions build_tools/redis-configs/002-slave

This file was deleted.

Loading