Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate Python 3.7, switch to Python 3.11 in CI #621

Merged
merged 4 commits into from
Jul 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/check-style.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: 3.8
python-version: 3.11
- uses: isort/isort-action@master
with:
isortVersion: "5.10.1"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/run-benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: 3.9
python-version: 3.11
- name: Cache dependencies
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: Key-v1-3.9-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }}
key: Key-v1-3.11-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [ '3.7', '3.8', '3.9', '3.10', '3.11' ]
python-version: [ '3.8', '3.9', '3.10', '3.11' ]
timeout-minutes: 15
steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -52,12 +52,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: '3.8'
python-version: '3.11'
- name: Cache dependencies
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: Key-v1-3.8-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }}
key: Key-v1-3.11-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
Expand All @@ -74,18 +74,18 @@ jobs:
codecov_in_develop_mode:

runs-on: ubuntu-latest
timeout-minutes: 15
timeout-minutes: 20
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: '3.8'
python-version: '3.11'
- name: Cache dependencies
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: Key-v1-3.8-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }}
key: Key-v1-3.11-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
Expand Down
2 changes: 1 addition & 1 deletion .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ python:
build:
os: ubuntu-22.04
tools:
python: "3.7"
python: "3.11"
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ feel free to submit a pull request that adds your project to this list.

## Installation

Before installing, make sure that your environment has Python 3.7+
Before installing, make sure that your environment has Python 3.8+
and [PyTorch](https://pytorch.org/get-started/locally/#start-locally) 1.9.0 or newer. They can be installed either
natively or with [Anaconda](https://www.anaconda.com/products/individual).

Expand Down
2 changes: 1 addition & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
language = "en"

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
Expand Down
2 changes: 1 addition & 1 deletion docs/user/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This tutorial will teach you how to install `hivemind`, host your own experts an

## Installation

Just `pip install hivemind` to get the latest release (requires Python 3.7 or newer).
Just `pip install hivemind` to get the latest release (requires Python 3.8 or newer).

You can also install the bleeding edge version from GitHub:

Expand Down
9 changes: 4 additions & 5 deletions hivemind/averaging/matchmaking.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
from __future__ import annotations

import asyncio
import concurrent.futures
import contextlib
import random
from math import isfinite
Expand Down Expand Up @@ -168,8 +167,8 @@
elif len(self.current_followers) > 0:
await self.leader_disband_group()
continue
except (concurrent.futures.CancelledError, asyncio.CancelledError):
break # note: this is a compatibility layer for python3.7
except asyncio.CancelledError:
break
except Exception as e:
if not self.assembled_group.done():
self.assembled_group.set_exception(e)
Expand Down Expand Up @@ -321,8 +320,8 @@
ordered_peer_ids=[item.to_bytes() for item in group_info.peer_ids],
gathered=group_info.gathered,
)
except (concurrent.futures.CancelledError, asyncio.CancelledError):
return # note: this is a compatibility layer for python3.7
except asyncio.CancelledError:
return

Check warning on line 324 in hivemind/averaging/matchmaking.py

View check run for this annotation

Codecov / codecov/patch

hivemind/averaging/matchmaking.py#L323-L324

Added lines #L323 - L324 were not covered by tests
except Exception as e:
logger.exception(e)
yield averaging_pb2.MessageFromLeader(code=averaging_pb2.INTERNAL_ERROR)
Expand Down Expand Up @@ -491,7 +490,7 @@
self.update_finished.clear()
continue
else:
raise asyncio.TimeoutError("pop_next_leader was invalidated: re-declared averager in background")

Check failure on line 493 in hivemind/averaging/matchmaking.py

View workflow job for this annotation

GitHub Actions / codespell

re-declared ==> redeclared

del self.leader_queue[maybe_next_leader]
self.past_attempts.add((maybe_next_leader, entry.expiration_time))
Expand Down
4 changes: 1 addition & 3 deletions hivemind/utils/asyncio.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,7 @@ async def await_cancelled(awaitable: Awaitable) -> bool:
try:
await awaitable
return False
except (asyncio.CancelledError, concurrent.futures.CancelledError):
# In Python 3.7, awaiting a cancelled asyncio.Future raises concurrent.futures.CancelledError
# instead of asyncio.CancelledError
except asyncio.CancelledError:
return True
except BaseException:
logger.exception(f"Exception in {awaitable}:")
Expand Down
10 changes: 1 addition & 9 deletions hivemind/utils/mpfuture.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import os
import threading
import uuid
from concurrent.futures import InvalidStateError
from contextlib import nullcontext
from enum import Enum, auto
from multiprocessing.reduction import ForkingPickler
Expand All @@ -26,15 +27,6 @@
ALL_STATES = base.PENDING, base.RUNNING, base.FINISHED, base.CANCELLED, base.CANCELLED_AND_NOTIFIED
TERMINAL_STATES = {base.FINISHED, base.CANCELLED, base.CANCELLED_AND_NOTIFIED}

try:
from concurrent.futures import InvalidStateError
except ImportError:
# Python 3.7 doesn't raise concurrent.futures.InvalidStateError for repeating set_result/set_exception calls and
# doesn't even define this error. In this module, we simulate the Python 3.8+ behavior,
# defining and raising this error if necessary.
class InvalidStateError(Exception):
"""Raised when attempting to change state of a future in a terminal state (e.g. finished)"""


class SharedBytes:
"""
Expand Down
2 changes: 1 addition & 1 deletion requirements-docs.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
recommonmark==0.5.0
sphinx_rtd_theme==0.4.3
docutils==0.16
sphinx==4.2.0
sphinx==5.0.0
3 changes: 2 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,9 +183,10 @@ def run(self):
"Intended Audience :: Science/Research",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Topic :: Scientific/Engineering",
"Topic :: Scientific/Engineering :: Mathematics",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
Expand Down
3 changes: 1 addition & 2 deletions tests/test_util_modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -528,8 +528,7 @@ async def coro():
async with enter_asynchronously(lock2):
await asyncio.sleep(1e-2)

num_coros = max(100, mp.cpu_count() * 5 + 1)
# note: if we deprecate py3.7, this can be reduced to max(33, cpu + 5); see https://bugs.python.org/issue35279
num_coros = max(33, mp.cpu_count() * 5 + 1)
await asyncio.wait({asyncio.create_task(coro()) for _ in range(num_coros)})


Expand Down
Loading