Skip to content

Commit

Permalink
refactor: Cleaned pyroengine and refactored project (#101)
Browse files Browse the repository at this point in the history
* refactor: Refactored pyroengine package

* test: Updated unittests

* chore: Updated setup

* refactor: Moved engine wrapper to src/

* docs: Updated documentation theme

* docs: Added makefile

* ci: Updated CI jobs

* refactor: Updated ping script

* refactor: Updated Dockerfile

* ci: Updated collect_env

* feat: Added possibility to load local files

* docs: Added docker build to makefile

* docs: Updated documentation

* refactor: Updated run.py

* style: Fixed style

* test: Added unittest for local file loading

* test: Fixed unittests

* ci: Removed CI job

* docs: Fixed headers

* ci: Fixed RPI CI

* chore: Updated version specifier of pyroclient

* docs: Updated CONTRIBUTING

* docs: Updated README

* ci: Fixed typos

* chore: Updated gitignore

* chore: Updated build commands

* chore: Updated pyproject

* style: Fixed style

* ci: Extended header check

* refactor: Refactored docker orchestration

* docs: Updated makefile

* docs: Updated README

* refactor: Refactored states management

* test: Updated unittests

* docs: Updated header

* feat: Cleaned run script

* ci: Updated CI

* refactor: Removed CMD from docker

* docs: Updated makefile

* style: Fixed imports

* chore: Updated docker orchestration
  • Loading branch information
frgfm authored Aug 5, 2022
1 parent 121c675 commit 731391c
Show file tree
Hide file tree
Showing 64 changed files with 1,666 additions and 1,743 deletions.
2 changes: 0 additions & 2 deletions .coveragerc

This file was deleted.

5 changes: 3 additions & 2 deletions .flake8
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
[flake8]
max-line-length = 120
ignore = F401, E402, E265, F403, W503, W504, F821
exclude = venv*, .circleci, .git, docs
ignore = E203, E402, E265, F403, W503, W504, E731
exclude = .git, venv*, docs, build
per-file-ignores = **/__init__.py:F401
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,12 @@ Steps to reproduce the behavior:
## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pyronear/pyro-engine/master/scripts/collect_env.py)
[environment collection script](https://raw.githubusercontent.com/pyronear/pyro-engine/main/.github/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pyronear/pyro-engine/master/scripts/collect_env.py
wget https://raw.githubusercontent.com/pyronear/pyro-engine/main/.github/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```
Expand Down
218 changes: 218 additions & 0 deletions .github/collect_env.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
# Copyright (C) 2020-2022, Pyronear.

# This program is licensed under the Apache License 2.0.
# See LICENSE or go to <https://www.apache.org/licenses/LICENSE-2.0> for full license details.

"""
Based on https://github.com/pytorch/pytorch/blob/master/torch/utils/collect_env.py
This script outputs relevant system environment info
Run it with `python collect_env.py`.
"""

from __future__ import absolute_import, division, print_function, unicode_literals

import locale
import re
import subprocess
import sys
from collections import namedtuple

try:
import pyroengine

ENGINE_AVAILABLE = True
except (ImportError, NameError, AttributeError):
ENGINE_AVAILABLE = False

try:
import onnxruntime

ONNX_AVAILABLE = True
except (ImportError, NameError, AttributeError):
ONNX_AVAILABLE = False

PY3 = sys.version_info >= (3, 0)


# System Environment Information
SystemEnv = namedtuple(
"SystemEnv",
[
"pyroengine_version",
"onnxruntime_version",
"os",
"python_version",
],
)


def run(command):
"""Returns (return-code, stdout, stderr)"""
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
output, err = p.communicate()
rc = p.returncode
if PY3:
enc = locale.getpreferredencoding()
output = output.decode(enc)
err = err.decode(enc)
return rc, output.strip(), err.strip()


def run_and_read_all(run_lambda, command):
"""Runs command using run_lambda; reads and returns entire output if rc is 0"""
rc, out, _ = run_lambda(command)
if rc != 0:
return None
return out


def run_and_parse_first_match(run_lambda, command, regex):
"""Runs command using run_lambda, returns the first regex match if it exists"""
rc, out, _ = run_lambda(command)
if rc != 0:
return None
match = re.search(regex, out)
if match is None:
return None
return match.group(1)


def get_platform():
if sys.platform.startswith("linux"):
return "linux"
elif sys.platform.startswith("win32"):
return "win32"
elif sys.platform.startswith("cygwin"):
return "cygwin"
elif sys.platform.startswith("darwin"):
return "darwin"
else:
return sys.platform


def get_mac_version(run_lambda):
return run_and_parse_first_match(run_lambda, "sw_vers -productVersion", r"(.*)")


def get_windows_version(run_lambda):
return run_and_read_all(run_lambda, "wmic os get Caption | findstr /v Caption")


def get_lsb_version(run_lambda):
return run_and_parse_first_match(run_lambda, "lsb_release -a", r"Description:\t(.*)")


def check_release_file(run_lambda):
return run_and_parse_first_match(run_lambda, "cat /etc/*-release", r'PRETTY_NAME="(.*)"')


def get_os(run_lambda):
platform = get_platform()

if platform == "win32" or platform == "cygwin":
return get_windows_version(run_lambda)

if platform == "darwin":
version = get_mac_version(run_lambda)
if version is None:
return None
return "Mac OSX {}".format(version)

if platform == "linux":
# Ubuntu/Debian based
desc = get_lsb_version(run_lambda)
if desc is not None:
return desc

# Try reading /etc/*-release
desc = check_release_file(run_lambda)
if desc is not None:
return desc

return platform

# Unknown platform
return platform


def get_env_info():
run_lambda = run

if ENGINE_AVAILABLE:
pyroengine_str = pyroengine.__version__
else:
pyroengine_str = "N/A"

if ONNX_AVAILABLE:
onnxruntime_str = onnxruntime.__version__
else:
onnxruntime_str = "N/A"

return SystemEnv(
pyroengine_version=pyroengine_str,
onnxruntime_version=onnxruntime_str,
python_version=".".join(map(str, sys.version_info[:3])),
os=get_os(run_lambda),
)


env_info_fmt = """
PyroEngine version: {pyroengine_version}
ONNX runtime version: {onnxruntime_version}
OS: {os}
Python version: {python_version}
""".strip()


def pretty_str(envinfo):
def replace_nones(dct, replacement="Could not collect"):
for key in dct.keys():
if dct[key] is not None:
continue
dct[key] = replacement
return dct

def replace_bools(dct, true="Yes", false="No"):
for key in dct.keys():
if dct[key] is True:
dct[key] = true
elif dct[key] is False:
dct[key] = false
return dct

def maybe_start_on_next_line(string):
# If `string` is multiline, prepend a \n to it.
if string is not None and len(string.split("\n")) > 1:
return "\n{}\n".format(string)
return string

mutable_dict = envinfo._asdict()

# Replace True with Yes, False with No
mutable_dict = replace_bools(mutable_dict)

# Replace all None objects with 'Could not collect'
mutable_dict = replace_nones(mutable_dict)

return env_info_fmt.format(**mutable_dict)


def get_pretty_env_info():
"""Collects environment information for debugging purposes
Returns:
str: environment information
"""
return pretty_str(get_env_info())


def main():
print("Collecting environment information...")
output = get_pretty_env_info()
print(output)


if __name__ == "__main__":
main()
67 changes: 0 additions & 67 deletions .github/validate_headers.py

This file was deleted.

34 changes: 34 additions & 0 deletions .github/workflows/builds.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: builds

on:
push:
branches: main
pull_request:
branches: main

jobs:
build:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python: [3.8]
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python }}
architecture: x64
- name: Cache python modules
uses: actions/cache@v2
with:
path: ~/.cache/pip
key: ${{ runner.os }}-python-${{ matrix.python }}-${{ hashFiles('pyproject.toml') }}-build
- name: Install package
run: |
python -m pip install --upgrade pip
pip install -e . --upgrade
- name: Import package
run: python -c "import pyroengine; print(pyroengine.__version__)"
17 changes: 17 additions & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: docker

on:
push:
branches: main
pull_request:
branches: main

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Build docker image
run: docker build . -t pyronear/pyro-engine:python3.8.1-slim
- name: Run docker container
run: docker run pyronear/pyro-engine:python3.8.1-slim python -c 'import pyroengine'
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: doc-deploy
on:
push:
branches: master
branches: main

jobs:
docs-publish:
Expand Down
Loading

0 comments on commit 731391c

Please sign in to comment.