Enhancement: FWI direct download from EFFIS and GeoJSON conversion (#77)

* add: function to limit the number of files we have as an output for the function list_files

* add: function to query data from url and store it to s3

* add: function to write json to s3 bucket

* add: helper class to manipulate FWI file we download from EFFIS

* add: notebook to show how the downloading, formatting and uploading should be done

* chore: update README with credit for EFFIS

* add: main

* add localstack for dev purposes + comment deprecated code

* Update main.py

use environment variables for connection

* refactor: S3Bucket list_folders & list_files
improve readability and handle empty buckets

* Update main.py

- restructure task
- add retrieved_date option

* clean setup

* Update CONTRIBUTING.md

lint & remove `pip install -r requirements.txt` step.

* Update fwi_helpers.py

* Update CONTRIBUTING.md

add localstack guidelines

* Update setup.py

* refactor: type hinting in FWIHelpers class

* feat: add 1 cell to connect more easily to s3

---------

Co-authored-by: Camille Modeste <camillemodeste@camilles-macbook-air.home>
Co-authored-by: Alexis Cruveiller <33450491+Acruve15@users.noreply.github.com>
Co-authored-by: Jules Dupont <jules.dupont@supagro.fr>

CONTRIBUTING.md

@@ -8,44 +8,44 @@ Whatever the way you wish to contribute to the project, please respect the [code
 
 ### Codebase structure
 
--   [pyro-risks](https://github.com/pyronear/pyro-risks/tree/master/pyro_risks) - the pyro-risks library
--   [examples](https://github.com/pyronear/pyro-risks/tree/master/scripts) - examples scripts
--   [test](https://github.com/pyronear/pyro-risks/blob/master/test) - python unit tests
+- [pyro-risks](https://github.com/pyronear/pyro-risks/tree/master/pyro_risks) - the pyro-risks library
+- [examples](https://github.com/pyronear/pyro-risks/tree/master/scripts) - examples scripts
+- [test](https://github.com/pyronear/pyro-risks/blob/master/test) - python unit tests
 
 ### Continuous Integration
 
 This project uses the following integrations to ensure proper codebase maintenance:
 
--   [Github Actions](https://docs.github.com/en/free-pro-team@latest/actions/guides/about-continuous-integration) - run workflows for building and testing the package
--   [Codacy](https://www.codacy.com/) - analyzes commits for code quality
--   [Codecov](https://codecov.io/) - reports back coverage results
+- [Github Actions](https://docs.github.com/en/free-pro-team@latest/actions/guides/about-continuous-integration) - run workflows for building and testing the package
+- [Codacy](https://www.codacy.com/) - analyzes commits for code quality
+- [Codecov](https://codecov.io/) - reports back coverage results
 
 As a contributor, you will only have to ensure coverage of your code by adding appropriate unit testing of your code.
 
 ### Style conventions
 
--   **Code**:
-    -   Setup the `__all__` special variable for each module
-    -   Use type hints for every functions ([type hints cheat sheet](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html))
-    -   Format your code using the [black](https://github.com/psf/black) auto-formatter
-    -   Ensure to document your code using type hints compatible docstrings. In doing so, please follow [Google-style](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) so it can ease the process of documentation later.
+- **Code**:
+  - Setup the `__all__` special variable for each module
+  - Use type hints for every functions ([type hints cheat sheet](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html))
+  - Format your code using the [black](https://github.com/psf/black) auto-formatter
+  - Ensure to document your code using type hints compatible docstrings. In doing so, please follow [Google-style](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) so it can ease the process of documentation later.
 
--   **Commit message**: please follow [Udacity guide](http://udacity.github.io/git-styleguide/)
+- **Commit message**: please follow [Udacity guide](http://udacity.github.io/git-styleguide/)
 
-## Contributing to the project 
+## Contributing to the project
 
 In order to contribute to project, you will first need to **set up your *pyro-risks* development environment** and then follow the **contributing workflow** and the **code & commit guidelines**.
 
--   [Project Setup](#project-setup): *fork the project and install dependencies in a well-encapsulated development environment*
+- [Project Setup](#project-setup): *fork the project and install dependencies in a well-encapsulated development environment*
 
-    1.  [**Create a virtual environment**](#create-a-virtual-environment) to avoid collision with our OS and other projects
-    2.  [**Fork the pyro-risks project**](#fork-the-repository) to be able to start working on a local copy of the project
-    3.  [**Set origin and upstream remotes**](#set-origin-and-upstream-remotes-repositories) repositories: 
-    4.  [**Install project dependencies**](#install-project-dependencies)
+    1. [**Create a virtual environment**](#create-a-virtual-environment) to avoid collision with our OS and other projects
+    2. [**Fork the pyro-risks project**](#fork-the-repository) to be able to start working on a local copy of the project
+    3. [**Set origin and upstream remotes**](#set-origin-and-upstream-remotes-repositories) repositories:
+    4. [**Install project dependencies**](#install-project-dependencies)
 
--   [Contributing workflow](#contributing-workflow): *pull remote changes/new contributions and push your contributions to the original project*
+- [Contributing workflow](#contributing-workflow): *pull remote changes/new contributions and push your contributions to the original project*
 
--   [Follow the repository style conventions](#style-conventions)
+- [Follow the repository style conventions](#style-conventions)
 
 ### Project Setup
 
@@ -55,7 +55,7 @@ In order to contribute to project, you will first need to **set up your *pyro-ri
 
 <br>
 
--   We are going to create a python3.6 virtual environment dedicated to the `pyro-risks` project using [conda](https://docs.conda.io/en/latest/) as an environment management system. Please open a terminal and follow the instructions.
+- We are going to create a python3.6 virtual environment dedicated to the `pyro-risks` project using [conda](https://docs.conda.io/en/latest/) as an environment management system. Please open a terminal and follow the instructions.
 
     ```shell
     conda create --name pyro-risks python=3.6 anaconda 
@@ -66,11 +66,11 @@ In order to contribute to project, you will first need to **set up your *pyro-ri
 
 <br>
 
--   We are going to get a local copy of the remote project (*fork*) and set remotes so we stay up to date to recent contributions.
+- We are going to get a local copy of the remote project (*fork*) and set remotes so we stay up to date to recent contributions.
 
-    1.  **Create a fork** by clicking on the **fork button** on the current repository page
+    1. **Create a fork** by clicking on the **fork button** on the current repository page
 
-    2.  Clone *your* fork locally.
+    2. Clone *your* fork locally.
 
         ```shell
         # change directory to one for the project
@@ -87,9 +87,9 @@ In order to contribute to project, you will first need to **set up your *pyro-ri
 
 <br>
 
-1.  Configure your fork `YOUR_USERNAME/pyro-risks` as `origin` remote
+1. Configure your fork `YOUR_USERNAME/pyro-risks` as `origin` remote
 
-2.  Configure `pyronear/pyro-risks repository` as `upstream` remote
+2. Configure `pyronear/pyro-risks repository` as `upstream` remote
 
     ```shell
     # add the original repository as remote repository called "upstream"
@@ -115,20 +115,36 @@ In order to contribute to project, you will first need to **set up your *pyro-ri
 ```shell
 # install dependencies
 pip install black
-pip install -r requirements.txt
 
 # install current project in editable mode,
 # so local changes will be reflected locally (ie:at import)
 pip install -e .
 ```
 
+#### Local deployment
+
+To enable a smoother development experience, you can run AWS locally using docker containers with this command:
+
+```shell
+make setup-dev
+```
+
+If you cannot use Makefile, you can run the following commands:
+
+```shell
+# Create a localstack container
+docker compose -f docker-compose.localstack.yml up -d --build
+# Create a bucket
+docker compose exec localstack awslocal s3 mb s3://pyro-risk
+```
+
 ### Contributing workflow
 
 * * *
 
 Once the project is well set up, we are going to detail step by step a usual contributing workflow.
 
-1.  Merge recent contributions onto master (do this frequently to stay up-to-date)
+1. Merge recent contributions onto master (do this frequently to stay up-to-date)
 
     ```shell
     # fetch all changes from the upstream repository
@@ -151,16 +167,16 @@ Once the project is well set up, we are going to detail step by step a usual con
     git pull upstream/master
     ```
 
-2.  Create a local feature branch to work on
+2. Create a local feature branch to work on
 
     ```shell
     # Create a new branch with the name of your feature
     git checkout -b feature-branch
     ```
 
-3.  Commit your changes (remember to add unit tests for your code). Feel free to interactively rebase your history to improve readability. Follow the style guide See [Style Conventions](#style-conventions) to follow guidelines.
+3. Commit your changes (remember to add unit tests for your code). Feel free to interactively rebase your history to improve readability. Follow the style guide See [Style Conventions](#style-conventions) to follow guidelines.
 
-4.  Rebase your feature branch so that merging it will be a simple fast-forward that won't require any conflict resolution work.
+4. Rebase your feature branch so that merging it will be a simple fast-forward that won't require any conflict resolution work.
 
     ```shell
     # Switch to feature branch
@@ -170,7 +186,7 @@ Once the project is well set up, we are going to detail step by step a usual con
     git rebase master
     ```
 
-5.  Push your changes on remote feature branch.
+5. Push your changes on remote feature branch.
 
     ```shell
     git checkout feature-branch
@@ -184,7 +200,7 @@ Once the project is well set up, we are going to detail step by step a usual con
     black /path/to/local/pyronear/project/pyro-risks/
     ```
 
-6.  When satisfied with your branch, open a [PR](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork) from your fork in order to integrate your contribution to original project.
+6. When satisfied with your branch, open a [PR](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork) from your fork in order to integrate your contribution to original project.
 
 ## Opening an issue
 

Makefile

@@ -0,0 +1,3 @@
+setup-dev:
+	docker compose -f docker-compose.localstack.yml up -d --build
+	docker compose exec localstack awslocal s3 mb s3://pyro-risk

README.md

@@ -108,6 +108,11 @@ Please refer to the [`CONTRIBUTING`](./CONTRIBUTING.md) guide if you wish to con
 
 This project is developed and maintained by the repo owner and volunteers from [Data for Good](https://dataforgood.fr/).
 
+This project uses data from EFFIS (European Forest Fire Information System) for the FWI (Fire Weather Index). This data is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. 
+
+- Dataset: [EFFIS FWI Dataset](https://effis.jrc.ec.europa.eu/applications/data-and-services)
+- License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
+
 ## License
 
 Distributed under the Apache v2 License. See [`LICENSE`](./LICENSE) for more information.

docker-compose.localstack.yml

@@ -0,0 +1,13 @@
+version: '3.7'
+
+services:
+  # ref link: https://github.com/localstack/localstack/blob/master/docker-compose.yml
+  localstack:
+    image: localstack/localstack:1.4.0
+    ports:
+      - 4566:4566
+    # environment variables details: https://docs.localstack.cloud/references/configuration/
+    environment:
+      - EDGE_PORT=4566
+      - SERVICES=s3
+      - ALLOW_NONSTANDARD_REGIONS=1

pyro_risks/__init__.py

@@ -1,5 +0,0 @@
-from .version import __version__
-from pyro_risks import config
-from pyro_risks import datasets
-from pyro_risks import models
-from pyro_risks import pipeline