GDSC-Delft-Dev · paulmis · Apr 4, 2023 · Mar 18, 2023 · Mar 18, 2023 · Mar 18, 2023
diff --git a/.gitignore b/.gitignore
@@ -10,5 +10,6 @@ src/backend/pipeline/data
 .vscode
 *json
 nutrient_masks.npy
+PlantVillage
 .vscode
 .idea
diff --git a/README.md b/README.md
@@ -3,97 +3,168 @@
 
 ![example workflow](https://github.com/GDSC-Delft-Dev/apa/actions/workflows/pipeline.yml/badge.svg)
 
-## About
+Terrafarm is an autonomous farming solution provides a comprehensive way to monitor crops at any scale. We provide farmers with the ability to scrutinize every square inch of their fields for a wide range of issues. By detecting crop diseases before they spread, Terrafarm can reduce the usage of harmful chemicals by up to 90% and eradicate invasive species regionally. As the application provides health reports, farmers can optimize fertilizer use and reduce preventive pesticide, herbicide, and fungicide use. 
 
-### Problem we solving
-Growing (high-quality) crops sustainably for an ever-increasing population is one of the biggest challenges we face today, as farmers all over the world are faced with complex decision making problems for a vast amount of crops. To this end, a variety of parameters need to be traced - think of application of fertilizer, soil humidity or availability of nutrients.
+Want to know more? Read our wiki [**here**](../../wiki).
 
-In traditional agriculture, fields are treated as homogeneous entities, which generally leads to sub-optimal treatment due to lack of (localized) traceability. This is problematic, as oversupply of agricultural inputs leads to environmental pollution. Moreover, unnecessary large quantities can go to waste if produce are not harvested at their optimal time. Finally, this clearly leads to low yield density and hence missed profits for farmers.
+## Getting Started
 
-[Precision agriculture](https://en.wikipedia.org/wiki/Precision_agriculture) on the other hand, aims to produce more crops with fewer resources while maintaining quality. This sustainable agricultural model utilizes IT solutions to allow for localized treatment to a much finer degree. This paradigm shift is becoming increasingly urgent because of the worldwide increase in food demands for example: the number of people who will require food in 2050 is estimated at nine billion.
+Our code can be found in the `src` directory. Read below to learn how to explore, run, and modify the backend and frontend, or play with the notebooks in the `notebooks` directory.
 
-### Our solution
-Our **mobile app Terrafarm** allows farmers to perform **smart monitoring, analysis and planning** in an intuitive and affordable manner. In fact, our system uses **image processing and deep learning** to extact **actionable insights** from multispectral drone images. These insights - think of pest infestations, moisture content or nutrient deficiencies - are visualized to users, thereby providing full transparancy. We aim to target both small- and medium-scale farmers. Detailed information about our image processing pipeline and Flutter mobile app can be found under `apa/src/backend` and `apa/src/frontend` respectively.
+### Backend
 
-<div>
-    <img src="assets/Terrafarm-poster-0.jpg" alt="Image 1" width="500" style="display:inline-block;">
-    <img src="assets/Terrafarm-poster-1.jpg" alt="Image 2" width="500" style="display:inline-block;">
-</div>
+The backend comprises of the image processing pipleline that processes mutlispectral images from farms. You can run it locally, or remotely on GCP (in a container). If you'd like to know more about the pipeline, read our wiki [**here**](../../wiki/Pipeline).
 
-<p style="text-align:center;">Figure: Information poster presenting Terrafarm</p>
+#### Local setup
+Run the image processing pipeline locally. Tested on linux (`Ubuntu 20`) and Mac (`Ventura 13`). Components that do not involve ML training can also be run on `Windows 10`.
 
+1. Install [**Python 3.10**](https://www.python.org/downloads/)
 
+2. Clone the repo
 
-# Build Tools
+```
+git clone https://github.com/GDSC-Delft-Dev/apa.git
+```
 
-![image](https://img.shields.io/badge/Flutter-02569B?style=for-the-badge&logo=flutter&logoColor=white)
-</br>
-![image](https://img.shields.io/badge/Dart-0175C2?style=for-the-badge&logo=dart&logoColor=white)
-</br>
-![image](https://img.shields.io/badge/Python-FFD43B?style=for-the-badge&logo=python&logoColor=blue)
-</br>
-![image](https://img.shields.io/badge/firebase-ffca28?style=for-the-badge&logo=firebase&logoColor=black)
-</br>
-![image](https://img.shields.io/badge/Google_Cloud-4285F4?style=for-the-badge&logo=google-cloud&logoColor=white)
-</br>
-![image](https://img.shields.io/badge/TensorFlow-FF6F00?style=for-the-badge&logo=tensorflow&logoColor=white)
-</br>
-![image](https://img.shields.io/badge/OpenCV-27338e?style=for-the-badge&logo=OpenCV&logoColor=white)
-</br>
-![image](https://img.shields.io/badge/GitHub_Actions-2088FF?style=for-the-badge&logo=github-actions&logoColor=white)
+Note that this might take a while.
 
-# Getting Started
+3. Setup the Python virtual environment 
+```
+pip install virtualenv
+virtualenv env
+source env/bin/activate (linux, mac)
+source env/Scripts/activate (windows)
+```
+
+4. Install Python requirements
+```
+cd src/backend
+pip install -r requirements.txt
+```
+
+5. Run the pipeline
+```
+py main.py
+```
+
+The supported arguments for `main.py` are:
+- `mode` (`local`/`cloud`) - specify if the input images are already in the cloud or need to be uploaded first from the local filesystem
+- `path` - path to the input images, relative to the local/cloud root
+- `name` - a unique name for the created job
+
+Run the pipeline with images already in the cloud:
+```
+py main.py --path path/to/images --mode cloud
+```
+
+Run the pipeline with images on your local filesystem:
+```
+py main.py --path path/to/images --mode local
+```
 
-Follow these steps to set up your project locally.
+#### Cloud setup
+To use infrastructure, please request the GCP service account key at `pawel.mist@gmail.com`.
 
-Clone the repo
+1. Clone the repo
 ```
 git clone https://github.com/GDSC-Delft-Dev/apa.git
 ```
 
-## Setup backend
+Note that this might take a while.
 
-Setup virtual python environment
+2. Set the GCP service account environmental variable
 ```
-pip install virtualenv
-virtualenv env
+export GCP_FA_PRIVATE_KEY=<key> (linux, mac)
+set GCP_FA_PRIVATE_KEY=<key> (windows)
 ```
 
-Activate on MacOS or Linux
+3. Trigger the pipeline
+
+Manual triggers allow you to run the latest pipeline builds from the Artifact Registry with custom input data using Cloud Run.  You can run a job with either input data from your local file system or input data that already resides in the cloud.
+
+```bash
+cd src/backend
+sudo chmod +x trigger.sh
+./trigger.sh
+```
+
+The supported arguments for `trigger.sh` are:
+- `l` - path to the local images
+- `c` - path to the images on the cloud (Cloud Storage)
+- `n` - a unique name for the pipeline job
+
+Note that local inputs are first copied to a staging directory in Cloud Storage, and will only be removed if the job succeeeds.
+
+Provide input data from a local filesystem
+
+```bash
+./trigger.sh -l /path/to/data/ -n name-of-the-job
 ```
-source env/bin/activate
+
+Provide input data from Cloud Storage
+
+```bash
+./trigger.sh -c /path/to/data/ -n name-of-the-job
 ```
 
-Activate on Windows
+### Testing
+To executed the automated tests, run `pytest` unit tests:
+
 ```
-source env/Scripts/activate
+python -m pytest
 ```
-Install Python requirements
+
+You can find our tests in `src\backend\pipeline\test\unit`.
+
+### Static analysis
+Our project uses `mypy` and `pylint` to assert the quality of the code. You can run these with:
+
 ```
-pip install -r requirements.txt
+python -m mypy . --explicit-package-bases
+python -m pylint ./pipeline
 ```
-Please refer to `apa/src/backend/README.md` for detailed information on the image processing pipeline.
-<!-- TODO: Perhaps more info? -->
 
-## Setup frontend
-Please refer to `apa/src/frontend/README.md`.
+### CI/CD
+The CI/CD pushes the build from the latest commit to the `pipelines-dev` repository in the Google Artifact Registry. Note that only the backend is covered.
+
+You can find the pipeline declaration in `.github\workflows\pipeline.yml`.
 
+## Frontend setup
+Please refer to `apa/src/frontend/README.md`.
 
-# Contributing
+## Contributing
 Anyone who is eager to contribute to this project is very welcome to do so. Simply take the following steps:
 1. Fork the project
 2. Create your own feature branch
 3. Commit your changes
 4. Push to the `dev` branch and open a PR
 
-# Datasets
+## Datasets
 You can play with the datasets in the `notebooks` folder.
 
+## Build Tools
+
+![image](https://img.shields.io/badge/Flutter-02569B?style=for-the-badge&logo=flutter&logoColor=white)
+</br>
+![image](https://img.shields.io/badge/Dart-0175C2?style=for-the-badge&logo=dart&logoColor=white)
+</br>
+![image](https://img.shields.io/badge/Python-FFD43B?style=for-the-badge&logo=python&logoColor=blue)
+</br>
+![image](https://img.shields.io/badge/firebase-ffca28?style=for-the-badge&logo=firebase&logoColor=black)
+</br>
+![image](https://img.shields.io/badge/Google_Cloud-4285F4?style=for-the-badge&logo=google-cloud&logoColor=white)
+</br>
+![image](https://img.shields.io/badge/TensorFlow-FF6F00?style=for-the-badge&logo=tensorflow&logoColor=white)
+</br>
+![image](https://img.shields.io/badge/OpenCV-27338e?style=for-the-badge&logo=OpenCV&logoColor=white)
+</br>
+![image](https://img.shields.io/badge/GitHub_Actions-2088FF?style=for-the-badge&logo=github-actions&logoColor=white)
+
 
-# License
+## License
 Distributed under the MIT License. See `LICENSE.txt` for more information.
 
-# Contact
+## Contact
 - Google Developers Student Club Delft - dsc.delft@gmail.com
 - Paul Misterka - pawel.mist@gmail.com
 - Mircea Lica - mirceatlx@gmail.com

diff --git a/assets/logo.png b/assets/logo.png
diff --git a/assets/logo_with_background.png b/assets/logo_with_background.png
diff --git a/src/backend/README.md b/src/backend/README.md
diff --git a/src/backend/main.py b/src/backend/main.py
@@ -4,7 +4,7 @@
 import argparse
 from pipeline.mat import Mat
 from google.cloud import storage
-from pipeline.templates import full_pipeline, default_pipeline, training_pipeline, nutrient_pipeline
+from pipeline.templates import full_pipeline, default_pipeline, training_pipeline, nutrient_pipeline, disease_pipeline
 import asyncio
 from pipeline.config import CloudConfig
 from typing import Any
@@ -55,7 +55,7 @@ def main(args: Any):
         imgs = [Mat.read(file) for file in sorted(glob.glob("pipeline/data/D*.JPG"))]
 
     # Get test data
-    imgs = imgs[:3]
+    imgs = imgs[:min(len(imgs), 3)]
 
     # Run the pipeline
     pipeline = nutrient_pipeline(cloud=cloud_config)