Skip to content

Commit

Permalink
Merge pull request #787 from WildMeOrg/development-docker-instruction…
Browse files Browse the repository at this point in the history 
…-rework

development docker instruction rework
  • Loading branch information
@TanyaStere42
TanyaStere42 authored Sep 25, 2024
2 parents 50cda8c + ebfaa55 commit f89e32c
Show file tree
Hide file tree
Showing 3 changed files with 60 additions and 163 deletions.
117 changes: 3 additions & 114 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,9 @@
Wildbook is an open source software framework to support mark-recapture, molecular ecology, and social ecology studies. The biological and statistical communities already support a number of excellent tools, such as Program MARK,GenAlEx, and SOCPROG for use in analyzing wildlife data. Wildbook is a complementary software application that:

- provides a scalable and collaborative platform for intelligent wildlife data storage and management, including advanced, consolidated searching

- provides an easy-to-use software suite of functionality that can be extended to meet the needs of wildlife projects, especially where individual identification is used

- provides an API to support the easy export of data to cross-disciplinary analysis applications (e.g., GenePop ) and other software (e.g., Google Earth)

- provides a platform that supports the exposure of data in biodiversity databases (e.g., GBIF and OBIS)

- provides a platform for animal biometrics that supports easy data access and facilitates matching application deployment for multiple species

## Getting Started with Wildbook
Expand Down Expand Up @@ -43,12 +39,10 @@ You will want to work in a branch when doing any feature development you want to
### Set Up Development Environment with Docker
For easiest development, you will need to set up your development environment to work with Docker. See [`devops/README.md`](devops/README.md) for detailed instructions.

### Deploy frontend
To setup frontend, we need to deploy the React build to Wildbook, please follow the detailed instructions provided in the `frontend/README.md` file within the project directory.

### Making Local Changes
Make the code changes necessary for the issue you're working on. The following git commands may prove useful.
Make the code changes necessary for the issue you're working on. You will need to either redeploy your war file (see [`devops/README.md`](devops/README.md)) or redeploy your front end directly (see [`frontend.README.md`](frontend/README.md)) for testing locally.

The following git commands may prove useful.
* `git log`: lastest commits of current branch
* `git status`: current staged and unstaged modifications
* `git diff --staged`: the differences between the staging area and the last commit
Expand Down Expand Up @@ -91,115 +85,10 @@ We provide support during regular office hours on Mondays and Tuesdays.
## Support resources
* User documentation is available at [Wild Me Documentation](http://wildbook.docs.wildme.org)
* For user support, visit the [Wild Me Community Forum](https://community.wildme.org)
* For contribution guidelines, visit [Wildbook Code Contribution Guidelines](https://wildbook.docs.wildme.org/contribute/code-guide.html)
* For developer support, visit the [Wild Me Development Discord](https://discord.gg/zw4tr3RE4R)
* Email the team at opensource@wildme.org

## Contribution Guidelines

### Variable naming conventions
* Camel case
* Don’t use single-letter variable names (no matter how temporary you think the code is)
* Code should be clear enough to speak for itself without comments, but use your judgement on if a comment is necessary
* Code for clarity rather than for efficiency (one-liners are cool, but not at the expense of future obfuscation)

### Overall outline of code framework<
Spell out how .jsp files relate to servlet files relate to java files, etc. Someone new to the codebase should be able to orient themselves based on your notes.

### Java/jsp style
Initialize variables and type signatures at the abstract/interface level when possible.

Instead of:

```
ArrayList encounters = new ArrayList<Encounter>();
...
public int getMax(ArrayList<int> numbers) {
```

Try:

```
List encounters = new ArrayList<Encounter>();
...
public int getMax(Collection<int> numbers) {
```

It’s easier to read and more intuitive for a function to take a Map or List than a HashMap or ArrayList.

The List interface defines how we want that variable to behave, and whether it’s an ArrayList or LinkedList is incidental. Keeping the variable and method signatures abstract means we can change the implementation later (eg swapping ArrayList->LinkedList) without changing the rest of our code.
https://stackoverflow.com/questions/2279030/type-list-vs-type-arraylist-in-java

Related: when writing utility methods, making the input type as abstract as possible makes the method versatile. See Util.asSortedList in Wildbook: since the input is an abstract Collection, it can accept a List, Set, PriorityQueue, or Vector as input, and return a sorted List.

Runtime (not style): Use Sets (not Lists or arrays) if you’re only keeping track of collection membership / item uniqueness.

Instead of:

```
List<MarkedIndividual> uniqueIndividuals = new ArrayList<MarkedIndividual>();
for(Encounter currentEncounter: encounters){
MarkedIndividual currentInd = enc.getIndividual();
if !(uniqueIndividuals.contains(currentInd) {
uniqueIndividuals.add(currentInd);
doStuff();
```
Try:

```
Set<MarkedIndividual> uniqueIndividuals = new HashSet<MarkedIndividual>();
for(Encounter currentEncounter: encounters){
MarkedIndividual currentInd = enc.getIndividual();
if !(uniqueIndividuals.contains(currentInd) {
uniqueIndividuals.add(currentInd);
doStuff();
```

The reason is a little deep in the data types. Sets are defined as unordered collections of unique elements; and Lists/arrays are ordered collections with no bearing on element-uniqueness. If the order of a collection doesn’t matter and you’re just checking membership, you’ll have faster runtime using a Set.

Sets implement contains, add, and remove methods much faster than lists [contains is O(log(n)) vs O(n) runtime]. A list has to iterate through the entire list every time it runs contains (it checks each item once at a time) while a set (especially a HashSet) keeps track of an item index for quick lookup.


Use for-each loops aka “enhanced for loops” to make loops more concise and readable.

Instead of:

```
for (int i=0; i<encounters.length(); i++) {
Encounter enc = encounters.get(i)
doStuff();
```

try:

```
for (Encounter enc: encounters) {
doStuff();
```

Note that in both cases you might want to check if `encounters == null` if relevant, but you rarely need to check if `encounters.length()>0` because the for-loops take care of that.

Also note that if you want access to the `i` variable for logging or otherwise, the classic for-loop is best.


`Util.stringExists` is shorthand for a common string check:

Instead of:

```
if (str!=Null && !str.equals("")) {
doStuff();
```

Try:

```
if (Util.stringExists(str)) {
doStuff();
```

This method also checks for the strings “none” and “unknown” which have given us trouble in displays in the past.

## History
Wildbook started as a collaborative software platform for globally-coordinated whale shark (Rhincodon typus ) research as deployed in the Wildbook for Whale Sharks (now part of http://www.sharkbook.ai). After many requests to use our software outside of whale shark research, it is now an open source, community-maintained standard for mark-recapture studies.

Expand Down
81 changes: 51 additions & 30 deletions devops/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,52 +4,73 @@ We provide two different containerized Wildbooks using Docker. Unless you are se

## Development Docker

All support materials are found in the `devops/development/` subdirectory. This launches docker containers sufficient for you to deploy a `wildbook.war` file which you have developed
via your own Java environment.
All support materials are found in the `devops/development/` subdirectory. This launches docker containers sufficient for you to deploy a `wildbook.war` file which you have developed via your own Java environment.

### Overview of docker containers deployed

- **db** - postgresql database for storing Wildbook data
- **wildbook** -- the tomcat (java server) container which runs Wildbook
- **opensearch** -- runs OpenSearch to support searching in Wildbook
- **smtp** -- handles outgoing email (for password reset etc.) It is beyond the scope of this current document to address setting up email relays on the open internet.
Environment variables for this are set in `.env` file - see below.
- **smtp** -- handles outgoing email (for password reset etc.) It is beyond the scope of this current document to address setting up email relays on the open internet. Environment variables for this are set in `.env` file.
- ~~**wbia**~~ - Presently, this _does not_ start a local WBIA (image analysis) docker container. The current roadmap is focused on removing WBIA as a requirement as we modernize our machine learning tech.

### Setup and running
## Prereqs
You need the following installed on your system:
* `default-jdk`
* `build-essential`
* `maven`
* `npm`
* `node`
* `docker-compose`

## System setup
1. Run `sudo sysctl -w vm.max_map_count=262144` (A requirement for OpenSearch, it only needs to be run once on your system.)
1. In `devops/development/` folder, create a `.env` file with a copy the contents of `_env.template`. By default, no changes should be needed, but you can edit this new file if needed.
1. In your terminal, create your base directory (value of `WILDBOOK_BASE_DIR` from `.env` file above) and the required subdirectories. The default is `~/wildbook-dev/`). For example:
```
mkdir -p ~/wildbook-dev/webapps/wildbook
mkdir ~/wildbook-dev/logs
```
1. Deploy your `.war` file (see section below) in the above `wildbook/` directory, using `jar`:
```
cd ~/wildbook-dev/webapps/wildbook
jar -xvf /path/to/wildbook-xxx.war
```
1. Return to the `devops/development/` directory in the wildbook repo
1. Run `docker-compose up [-d]`, which launches all of the aforementioned docker images
1. To verify successful launch, open in browser http://localhost:81/ when tomcat has started. Default login of username/password `tomcat`/`tomcat123` should work.

### Development environment setup for compiling Wildbook

1. Run `npm install react-app-rewired`
1. `git clone` the Wildbook repo (referred to as the **code directory** going forward)

### Code directory setup
1. In `devops/development/`, create a `.env` file with a copy the contents of `_env.template`. By default, no changes should be needed.
1. `cd` to the root of the code directory
1. run `npm install`
1. run `chmod +x .husky/pre-commit` to enable husky linting
1. `cd /frontend`
1. run `npm install` to install all dependencies
1. create a `.env` for React environment variables.
1. Add the public URL: PUBLIC_URL= /react/
1. Add the site name: SITE_NAME=Amphibian Wildbook

### Deploy directory setup
1. Create your **deploy directory**, matching the value of `WILDBOOK_BASE_DIR` in the .env file. The default is `~/wildbook-dev/`)
1. Create the necessary subdirectories
```
wildbook-dev
|--logs
|--webapps
|--wildbook
```

### Build war file
To run Wildbook in the development docker environment, even to try out the software, you need a "war file" which is made by compiling the Wildbook java project.
This requires some software to be set up on your development machine:

- Java JDK (`openjdk`) and `build-essential` linux package, as well as `maven` <span style="background-color: yellow;">[probably a link to generic setup doc elsewhere?]</span>
- node and npm for React build <span style="background-color: yellow;">[likewise, link to generic help?]</span>, more details in [frontend/README.md](../frontend/README.md).
To create the war file:
1. `cd` to the root of the code directory
1. Build your war file with `mvn clean install`
1. verify the war file was created in `target/wildbook-X.Y.Z.war` (where X.Y.Z is the current version number).
1. cd to the deploy directory `cd ~/wildbook-dev/webapps/wildbook`
1. deploy your warfile `jar -xvf /code/directory/Wildbook/target/wildbook-X.Y.Z.war`

### Deploy
1. `cd` to the `devops/development` directory in the code repo
1. run `docker-compose up [-d]`
1. To verify successful launch, open in browser `http://localhost:81/` when tomcat has started. Default login of username/password `tomcat`/`tomcat123` should work.

#### Compiling
Note: if you're running docker as root, you may way to explicitly set your deploy directory path to include your user, i.e., `WILDBOOK_BASE_DIR=~USER/wildbook-dev`

Once the above requirements are met, the war file can be created by running `mvn clean install`. This creates the war file to be used in `target/wildbook-X.Y.Z.war` (with current version number).
### Rebuild war file for local testing

If you make code changes and compile new war files, they can be deployed into the `wildbook` dir (as in step 3 above) and then tomcat restarted with
`docker-compose restart wildbook`.
If you make code changes and want to test them locally, you can create and deploy a new war file using the Build war file and Deploy instructions above. Then use `docker-compose restart wildbook` to test your changes.

---
Note: If you are only making changes to the React frontend code, see the [frontend/README.md](../frontend/README.md) for a way to rebuild and deploy the frontend changes only.

## Deploy (e.g. Production) Docker Image - DRAFT

Expand Down
25 changes: 6 additions & 19 deletions frontend/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,29 +1,16 @@
### pre-reqs
- node, npm
- `npm install react-app-rewired`
## Frontend Setup
The frontend is currently split between native tomcat functions, jsp pages, and a react app. New development is targeting a full react app rewrite. This setup is focused on the react specific requirements. However, you must do the full system setup referenced in the [development README](../devops/README.md) for these to work.

### steps to setup dev environment
1. cd to the root folder of your codebase, for example `/Wildbook`
2. run `npm install`
3. run `chmod +x .husky/pre-commit` to enable husky pre-commit hooks
4. cd to the react folder `Wildbook/frontend/`,
5. also run `npm install` to install all dependencies
6. now you should be able to commit and Husky will check your code for any issues before each commit
7. create a `.env` for React environment variables under the root of `REPO/frontend/`. In this file:
1. Add the public URL. For local tomcat development, use `PUBLIC_URL=http://localhost:81/react/` (or whatever port your local server is running on). For public deployment, use the following, where `public.url.example.com` is your deployed URL: `PUBLIC_URL=https://public.url.example.com/react/`
2. Add site name like this: `SITE_NAME=Amphibian Wildbook`
### Build and deploy react-only changes
If you are working on react-only work, you can test your changes without updating the full war file.

### steps to set up deploy directory
1. create a folder `react` under deployed `wildbook/` dir

### steps to build and deploy react
#### using npm to build and deploy to your local deployment
#### Use npm to build and deploy to your local deployment
If you have your dev environment set up correctly, this will build the React app and copy it into your local deployment directory for you.
1. cd to `REPO/frontend/`
2. run `npm run deploy-dev`
3. refresh your browser page by visiting either `http://localhost:81/react/` for local testing or `https://public.url.example.com/react/` for the public-facing deployment

#### manually building and deploying
#### Manually build and deploy
1. cd to `REPO/frontend/` and run `npm run build`
2. copy everything under `frontend/build/` to the deployed `wildbook/react/` directory you created during setup
3. refresh your browser page by visiting either `http://localhost:81/react/` for local testing or `https://public.url.example.com/react/` for the public-facing deployment

0 comments on commit f89e32c

Please sign in to comment.