Merge branch 'develop'

README.md

@@ -72,22 +72,47 @@ bam.remove_addons()
 bam.remove_addon('benchbot-addons/ssu,benchbot-addons/sqa')
 ```
 
-## How to add your own add-ons
+## Creating your own add-on content
 
-There are two different types of add-ons: 'official' add-ons and third-party add-ons.
+Add-ons are designed to make it easy to add your own local content to a BenchBot installation. You can add your own local content to the "local add-ons" folder provided with your install. The location on your machine can be printed via the following:
 
-'Official' are add-ons that we've verified, and are stored in our [benchbot-addons](https://github.com/benchbot-addons) GitHub organisation. You can get a full list of official add-ons through the `manager.official_addons()` helper function, or `benchbot_install --list-addons` script in the [BenchBot software stack](https://github.com/qcr/benchbot).
+```python
+from benchbot_addons import manager as bam
+
+print(bam.local_addon_path())
+```
+
+BenchBot expects add-on content to be in named folders denoting the type of content. For example, robots must be in a folder called `'robots'`, tasks in a folder called `'tasks'`, and so on. A list of valid content types is available via the `SUPPORTED_TYPES` field in the add-ons manager.
+
+Below is an example of the process you would go through to create your own custom task locally:
+
+1. Find the location for your custom local add-ons:
+   ```
+   u@pc:~$ python3 -c 'from benchbot_addons import manager as bam; print(bam.local_addon_path())'
+   /home/ben/repos/benchbot/addons/benchbot_addons/.local/my_addons
+   ```
+2. Create the following YAML file for your task: `/home/ben/repos/benchbot/addons/benchbot_addons/.local/my_addons/tasks/my_task.yaml`
+3. Use the fields described below in the [task add-ons specification](#task-add-ons) to define your task
+4. Save the file
+
+Done. Your new custom task should now be available for use in your BenchBot system (e.g. [`benchbot_run --list-tasks`](https://github.com/qcr/benchbot)).
 
-Third-party add-ons only differ in that we haven't looked at them, and they can be hosted anywhere on GitHub you please.
+## Sharing your custom add-ons
 
-Creating all add-ons is exactly the same process, the only difference is whether the repository is inside or outside of the [benchbot-addons](https://github.com/benchbot-addons) GitHub organisation:
+Custom add-on content can be grouped together into an add-on package, of which there are two different types: 'official' and third-party.
+
+'Official' packages are those we've verified, and are stored in our [benchbot-addons](https://github.com/benchbot-addons) GitHub organisation. You can get a full list of official add-on packages through the `manager.official_addons()` helper function, or `benchbot_install --list-addons` script in the [BenchBot software stack](https://github.com/qcr/benchbot).
+
+Third-party add-on packages differ only in that we haven't looked at them, and they can be hosted anywhere on GitHub you please.
+
+Creating all add-on packages is exactly the same process, the only difference is whether the repository is inside or outside of the [benchbot-addons](https://github.com/benchbot-addons) GitHub organisation:
 
 1. Create a new GitHub repository
 2. Add folders corresponding to the type of content your add-ons provide (i.e. an environments add-on has an `environments` directory at the root).
 3. Add YAML / JSON files for your content, and make sure they match the corresponding format specification from the section below
 4. Add in any extra content your add-on may require: Python files, simulator binaries, images, etc. (if your add-on gets too big for a Git repository, you can zip the content up, host it somewhere, and use the `.remote` metadata file described in the next section)
-5. Decide if your add-on is dependent on any others, and declare any dependencies in a `.dependencies` file
-6. Push everything up to git on your default branch
+5. Decide if your package has any dependencies, and declare them using the appropriate `.dependencies*` files
+6. Push everything up to GitHub on your default branch
 
 _**Note:** it's a good idea to only include one type of add-on per repository as it makes your add-on package more usable for others. It's not a hard rule though, so feel free to add multiple folders to your add-on if you require._
 
@@ -155,6 +180,20 @@ Evaluation methods expect the following named functions:
 | `'evaluate'` | `fn(dict: results, list: ground_truths) -> dict` | Evaluates the performance using a `results` dictionary, and returns a dictionary of containing the scores. It also takes a list of dictionaries containing each ground truth that will be used in evaluation. |
 | `'combine'`  | `fn(list: scores) -> dict`                       | Takes a list of `scores` dictionaries, and returns an aggregate score. If this method isn't declared, [`benchbot_eval`](https://github.com/qcr/benchbot_eval) won't return a summary score.                   |
 
+### Example method add-ons
+
+A YAML file, that must in a folder called `examples` in the root of the add-on package (e.g. `examples/my_example.yaml`).
+
+The following keys are supported for example add-ons:
+
+| Key                   | Required | Description                                                                                                                                                                                                                                  |
+| --------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `name`                | Yes      | A string used to refer to this example (must be unique!)                                                                                                                                                                                     |
+| `native_command`      | Yes      | A string describing the command used to run your example natively, relative to the directory of this YAML file! For example running your `my_example.py` file which is in the same director as this YAML would be `python3 ./my_example.py`. |
+| `container_directory` | No       | Directory to be used for Docker's build context. The submission process will automatically look for a file called `Dockerfile` in that directory unless the `'container_filename'` key is also provided.                                     |
+| `container_filename`  | No       | Custom filename for your example's Dockerfile. `Dockerfile` in `container_directory` will be used if this key is not included. This path is relative to this YAML file, **not** `'container_directory'`.                                     |
+| `description`         | No       | A string describing what the example is and how it works. Should be included if you want users to understand how your example can be expanded.                                                                                               |
+
 ### Format definition add-ons
 
 A YAML file, that must exist in a folder called `formats` in the root of the add-on package (e.g. `formats/my_format.yaml`).
@@ -226,7 +265,8 @@ The following keys are supported for task add-ons:
 | `'name'`           | Yes      | A string used to refer to this task (must be unique!).                                                                                                                                                                      |
 | `'actions'`        | Yes      | A list of named connections to be provided as actions through the [BenchBot API](https://github.com/qcr/benchbot_api). Running this task will fail if the robot doesn't provide these named connections.                    |
 | `'observations'`   | Yes      | A list of named connections to be provided as observations through the [BenchBot API](https://github.com/qcr/benchbot_api). Running this task will fail if the robot doesn't provide these named connections.               |
-| `'results_format'` | Yes      | A string naming the format for results. The format must be installed, as [BenchBot API](https://github.com/qcr/benchbot_api) will use the format's functions to provide the user with empty results.                        |
+| `'localisation'`   | No       | A string describing the level of localisation. Only supported values currently are `'ground_truth'` and `'noisy'`. The default value is '`ground_truth`'.                                                                   |
+| `'results_format'` | No       | A string naming the format for results. The format must be installed, as [BenchBot API](https://github.com/qcr/benchbot_api) will use the format's functions to provide the user with empty results.                        |
 | `'description'`    | No       | A string describing what the task is, and how it works. Should be included if you want users to understand what challenges your task is trying to capture.                                                                  |
 | `'type'`           | No       | A string describing what robot / environment types are valid for this task. For example, a task that provides a magic image segmentation sensor would only be made available for `'sim_unreal'` type robots / environments. |
 | `'scene_count'`    | No       | Integer representing the number of scenes (i.e. environment variations required for a task). If omitted, a default value of 1 will be used for the task.                                                                    |

benchbot_addons/manager.py

@@ -19,9 +19,11 @@
 FILENAME_PYTHON_DEPENDENCIES = '.dependencies-python'
 FILENAME_REMOTE = '.remote'
 
+HASH_SHORT = 8
+
 KEY_FILE_PATH = '_file_path'
 
-HASH_SHORT = 8
+LOCAL_NAME = '.local/my_addons'
 
 SUPPORTED_TYPES = [
     'batches', 'environments', 'evaluation_methods', 'examples', 'formats',
@@ -36,10 +38,6 @@ def _abs_path(path):
         os.path.join(os.path.dirname(__file__), path)))
 
 
-def _addon_path(repo_user, repo_name):
-    return os.path.join(_install_location(), repo_user, repo_name)
-
-
 def _install_location():
     return _abs_path(
         os.environ.get(ENV_INSTALL_LOCATION, DEFAULT_INSTALL_LOCATION))
@@ -64,6 +62,22 @@ def _validate_type(type_string):
             (type_string, SUPPORTED_TYPES))
 
 
+def addon_path(repo_user, repo_name):
+    return os.path.join(_install_location(), repo_user, repo_name)
+
+
+def dirty_addons():
+    dirty = []
+    for n in get_state().keys():
+        _, repo_user, repo_name, _ = _parse_name(n)
+        if (run('[[ -z $(git status -s --porcelain) ]]',
+                shell=True,
+                executable='/bin/bash',
+                cwd=addon_path(repo_user, repo_name)).returncode != 0):
+            dirty.append(n)
+    return dirty
+
+
 def dump_state(state):
     # State is a dictionary with:
     # - keys for each installed addon
@@ -164,9 +178,13 @@ def load_yaml_list(filenames_list):
     return [load_yaml(f) for f in filenames_list]
 
 
+def local_addon_path():
+    return addon_path(*LOCAL_NAME.split('/'))
+
+
 def install_addon(name):
     url, repo_user, repo_name, name = _parse_name(name)
-    install_path = _addon_path(repo_user, repo_name)
+    install_path = addon_path(repo_user, repo_name)
 
     print("Installing addon '%s' in '%s':" % (name, _install_location()))
 
@@ -283,8 +301,8 @@ def install_external_deps(dry_mode=False):
     state = get_state()
     deps = []
     for a in ([
-            _addon_path(_parse_name(k)[1],
-                        _parse_name(k)[2]) for k in state.keys()
+            addon_path(_parse_name(k)[1],
+                       _parse_name(k)[2]) for k in state.keys()
     ]):
         fn = os.path.join(a, FILENAME_PYTHON_DEPENDENCIES)
         if os.path.exists(fn):
@@ -307,16 +325,22 @@ def print_state():
     if not state.keys():
         print("\tNone.")
     else:
-        for k, v in state.items():
+        sorted_keys = sorted(state.keys())
+        for k in sorted_keys:
             print("\t%s (%s%s)" %
-                  (k, v['hash'][:HASH_SHORT],
-                   ', with remote content' if 'remote' in v else ''))
+                  (k, state[k]['hash'][:HASH_SHORT],
+                   ', with remote content' if 'remote' in state[k] else ''))
     print(
         "\nOur GitHub organisation (https://github.com/benchbot-addons) "
-        "contains all of our official add-ons.\nThe following are available, "
-        "with more details available at the above URL:")
-    for o in official_addons():
-        print("\t%s" % o)
+        "contains all of our official add-ons.\nThe following additional "
+        "official add-ons are available, with more details at the above URL:")
+    missing_officials = sorted(
+        list(set(official_addons()) - set(state.keys())))
+    if missing_officials:
+        for o in missing_officials:
+            print("\t%s" % o)
+    else:
+        print("\tNone!")
 
     print("\nIf you would like to add your community-created add-on to the "
           "official list, please follow the\ninstructions here:\n\t"
@@ -343,7 +367,7 @@ def outdated_addons():
     }
     outdated = []
     for a in state.keys():
-        cwd = _addon_path(*_parse_name(a)[1:3])
+        cwd = addon_path(*_parse_name(a)[1:3])
         if (run('git rev-parse HEAD', cwd=cwd, **
                 cmd_args).stdout.decode('utf8').strip() !=
                 run('git rev-parse origin/HEAD', cwd=cwd, **
@@ -354,7 +378,7 @@ def outdated_addons():
 
 def remove_addon(name):
     url, repo_user, repo_name, name = _parse_name(name)
-    install_path = _addon_path(repo_user, repo_name)
+    install_path = addon_path(repo_user, repo_name)
     install_parent = os.path.dirname(install_path)
 
     # Confirm the addon exists

setup.py

@@ -4,7 +4,7 @@
     long_description = fh.read()
 
 setup(name='benchbot_addons',
-      version='2.0.0',
+      version='2.1.0',
       author='Ben Talbot',
       author_email='b.talbot@qut.edu.au',
       description=
@@ -14,13 +14,7 @@
       packages=find_packages(),
       install_requires=['pyyaml', 'requests'],
       classifiers=(
-          "Programming Language :: Python :: 2",
-          "Programming Language :: Python :: 2.7",
           "Programming Language :: Python :: 3",
-          "Programming Language :: Python :: 3.3",
-          "Programming Language :: Python :: 3.4",
-          "Programming Language :: Python :: 3.5",
-          "Programming Language :: Python :: 3.6",
           "License :: OSI Approved :: BSD License",
           "Operating System :: OS Independent",
       ))