Image nav task #333

thibautlavril · 2020-03-13T09:27:35Z

Motivation and Context

Adding a new task: Image goal Navigation.

How Has This Been Tested

Ran new environment in ipython notebook and visualize results.
Unitest on sensor

Types of changes

New feature (non-breaking change which adds functionality)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

codecov · 2020-03-13T09:46:33Z

Codecov Report

Merging #333 into master will increase coverage by 0.15%.
The diff coverage is 96.82%.

@@            Coverage Diff             @@
##           master     #333      +/-   ##
==========================================
+ Coverage   77.36%   77.51%   +0.14%     
==========================================
  Files         108      108              
  Lines        7444     7507      +63     
==========================================
+ Hits         5759     5819      +60     
- Misses       1685     1688       +3

Impacted Files	Coverage Δ
test/test_sensors.py	`96.04% <96.42%> (+0.04%)`	⬆️
habitat/tasks/nav/nav.py	`94.57% <96.96%> (-0.05%)`	⬇️
habitat/config/default.py	`100.00% <100.00%> (ø)`
habitat_baselines/common/environments.py	`97.72% <0.00%> (-2.28%)`	⬇️
habitat/utils/visualizations/utils.py	`78.94% <0.00%> (+1.05%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0cd6410...7f6961a. Read the comment docs.

dhruvbatra · 2020-03-14T22:52:32Z

It will be great to have ImageGoalNav in Habitat. Thanks for working on this!

habitat/tasks/nav/nav.py

mathfac · 2020-03-16T21:51:47Z

Looks great! I would add a sensor test that checks the size of the sensor output and check that it doesn't change till episode switched.

mathfac

I really like the way you addressed missing rotation from PointNav dataset. PR is close to landing, only one thing is missing is a test. I would add similar test to test_get_observations_at, enable the added sensor and check if value is the same as observation at goal position.
Also you can add a new sensor to habitat_all_sensors_test.yaml to have some simple tests for free.

mathfac · 2020-03-19T18:25:59Z

habitat/tasks/nav/nav.py

+    r"""Sensor for ImageGoal observations which are used in ImageGoal Navigation.
+
+    RGBSensor needs to be one of the Simulator sensors.
+    Currently the task is using pointnav Dataset.


Currently the task is using pointnav Dataset. is related to the task, not the sensor. Looks like even with dedicated dataset it may work similar.

habitat/tasks/nav/nav.py

mathfac · 2020-03-19T18:33:20Z

habitat/tasks/nav/nav.py

+        if episode.episode_id == self._current_episode_id:
+            return self._current_image_goal
+
+        self._current_episode_id = episode.episode_id


Maybe, let's move image goal evaluation to separate internal method _get_episode_image_goal or get_pointnav_episode_image_goal. Then it will be easier to update the code when the dataset will have fixed rotation as well.

mathfac · 2020-03-20T21:37:12Z

@thibautlavril, to fix lint error make sure you installed pip install pre-commit && pre-commit install and re-add/reset the files to the commit so checks will run.

drothermel · 2020-03-24T19:50:10Z

@mathfac, I added a basic test_imagegoal_sensor (in the vein of test_pointgaol_sensor), but have some questions about your other suggested tests:

I would add a sensor test that checks the size of the sensor output and check that it doesn't change till episode switched.

What about switching an episode (at least in the way tests are constructed now) would cause the size of the output to change? If nothing, then testing that the output values are constant across the episode (as the added test does) should be sufficient?

Also you can add a new sensor to habitat_all_sensors_test.yaml to have some simple tests for free.

Initially I thought you meant I could just add IMAGEGOAL_SENSOR to the task sensor list in the config, but when I do that some tests fail because the agent isn't provided with an RGB_SENSOR (eg in test/test_baseline_agents.py, test_ppo_agents). Did you mean something else?

I would add similar test to test_get_observations_at, enable the added sensor and check if value is the same as observation at goal position.

The observation is only going to match at the goal position if the angle of the agent is the same as the (randomly chosen and inaccessible, afaik) angle that the goal observation is taken. Do you see a way around this?

dhruvbatra · 2020-04-13T15:56:57Z

Hi @thibautlavril and @drothermel -- thanks for contributing this task, it's a valuable addition.

Would you mind adding a visual (video or gif) showing an example episode (or shortest path)?
See e.g.
-- ObjectNav visuals here: #284
-- VLN visuals here: https://twitter.com/ai_habitat/status/1217117883517419520

Your visuals will serve as a reference for future users and also provide us materials to explain your contributions.

mathfac

Just noticed the update. Thank you a lot for tedious follow up on all the comments Added answers in the code regarding points you mentioned.

Initially I thought you meant I could just add IMAGEGOAL_SENSOR to the task sensor list in the config, but when I do that some tests fail because the agent isn't provided with an RGB_SENSOR (eg in test/test_baseline_agents.py, test_ppo_agents). Did you mean something else?

Yes, that what I meant. If you didn't remove RGB_SENSOR from config then it's strange. Let's just skip this step.

The observation is only going to match at the goal position if the angle of the agent is the same as the (randomly chosen and inaccessible, afaik) angle that the goal observation is taken. Do you see a way around this?

Workaround is to replicate getting the goal rotation from episode_id, but that makes the test too stiff. The test you provided looks great.

I created the task visual @dhruvbatra mentioned, feel free to use it:

Approving, left minor comments and should be good to merge.

habitat/tasks/nav/nav.py

test/test_sensors.py

mathfac · 2020-04-16T19:18:47Z

@thibautlavril, tried to commit your PR, but doesn't have sufficient permissions. Let's add links to the new task in the tasks table and add configs for different datasets:

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
modified: README.md
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ README.md:166 @ Download the Habitat related Gibson dataset following the instructions [here](ht
| Object goal navigation | MatterPort3D | [objectnav_mp3d_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/objectnav/m3d/v1/objectnav_mp3d_v1.zip) | `data/datasets/objectnav/mp3d/v1/` | [`datasets/objectnav/mp3d.yaml`](config
s/datasets/objectnav/mp3d.yaml) | 170 MB |
| [Embodied Question Answering](https://embodiedqa.org/) | MatterPort3D | [eqa_mp3d_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/eqa/mp3d/v1/eqa_mp3d_v1.zip) | `data/datasets/eqa/mp3d/v1/` | [`datasets/eqa/mp3d.yaml`](con
figs/datasets/eqa/mp3d.yaml) | 44 MB |
| [Visual Language Navigation](https://bringmeaspoon.org/) | MatterPort3D | [vln_r2r_mp3d_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/vln/mp3d/r2r/v1/vln_r2r_mp3d_v1.zip) | `data/datasets/vln/mp3d/r2r/v1` | [`datasets/vl
n/mp3d_r2r.yaml`](configs/datasets/vln/mp3d_r2r.yaml) | 2.7 MB |
| [Image goal navigation](https://github.com/facebookresearch/habitat-api/pull/333) | Gibson | [pointnav_gibson_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/pointnav/gibson/v1/pointnav_gibson_v1.zip) | `data/datasets/poin
tnav/gibson/v1/` |  [`datasets/imagenav/gibson.yaml`](configs/datasets/imagenav/gibson.yaml) | 385 MB |
| [Image goal navigation](https://github.com/facebookresearch/habitat-api/pull/333) | MatterPort3D | [pointnav_mp3d_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/pointnav/mp3d/v1/pointnav_mp3d_v1.zip) | `data/datasets/poin
tnav/mp3d/v1/` | [`datasets/imagenav/mp3d.yaml`](configs/datasets/imagenav/mp3d.yaml) | 400 MB |


To use an episode dataset provide related config to the Env in [the example](#example) or use the config for [RL agent training](habitat_baselines/README.md#reinforcement-learning-rl).

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
dded: configs/datasets/imagenav/gibson.yaml
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ configs/datasets/imagenav/gibson.yaml:4 @
ENVIRONMENT:
  MAX_EPISODE_STEPS: 500
SIMULATOR:
  AGENT_0:
    SENSORS: ['RGB_SENSOR']
  HABITAT_SIM_V0:
    GPU_DEVICE_ID: 0
  RGB_SENSOR:
    WIDTH: 256
    HEIGHT: 256
  DEPTH_SENSOR:
    WIDTH: 256
    HEIGHT: 256
TASK:
  TYPE: Nav-v0
  SUCCESS_DISTANCE: 0.2

  SENSORS: ['IMAGEGOAL_SENSOR']
  GOAL_SENSOR_UUID: imagegoal

  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
  SPL:
    TYPE: SPL
    SUCCESS_DISTANCE: 0.2

DATASET:
  TYPE: PointNav-v1
  SPLIT: train
  DATA_PATH: data/datasets/pointnav/gibson/v1/{split}/{split}.json.gz
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
added: configs/datasets/imagenav/mp3d.yaml
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ configs/datasets/imagenav/mp3d.yaml:4 @
ENVIRONMENT:
  MAX_EPISODE_STEPS: 500
SIMULATOR:
  AGENT_0:
    SENSORS: ['RGB_SENSOR']
  HABITAT_SIM_V0:
    GPU_DEVICE_ID: 0
  RGB_SENSOR:
    WIDTH: 256
    HEIGHT: 256
  DEPTH_SENSOR:
    WIDTH: 256
    HEIGHT: 256
TASK:
  TYPE: Nav-v0
  SUCCESS_DISTANCE: 0.2

  SENSORS: ['IMAGEGOAL_SENSOR']
  GOAL_SENSOR_UUID: imagegoal

  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
  SPL:
    TYPE: SPL
    SUCCESS_DISTANCE: 0.2

DATASET:
  TYPE: PointNav-v1
  SPLIT: train
  DATA_PATH: data/datasets/pointnav/mp3d/v1/{split}/{split}.json.gz

Otherwise the PR is good to be merged.

mathfac · 2020-04-21T15:39:56Z

Will send those changes separately. Merging.

* Adds navmesh recomputation without scene file reload * Added recomputeNavMesh function to Simulator in C++ and exposed as recompute_navmesh in python. * Added recomputeNavMesh function to viewer demo in C++ * NavMeshSettings object exposed to python. * Added python CI test for navmesh recomputation: test_recompute_navmesh

Adding a new task: Image goal Navigation.

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Mar 13, 2020

mathfac reviewed Mar 16, 2020

View reviewed changes

habitat/tasks/nav/nav.py Show resolved Hide resolved

mathfac reviewed Mar 19, 2020

View reviewed changes

thibautlavril marked this pull request as ready for review March 24, 2020 20:00

mathfac approved these changes Apr 13, 2020

View reviewed changes

habitat/tasks/nav/nav.py Outdated Show resolved Hide resolved

habitat/tasks/nav/nav.py Outdated Show resolved Hide resolved

test/test_sensors.py Show resolved Hide resolved

test/test_sensors.py Show resolved Hide resolved

thibautlavril and others added 17 commits April 16, 2020 14:53

Duplicate PointGoalSensor

297b0e9

Add observation for image goal

3be8961

Remove unsued parameters

34884b8

Update way angle is sampled

b058b53

Reformat with black

541d526

Revert changes on example

3cf0710

Remove "rgb" hardocding

92d093c

Update ImageGoal sensor documentation

996c715

Update way rgb_sensor_uuid is detected

f797839

Update imagenav.yaml

ca77006

Add state in image goal sensor

8ac9f20

Add _get_pointnav_episode_image_goal function

a4eccac

Reformat with black

a5d2353

added test_imagegoal_sensor

235468b

Change episode uniq id

f67e711

Check that imagegoal change between episodes

6a1dfd7

Add check on the shape

7f6961a

thibautlavril force-pushed the image-nav-task branch from 69954d8 to 7f6961a Compare April 16, 2020 13:30

mathfac merged commit 3e3181e into facebookresearch:master Apr 21, 2020

dannymcy pushed a commit to dannymcy/habitat-lab that referenced this pull request Jul 8, 2024

Image nav task (facebookresearch#333)

4214231

Adding a new task: Image goal Navigation.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image nav task #333

Image nav task #333

thibautlavril commented Mar 13, 2020 •

edited by mathfac

Loading

codecov bot commented Mar 13, 2020 •

edited

Loading

dhruvbatra commented Mar 14, 2020

mathfac commented Mar 16, 2020 •

edited

Loading

mathfac left a comment

mathfac Mar 19, 2020

mathfac Mar 19, 2020

mathfac commented Mar 20, 2020

drothermel commented Mar 24, 2020

dhruvbatra commented Apr 13, 2020

mathfac left a comment •

edited

Loading

mathfac commented Apr 16, 2020

mathfac commented Apr 21, 2020

Image nav task #333

Image nav task #333

Conversation

thibautlavril commented Mar 13, 2020 • edited by mathfac Loading

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

codecov bot commented Mar 13, 2020 • edited Loading

Codecov Report

dhruvbatra commented Mar 14, 2020

mathfac commented Mar 16, 2020 • edited Loading

mathfac left a comment

Choose a reason for hiding this comment

mathfac Mar 19, 2020

Choose a reason for hiding this comment

mathfac Mar 19, 2020

Choose a reason for hiding this comment

mathfac commented Mar 20, 2020

drothermel commented Mar 24, 2020

dhruvbatra commented Apr 13, 2020

mathfac left a comment • edited Loading

Choose a reason for hiding this comment

mathfac commented Apr 16, 2020

mathfac commented Apr 21, 2020

thibautlavril commented Mar 13, 2020 •

edited by mathfac

Loading

codecov bot commented Mar 13, 2020 •

edited

Loading

mathfac commented Mar 16, 2020 •

edited

Loading

mathfac left a comment •

edited

Loading