Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nuImages data export #447

Merged
merged 17 commits into from
Aug 17, 2020
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -51,7 +51,7 @@ For an advanced installation, see [installation](https://github.com/nutonomy/nus
### nuImages setup
To download nuImages you need to go to the [Download page](https://www.nuscenes.org/download),
create an account and agree to the nuScenes [Terms of Use](https://www.nuscenes.org/terms-of-use).
For the devkit to work you will need to download *all* archives.
For the devkit to work you will need to download *at least the metadata and samples*, the *sweeps* are optional.
Please unpack the archives to the `/data/sets/nuimages` folder \*without\* overwriting folders that occur in multiple archives.
Eventually you should have the following folder structure:
```
@@ -63,6 +63,7 @@ Eventually you should have the following folder structure:
If you want to use another folder, specify the `dataroot` parameter of the NuImages class (see tutorial).

### Getting started with nuImages

Please follow these steps to make yourself familiar with the nuImages dataset:
- Get the [nuscenes-devkit code](https://github.com/nutonomy/nuscenes-devkit).
- Run the tutorial using:
@@ -127,11 +128,11 @@ To install this expansion, please follow these steps:

### Getting started with nuScenes
Please follow these steps to make yourself familiar with the nuScenes dataset:
- Read the [dataset description](https://www.nuscenes.org/overview).
- [Explore](https://www.nuscenes.org/explore/scene-0011/0) the lidar viewer and videos.
- Read the [dataset description](https://www.nuscenes.org/nuscenes#overview).
- [Explore](https://www.nuscenes.org/nuscenes#explore) the lidar viewer and videos.
- [Download](https://www.nuscenes.org/download) the dataset.
- Get the [nuscenes-devkit code](https://github.com/nutonomy/nuscenes-devkit).
- Read the [online tutorial](https://www.nuscenes.org/tutorial) or run it yourself using:
- Read the [online tutorial](https://www.nuscenes.org/nuscenes#tutorials) or run it yourself using:
```
jupyter notebook $HOME/nuscenes-devkit/python-sdk/tutorials/nuscenes_tutorial.ipynb
```
9 changes: 3 additions & 6 deletions docs/faqs.md
Original file line number Diff line number Diff line change
@@ -6,17 +6,14 @@ On this page we try to answer questions frequently asked by our users.
- For issues and bugs *with the devkit*, file an issue on [Github](https://github.com/nutonomy/nuscenes-devkit/issues).
- For any other questions, please post in the [nuScenes user forum](https://forum.nuscenes.org/).

- Can I use nuScenes for free?
- For non-commercial use [nuScenes is free](https://www.nuscenes.org/terms-of-use), e.g. for educational use and some research use.
- Can I use nuScenes and nuImages for free?
- For non-commercial use [nuScenes and nuImages are free](https://www.nuscenes.org/terms-of-use), e.g. for educational use and some research use.
- For commercial use please contact [nuScenes@nuTonomy.com](mailto:nuScenes@nuTonomy.com). To allow startups to use our dataset, we adjust the pricing terms to the use case and company size.

- How can I participate in the nuScenes challenges?
- See the overview site for the [object detection challenge](https://www.nuscenes.org/object-detection).
- See the overview site for the [tracking challenge](https://www.nuscenes.org/tracking).

- What's next for nuScenes?
- Raw IMU & GPS data.
- Object detection, tracking and other challenges (see above).
- See the overview site for the [prediction challenge](https://www.nuscenes.org/prediction).

- How can I get more information on the sensors used?
- Read the [Data collection](https://www.nuscenes.org/data-collection) page.
6 changes: 3 additions & 3 deletions docs/installation.md
Original file line number Diff line number Diff line change
@@ -107,17 +107,17 @@ pip install -r setup/requirements.txt

## Setup environment variable
Finally, if you want to run the unit tests you need to point the devkit to the `nuscenes` folder on your disk.
Set the NUSCENES environment variable to point to your data folder, e.g. `/data/sets/nuscenes`:
Set the NUSCENES environment variable to point to your data folder:
```
export NUSCENES="/data/sets/nuscenes"
```
or for nuImages:
or for NUIMAGES:
```
export NUIMAGES="/data/sets/nuimages"
```

## Verify install
To verify your environment run `python -m unittest` in the `python-sdk` folder.
You can also run `assert_download.py` in the `nuscenes/scripts` folder.
You can also run `assert_download.py` in the `python-sdk/nuscenes/tests` and `python-sdk/nuimages/tests` folders to verify that all files are in the right place.

That's it you should be good to go!
66 changes: 66 additions & 0 deletions python-sdk/nuimages/export/export_release.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# nuScenes dev-kit.
# Code written by Holger Caesar, 2020.

import fire
import os
import json
import tarfile
from typing import List


def export_release(dataroot='/data/sets/nuimages', version: str = 'v1.0') -> None:
"""
This script tars the image and metadata files for release on https://www.nuscenes.org/download.
:param dataroot: The nuImages folder.
:param version: The nuImages dataset version.
"""
# Create export folder.
export_dir = os.path.join(dataroot, 'export')
if not os.path.isdir(export_dir):
os.makedirs(export_dir)

# Determine the images from the mini split.
mini_src = os.path.join(dataroot, version + '-mini')
with open(os.path.join(mini_src, 'sample_data.json'), 'r') as f:
sample_data = json.load(f)
file_names = [sd['filename'] for sd in sample_data]

# Hard-code the mapping from archive names to their relative folder paths.
archives = {
'all-metadata': [version + '-train', version + '-val', version + '-test', version + '-mini'],
'all-samples': ['samples'],
'all-sweeps-cam-back': ['sweeps/CAM_BACK'],
'all-sweeps-cam-back-left': ['sweeps/CAM_BACK_LEFT'],
'all-sweeps-cam-back-right': ['sweeps/CAM_BACK_RIGHT'],
'all-sweeps-cam-front': ['sweeps/CAM_FRONT'],
'all-sweeps-cam-front-left': ['sweeps/CAM_FRONT_LEFT'],
'all-sweeps-cam-front-right': ['sweeps/CAM_FRONT_RIGHT'],
'mini': [version + '-mini'] + file_names
}

# Pack each folder.
for key, folder_list in archives.items():
out_path = os.path.join(export_dir, 'nuimages-%s-%s.tgz' % (version, key))
if os.path.exists(out_path):
print('Warning: Skipping export for file as it already exists: %s' % out_path)
continue
print('Compressing archive %s...' % out_path)
pack_folder(out_path, dataroot, folder_list)


def pack_folder(out_path: str, dataroot: str, folder_list: List[str], tar_format: str = 'w:gz') -> None:
"""
:param out_path:
:param dataroot: The nuImages folder.
:param folder_list: List of files or folders to include in the archive.
:param tar_format: The compression format to use. See tarfile package for more options.
"""
tar = tarfile.open(out_path, tar_format)
for name in folder_list:
folder_path = os.path.join(dataroot, name)
tar.add(folder_path, arcname=name)
tar.close()


if __name__ == '__main__':
fire.Fire(export_release)
34 changes: 7 additions & 27 deletions python-sdk/nuimages/nuimages.py
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@
from PIL import Image, ImageDraw
from pyquaternion import Quaternion

from nuimages.utils.utils import annotation_name, mask_decode, get_font
from nuimages.utils.utils import annotation_name, mask_decode, get_font, name_to_index_mapping
from nuscenes.utils.color_map import get_colormap

PYTHON_VERSION = sys.version_info[0]
@@ -75,7 +75,7 @@ def __init__(self,
self.color_map = get_colormap()

if verbose:
print("Done loading in {:.1f} seconds (lazy={}).\n======".format(time.time() - start_time, self.lazy))
print("Done loading in {:.3f} seconds (lazy={}).\n======".format(time.time() - start_time, self.lazy))

# ### Internal methods. ###

@@ -555,29 +555,7 @@ def get_segmentation(self,
sample_data = self.get('sample_data', sd_token)
assert sample_data['is_key_frame'], 'Error: Cannot render annotations for non keyframes!'

# Build a mapping from name to index to look up index in O(1) time.
nuim_name2idx_mapping = dict()
# The 0 index is reserved for non-labelled background; thus, the categories should start from index 1.
# Also, sort the categories before looping so that the order is always the same (alphabetical).
i = 1
for c in sorted(self.category, key=lambda k: k['name']):
# Ignore the vehicle.ego and flat.driveable_surface classes first; they will be mapped later.
if c['name'] != 'vehicle.ego' and c['name'] != 'flat.driveable_surface':
nuim_name2idx_mapping[c['name']] = i
i += 1

assert max(nuim_name2idx_mapping.values()) < 24, \
'Error: There are {} classes (excluding vehicle.ego and flat.driveable_surface), ' \
'but there should be 23. Please check your category.json'.format(max(nuim_name2idx_mapping.values()))

# Now map the vehicle.ego and flat.driveable_surface classes.
nuim_name2idx_mapping['flat.driveable_surface'] = 24
nuim_name2idx_mapping['vehicle.ego'] = 31

# Ensure that each class name is uniquely paired with a class index, and vice versa.
assert len(nuim_name2idx_mapping) == len(set(nuim_name2idx_mapping.values())), \
'Error: There are {} class names but {} class indices'.format(len(nuim_name2idx_mapping),
len(set(nuim_name2idx_mapping.values())))
name_to_index = name_to_index_mapping(self.category)

# Get image data.
self.check_sweeps(sample_data['filename'])
@@ -601,10 +579,11 @@ def get_segmentation(self,
mask = mask_decode(ann['mask'])

# Draw mask for semantic segmentation.
semseg_mask[mask == 1] = nuim_name2idx_mapping[category_name]
semseg_mask[mask == 1] = name_to_index[category_name]

# Load object instances.
object_anns = [o for o in self.object_ann if o['sample_data_token'] == sd_token]

# Sort by token to ensure that objects always appear in the instance mask in the same order.
object_anns = sorted(object_anns, key=lambda k: k['token'])

@@ -619,7 +598,7 @@ def get_segmentation(self,
mask = mask_decode(ann['mask'])

# Draw masks for semantic segmentation and instance segmentation.
semseg_mask[mask == 1] = nuim_name2idx_mapping[category_name]
semseg_mask[mask == 1] = name_to_index[category_name]
instanceseg_mask[mask == 1] = i

# Ensure that the number of instances in the instance segmentation mask is the same as the number of objects.
@@ -683,6 +662,7 @@ def render_image(self,
font = get_font(font_size=font_size)
else:
font = None
im = im.convert('RGBA')
draw = ImageDraw.Draw(im, 'RGBA')

annotations_types = ['all', 'surfaces', 'objects', 'none']
3 changes: 3 additions & 0 deletions python-sdk/nuimages/scripts/render_images.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# nuScenes dev-kit.
# Code written by Holger Caesar, 2020.

import argparse
import gc
import os
3 changes: 3 additions & 0 deletions python-sdk/nuimages/scripts/render_rare_classes.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# nuScenes dev-kit.
# Code written by Holger Caesar, 2020.

import argparse
import random
from collections import defaultdict
46 changes: 46 additions & 0 deletions python-sdk/nuimages/tests/assert_download.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# nuScenes dev-kit.
# Code written by Holger Caesar, 2020.

import argparse
import os

from tqdm import tqdm

from nuimages import NuImages


def verify_setup(nuim: NuImages):
"""
Script to verify that the nuImages installation is complete.
Note that this may take several minutes or hours.
"""

# Check that each sample_data file exists.
print('Checking that sample_data files are complete...')
for sd in tqdm(nuim.sample_data):
file_path = os.path.join(nuim.dataroot, sd['filename'])
assert os.path.exists(file_path), 'Error: Missing sample_data at: %s' % file_path


if __name__ == "__main__":

# Settings.
parser = argparse.ArgumentParser(description='Test that the installed dataset is complete.',
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--dataroot', type=str, default='/data/sets/nuimages',
help='Default nuImages data directory.')
parser.add_argument('--version', type=str, default='v1.0-train',
help='Which version of the nuImages dataset to evaluate on, e.g. v1.0-train.')
parser.add_argument('--verbose', type=int, default=1,
help='Whether to print to stdout.')

args = parser.parse_args()
dataroot = args.dataroot
version = args.version
verbose = bool(args.verbose)

# Init.
nuim_ = NuImages(version=version, verbose=verbose, dataroot=dataroot)

# Verify data blobs.
verify_setup(nuim_)
3 changes: 3 additions & 0 deletions python-sdk/nuimages/tests/test_attributes.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# nuScenes dev-kit.
# Code written by Holger Caesar, 2020.

import os
import unittest
from typing import Any
3 changes: 3 additions & 0 deletions python-sdk/nuimages/tests/test_foreign_keys.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# nuScenes dev-kit.
# Code written by Holger Caesar, 2020.

import itertools
import os
import unittest
26 changes: 26 additions & 0 deletions python-sdk/nuimages/utils/test_nuimages.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# nuScenes dev-kit.
# Code written by Holger Caesar, 2020.

import os
import unittest

from nuimages import NuImages


class TestNuImages(unittest.TestCase):

def test_load(self):
"""
Loads up NuImages.
This is intended to simply run the NuImages class to check for import errors, typos, etc.
"""

assert 'NUIMAGES' in os.environ, 'Set NUIMAGES env. variable to enable tests.'
nuim = NuImages(version='v1.0-mini', dataroot=os.environ['NUIMAGES'], verbose=False)

# Trivial assert statement
self.assertEqual(nuim.table_root, os.path.join(os.environ['NUIMAGES'], 'v1.0-mini'))


if __name__ == '__main__':
unittest.main()
35 changes: 34 additions & 1 deletion python-sdk/nuimages/utils/utils.py
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@

import base64
import os
from typing import List
from typing import List, Dict
import warnings

import matplotlib.font_manager
@@ -71,3 +71,36 @@ def get_font(fonts_valid: List[str] = None, font_size: int = 15) -> ImageFont:
warnings.warn('No suitable fonts were found in your system. '
'A default font will be used instead (the font size will not be adjustable).')
return ImageFont.load_default()


def name_to_index_mapping(category: List[dict]) -> Dict[str, int]:
"""
Build a mapping from name to index to look up index in O(1) time.
:param category: The nuImages category table.
:return:
"""
# The 0 index is reserved for non-labelled background; thus, the categories should start from index 1.
# Also, sort the categories before looping so that the order is always the same (alphabetical).
name_to_index = dict()
i = 1
sorted_category: List = sorted(category.copy(), key=lambda k: k['name'])
for c in sorted_category:
# Ignore the vehicle.ego and flat.driveable_surface classes first; they will be mapped later.
if c['name'] != 'vehicle.ego' and c['name'] != 'flat.driveable_surface':
name_to_index[c['name']] = i
i += 1

assert max(name_to_index.values()) < 24, \
holger-motional marked this conversation as resolved.
Show resolved Hide resolved
'Error: There are {} classes (excluding vehicle.ego and flat.driveable_surface), ' \
'but there should be 23. Please check your category.json'.format(max(name_to_index.values()))

# Now map the vehicle.ego and flat.driveable_surface classes.
name_to_index['flat.driveable_surface'] = 24
name_to_index['vehicle.ego'] = 31

# Ensure that each class name is uniquely paired with a class index, and vice versa.
assert len(name_to_index) == len(set(name_to_index.values())), \
'Error: There are {} class names but {} class indices'.format(len(name_to_index),
len(set(name_to_index.values())))

return name_to_index
4 changes: 2 additions & 2 deletions python-sdk/nuscenes/eval/detection/README.md
Original file line number Diff line number Diff line change
@@ -47,7 +47,7 @@ Note that the [evaluation server](http://evalai.cloudcv.org/web/challenges/chall

## Submission rules
### Detection-specific rules
* The maximum time window of past sensor data and ego poses that may be used at inference time is approximately 0.5s (at most 6 camera images, 6 radar sweeps and 10 lidar sweeps). At training time there are no restrictions.
* The maximum time window of past sensor data and ego poses that may be used at inference time is approximately 0.5s (at most 6 *past* camera images, 6 *past* radar sweeps and 10 *past* lidar sweeps). At training time there are no restrictions.

### General rules
* We release annotations for the train and val set, but not for the test set.
@@ -112,7 +112,7 @@ Some of these only have a handful of samples.
Hence we merge similar classes and remove rare classes.
This results in 10 classes for the detection challenge.
Below we show the table of detection classes and their counterparts in the nuScenes dataset.
For more information on the classes and their frequencies, see [this page](https://www.nuscenes.org/data-annotation).
For more information on the classes and their frequencies, see [this page](https://www.nuscenes.org/nuscenes#data-annotation).

| nuScenes detection class| nuScenes general class |
| --- | --- |
2 changes: 1 addition & 1 deletion python-sdk/nuscenes/eval/detection/tests/test_evaluate.py
Original file line number Diff line number Diff line change
@@ -74,7 +74,7 @@ def random_attr(name: str) -> str:
if nusc.get('scene', sample['scene_token'])['name'] in splits[split]:
val_samples.append(sample)

for sample in tqdm(val_samples):
for sample in tqdm(val_samples, leave=False):
sample_res = []
for ann_token in sample['anns']:
ann = nusc.get('sample_annotation', ann_token)
Loading