Skip to content

Commit

Permalink
Merge lidarseg_v1.0 into nuscenes_v2.0 (#434)
Browse files Browse the repository at this point in the history
* nuScenes-lidarseg (#343)

* initial commit for NuScenes-lidarseg

* add function to list lidarseg categories and create colormap

* integrated lidarseg function into nuscenes

* print number of lidarseg annotations

* Render lidarseg labels in image

* Improve error message when pointsensor is not lidar

* Calculate stats for lidarseg

* Assertion for nsweeps to display lidarseg labels

* Render only points in image which are labeled with classes the user chooses

* Modularize function to filter colormap

* Modularize function for generating colors for lidarseg classes

* Clean up example usages

* Allow user to filter and show desired lidarseg labels for render_sample_data

* Save figure from render_pointcloud_in_image without excessive border

* Render scene with pointclouds and lidarseg labels

* Shift conditional statements in render_sample_data()

* Check that lidar pointcloud is a keyframe

* Render all camera channels with pointclouds and lidarseg labels for all scenes

* Allow user to show lidarseg labels for render_sample()

* Hacks to render for VOs

* Added scene to filename for easy retrieval for VOs

* Render scenes for VOs per class

* Updated rendering of scenes for cameras to output videos

* Function to convert plt scatter plot to cv2

* Tidy up examples

* Print stats for lidarseg keyframe

* Enable sorting of counts

* Tweak verbosity for render_sample

* Include class index when printing sample stats

* Assert that path for video given by user ends with .avi

* Enable showing of legend when mapping pc to image

* Style edits

* nuScenes-lidarseg initial tutorial

* Cleanup, move methods out of tutorial, copyright

* Cleanup

* Remove empty cell

* Throw errors rather than warnings

* Address some comments on style.

* Add full stops to comments

* Tidy up utils

* Removed TODOs

* Initial draft of lidarseg tutorial

* Added gifs for tutorial

* Tidy up tutorial

* Add setup instructions to tutorial

* Edit style for one assertion

* Changed show_lidarseg_labels default value in render_sample

* Change type to np.ndarray

* Fix some typos

* Remove version from lidarseg.json

* Bugfix

* Unbugfix

* Don't add category_lidarseg to table_names

* Comment out dangerous commands

Co-authored-by: Holger Caesar <holger@nutonomy.com>

* View predictions using lidarseg devkit (#396)

* Allow user to view preds using render_pointcloud_in_image

* Allow users to get stats of predictions for a sample

* Add visualization for user's preds to relevant functions

* Updated tutorial

* Fixed some typos in tutorial

* Comment out code which may crash notebook

* Fix double #

* Get lidarseg file name from json instead, default coloring to depth if empty json

* Change show_lidarseg_preds to lidarseg_preds_bin_path

* Amend doc string for lidarseg_preds_bin_path

* Amend render video functions to take folder name for preds

* Amend tutorial

* Remove gifs from folder

* Clear outputs from tutorial

* Update documentations for lidarseg (#404)

* Update docs for lidarseg

* Fix some typos

* Address comments for docs

* Update folder structure description in notebook

* Automatically determine which lidarseg classes are present in a pointcloud projected onto an image (#410)

* Auto find liarseg labels present in projected pcl

* Add assertion to check num of bin files equals num of lidarseg records

* Address comments

* Adjust legend for aesthetics

* Improve aesthetics for render_scene_with_pointclouds_for_all_cameras (#412)

* Flip back cams horizontally for aesthetics

* Explicitly set margins to zero and turn axes off

* Prevent final frames from showing up in notebook if user stops render

* Allow render_scene_with_pointclouds_for_all_cameras to output frames as images

* Update indices in tutorial examples

* Style changes

* Clear memory in render_camera_channel_with_pointclouds if user stops rendering

* Graceful exit if users stops rendering

* Change random seed for color scheme

* Style edits

* Enable bboxes to be plotted with lidarseg (#418)

* Include rendering of lidarseg with bboxes for render_scene_with_pointclouds_for_all_cameras

* Add option to render bboxes in lidarseg videos

* Style edits

* Rename methods

* Renamed argument to be more similar to nuScenes

* Rename show_lidarseg_labels to show_lidarseg

* Added comment on videos and images

* Rewording

* Making plotting of bboxes with lidarseg more modular

* Remove option to save renders as images for render_scene_lidarseg

* Use render_mode instead of render_if_no_points

* Style edits

* Update notebook

* Style edits for notebook

* Remove render_if_no_points for render_pointcloud_in_image

* Style edits

* Update file names for renders

* Save image even for verbose=False

* Made plot_points_and_bboxes

* Update coloring

* Update colormap

* Choose nicer colors for some classes

* Deep copy for colormap

* Reduce bbox line width and update colormap

* One line per arg

* Update colormap

* Remove unused method for arbitrary colormap

* Remove leading 0 for filename

* One colormap to rule them all

* Update error msg for checking colormap to colors conversion

* Add dpi as an arg for rendering lidarseg scenes

* Amend docstring in get_colormap()

* Make colormap DRY

Co-authored-by: Holger Caesar <holger@nutonomy.com>

* Remove render_if_no_points argument from lidarseg tutorial (#423)

* Remove render_if_no_points arg and section

* Rephrasing

* Single category.json for both nuScenes and nuScenes-lidarseg (#424)

* Change 'label' to 'name' when loading category json; check if preds folder exists

* Check in various functions that  lidarseg is installed

* Shorten assertion statement

* Check that new version of category.json is used, if loading lidarseg

* Check only for 'index' in category records

* nutonomy green for driveable surface (#425)

* Improve lidarseg rendering methods, update tutorial, add unit tests (#429)

* Initial commit to address comments

* Fix bug where passing np.array instead of array into filter_lidarseg_labels throws an error

* Rephrase some parts of tutorial

* Include legend for render_sample_data

* Add arg for show_lidarseg_legend in render_sample_data

* Adjust aesthetics of legend in render_sample_data

* Update colormap

* Make creating legend modular

* Ensure only labels in pc is included in legend

* Explain verbose

* Check that filter_lidarseg_labels is either a list or np.array

* Allow user to list stats by class index

* Update docstrings

* Update tutorial to demo sort_by

* Add unit tests

* Make painting of labels modular

* Add docstrings

* Add lidarseg annotation instructions, update lidarseg tutorial intro (#431)

* Initial commit

* Add links from classes to examples

* Link from examples to class definitions using Top

* Remove extra white line

* Include script to render lidarseg histogram for split (#433)

* Initial commit

* Arrange order of seaborn in requirements.txt

* Add docstrings

* Add more docstrings

* Remove seaborn from requirements

* Address comments

* Add horizontal gridlines

* Replicate look and feel of seaborn without needing package

* Style edits

* Add type in truncate_class_name

Co-authored-by: Holger Caesar <holger@nutonomy.com>
  • Loading branch information
whyekit-motional and holger-motional authored Jul 13, 2020
1 parent 274725a commit db6805f
Show file tree
Hide file tree
Showing 9 changed files with 1,988 additions and 78 deletions.
13 changes: 11 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ In February 2020 we published the CAN bus expansion.
It contains low-level vehicle data about the vehicle route, IMU, pose, steering angle feedback, battery, brakes, gear position, signals, wheel speeds, throttle, torque, solar sensors, odometry and more.
To install this expansion, please follow these steps:
- Download the expansion from the [Download page](https://www.nuscenes.org/download),
- Move the can_bus folder to your nuScenes root directory (e.g. `/data/sets/nuscenes/can_bus`).
- Extract the can_bus folder to your nuScenes root directory (e.g. `/data/sets/nuscenes/can_bus`).
- Get the latest version of the nuscenes-devkit.
- If you already have a previous version of the devkit, update the pip requirements (see [details](https://github.com/nutonomy/nuscenes-devkit/blob/master/setup/installation.md)): `pip install -r setup/requirements.txt`
- Get started with the [CAN bus readme](https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/can_bus/README.md) or [tutorial](https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/tutorials/can_bus_tutorial.ipynb).
Expand All @@ -64,11 +64,20 @@ To install this expansion, please follow these steps:
In July 2019 we published a map expansion with 11 semantic layers (crosswalk, sidewalk, traffic lights, stop lines, lanes, etc.).
To install this expansion, please follow these steps:
- Download the expansion from the [Download page](https://www.nuscenes.org/download),
- Move the .json files to your nuScenes `maps` folder.
- Extract the .json files to your nuScenes `maps` folder.
- Get the latest version of the nuscenes-devkit.
- If you already have a previous version of the devkit, update the pip requirements (see [details](https://github.com/nutonomy/nuscenes-devkit/blob/master/setup/installation.md)): `pip install -r setup/requirements.txt`
- Get started with the [map expansion tutorial](https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/tutorials/map_expansion_tutorial.ipynb).

## nuScenes-lidarseg
In June 2020 we published nuScenes-lidarseg which contains the semantic labels of the point clouds for the approximately 40,000 keyframes in nuScenes.
To install nuScenes-lidarseg, please follow these steps:
- Download the dataset from the [Download page](https://www.nuscenes.org/download),
- Extract the `lidarseg` and `v1.0-*` folders to your root directory (e.g. `/data/sets/nuscenes/lidarseg`, `/data/sets/nuscenes/v1.0-*`).
- Get the latest version of the nuscenes-devkit.
- If you already have a previous version of the devkit, update the pip requirements (see [details](https://github.com/nutonomy/nuscenes-devkit/blob/master/setup/installation.md)): `pip install -r setup/requirements.txt`
- Get started with the [tutorial](https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/python-sdk/tutorials/nuscenes_lidarseg_tutorial.ipynb).

## Devkit setup
The devkit is tested for Python 3.6 and Python 3.7.
To install Python, please check [here](https://github.com/nutonomy/nuscenes-devkit/blob/master/setup/installation.md#install-python).
Expand Down
149 changes: 149 additions & 0 deletions docs/instructions_lidarseg.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# nuScenes-lidarseg Annotator Instructions

# Overview
- [Introduction](#introduction)
- [General Instructions](#general-instructions)
- [Detailed Instructions](#detailed-instructions)
- [Classes](#classes)

# Introduction
In nuScenes-lidarseg, we annotate every point in the lidar pointcloud with a semantic label.
All the labels from nuScenes are carried over into nuScenes-lidarseg; in addition, more ["stuff" (background) classes](#classes) have been included.
Thus, nuScenes-lidarseg contains both foreground classes (pedestrians, vehicles, cyclists, etc.) and background classes (driveable surface, nature, buildings, etc.).


# General Instructions
- Label each point with a class.
- Use the camera images to facilitate, check and validate the labels.
- Each point belongs to only one class, i.e., one class per point.


# Detailed Instructions
+ **Extremities** such as vehicle doors, car mirrors and human limbs should be assigned the same label as the object.
Note that in contrast to the nuScenes 3d cuboids, the lidarseg labels include car mirrors and antennas.
+ **Minimum number of points**
+ An object can have as little as **one** point.
In such cases, that point should only be labeled if it is certain that the point belongs to a class
(with additional verification by looking at the corresponding camera frame).
Otherwise, the point should be labeled as `static.other`.
+ **Other static object vs noise.**
+ **Other static object:** Points that belong to some physical object, but are not defined in our taxonomy.
+ **Noise:** Points that do not correspond to physical objects or surfaces in the environment
(e.g. noise, reflections, dust, fog, raindrops or smoke).
+ **Terrain vs other flat.**
+ **Terrain:** Grass, all kinds of horizontal vegetation, soil or sand. These areas are not meant to be driven on.
This label includes a possibly delimiting curb.
Single grass stalks do not need to be annotated and get the label of the region they are growing on.
+ Short bushes / grass with **heights of less than 20cm**, should be labeled as terrain.
Similarly, tall bushes / grass which are higher than 20cm should be labeled as vegetation.
+ **Other flat:** Horizontal surfaces which cannot be classified as ground plane / sidewalk / terrain, e.g., water.
+ **Terrain vs sidewalk**
+ **Terrain:** See above.
+ **Sidewalk:** A sidewalk is a walkway designed for pedestrians and / or cyclists. Sidewalks are always paved.


# Classes
The following classes are in **addition** to the existing ones in nuScenes:

| Label ID | Label | Short Description |
| --- | --- | --- |
| 0 | [`noise`](#1-noise-class-0) | Any lidar return that does not correspond to a physical object, such as dust, vapor, noise, fog, raindrops, smoke and reflections. |
| 24 | [`flat.driveable_surface`](#2-flatdriveable_surface-class-24) | All paved or unpaved surfaces that a car can drive on with no concern of traffic rules. |
| 25 | [`flat.sidewalk`](#3-flatsidewalk-class-25) | Sidewalk, pedestrian walkways, bike paths, etc. Part of the ground designated for pedestrians or cyclists. Sidewalks do **not** have to be next to a road. |
| 26 | [`flat.terrain`](#4-flatterrain-class-26) | Natural horizontal surfaces such as ground level horizontal vegetation (< 20 cm tall), grass, rolling hills, soil, sand and gravel. |
| 27 | [`flat.other`](#5-flatother-class-27) | All other forms of horizontal ground-level structures that do not belong to any of driveable_surface, curb, sidewalk and terrain. Includes elevated parts of traffic islands, delimiters, rail tracks, stairs with at most 3 steps and larger bodies of water (lakes, rivers). |
| 28 | [`static.manmade`](#6-staticmanmade-class-28) | Includes man-made structures but not limited to: buildings, walls, guard rails, fences, poles, drainages, hydrants, flags, banners, street signs, electric circuit boxes, traffic lights, parking meters and stairs with more than 3 steps. |
| 29 | [`static.vegetation`](#7-staticvegetation-class-29) | Any vegetation in the frame that is higher than the ground, including bushes, plants, potted plants, trees, etc. Only tall grass (> 20cm) is part of this, ground level grass is part of `flat.terrain`.|
| 30 | [`static.other`](#8-staticother-class-30) | Points in the background that are not distinguishable. Or objects that do not match any of the above labels. |
| 31 | [`vehicle.ego`](#9-vehicleego-class-31) | The vehicle on which the cameras, radar and lidar are mounted, that is sometimes visible at the bottom of the image. |

## Examples of classes
Below are examples of the classes added in nuScenes-lidarseg.
For simplicity, we only show lidar points which are relevant to the class being discussed.


### 1. noise (class 0)
![noise_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/0_scene-0053_CAM_FRONT_LEFT_1532402428104844_crop.jpg)
![noise_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/0_scene-0163_CAM_FRONT_LEFT_1526915289904917_crop.jpg)
![noise_3](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/0_scene-0207_CAM_BACK_LEFT_1532621922197405_crop.jpg)
![noise_4](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/0_scene-0635_CAM_FRONT_1537296086862404_crop.jpg)

[Top](#classes)


### 2. flat.driveable_surface (class 24)
![driveable_surface_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/24_206_CAM_BACK.jpg)
![driveable_surface_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/24_250_CAM_FRONT.jpg)
![driveable_surface_3](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/24_9750_CAM_FRONT.jpg)
![driveable_surface_4](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/24_10000_CAM_BACK.jpg)

[Top](#classes)


### 3. flat.sidewalk (class 25)
![sidewalk_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/25_90_CAM_FRONT_LEFT.jpg)
![sidewalk_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/25_13250_CAM_FRONT_LEFT.jpg)
![sidewalk_3](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/25_280_CAM_FRONT_LEFT.jpg)
![sidewalk_4](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/25_680_CAM_FRONT_LEFT.jpg)

[Top](#classes)


### 4. flat.terrain (class 26)
![terrain_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/26_11750_CAM_BACK_RIGHT.jpg)
![terrain_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/26_10700_CAM_BACK_LEFT.jpg)
![terrain_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/26_886_CAM_BACK_LEFT.jpg)
![terrain_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/26_1260_CAM_BACK_LEFT.jpg)

[Top](#classes)


### 5. flat.other (class 27)
![flat_other_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/27_2318_CAM_FRONT.jpg)
![flat_other_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/27_3750_CAM_FRONT_RIGHT.jpg)
![flat_other_3](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/27_1230_CAM_FRONT.jpg)
![flat_other_4](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/27_1380_CAM_FRONT.jpg)

[Top](#classes)


### 6. static.manmade (class 28)
![manmade_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/28_13850_CAM_FRONT.jpg)
![manmade_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/28_15550_CAM_FRONT.jpg)
![manmade_3](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/28_5009_CAM_FRONT.jpg)
![manmade_4](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/28_5501_CAM_BACK.jpg)

[Top](#classes)


### 7. static.vegetation (class 29)
![vegetation_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/29_650_CAM_FRONT_LEFT.jpg)
![vegetation_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/29_3650_CAM_FRONT.jpg)
![vegetation_3](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/29_5610_CAM_BACK_RIGHT.jpg)
![vegetation_4](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/29_5960_CAM_FRONT_RIGHT.jpg)

[Top](#classes)


### 8. static.other (class 30)
![static_other_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/30_scene-0031_CAM_BACK_LEFT_1531886230947423.jpg)
![static_other_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/30_scene-0032_CAM_BACK_RIGHT_1531886262027893.jpg)
![static_other_3](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/30_scene-0160_CAM_BACK_LEFT_1533115303947423.jpg)
![static_other_4](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/30_scene-0166_CAM_BACK_RIGHT_1526915380527813.jpg)

[Top](#classes)


### 9. vehicle.ego (class 31)
Points on the ego vehicle generally arise due to self-occlusion, in which some lidar beams hit the ego vehicle.
When the pointcloud is projected into a chosen camera image, the devkit removes points which are less than
1m in front of the camera to prevent such points from cluttering the image. Thus, users will not see points
belonging to `vehicle.ego` projected onto the camera images when using the devkit. To give examples, of the
`vehicle.ego` class, the bird's eye view (BEV) is used instead:

![ego_1](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/31_479_BEV.jpg)
![ego_2](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/31_11200_BEV.jpg)
![ego_3](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/31_14500_BEV.jpg)
![ego_4](https://www.nuscenes.org/public/images/taxonomy_imgs/lidarseg/31_24230_BEV.jpg)

[Top](#classes)
Empty file.
Loading

0 comments on commit db6805f

Please sign in to comment.