predict lane class

[Yutong-gannis]

how can I predict lane class of the openlane

[qinjian623]

Hi,

Each grid in the BEV can correspond to a category, allowing for segmented category outputs. Alternatively, voting or aggregating the categories for entire laneline is possible.

Another approach involves extracting features of all grids associated with a specific lane line and utilizing language models or temporal models for prediction.

Thanks.

[Yutong-gannis]

@qinjian623 How can I integrate category infomation in bev gt in openlane_data.py

bev_lane_det/loader/bev_road/openlane_data.py

Lines 136 to 224 in 2d66872

	def get_seg_offset(self, idx, smooth=False):
	gt_path = os.path.join(self.gt_paths, self.cnt_list[idx][0], self.cnt_list[idx][1])
	image_path = os.path.join(self.image_paths, self.cnt_list[idx][0], self.cnt_list[idx][1].replace('json', 'jpg'))
	image = cv2.imread(image_path)
	image_h, image_w, _ = image.shape
	with open(gt_path, 'r') as f:
	gt = json.load(f)
	cam_w_extrinsics = np.array(gt['extrinsic'])
	maxtrix_camera2camera_w = np.array([[0, 0, 1, 0],
	[-1, 0, 0, 0],
	[0, -1, 0, 0],
	[0, 0, 0, 1]], dtype=float)
	cam_extrinsics = cam_w_extrinsics @ maxtrix_camera2camera_w #
	R_vg = np.array([[0, 1, 0],
	[-1, 0, 0],
	[0, 0, 1]], dtype=float)
	R_gc = np.array([[1, 0, 0],
	[0, 0, 1],
	[0, -1, 0]], dtype=float)
	cam_extrinsics_persformer = copy.deepcopy(cam_w_extrinsics)
	cam_extrinsics_persformer[:3, :3] = np.matmul(np.matmul(
	np.matmul(np.linalg.inv(R_vg), cam_extrinsics_persformer[:3, :3]),
	R_vg), R_gc)
	cam_extrinsics_persformer[0:2, 3] = 0.0
	matrix_lane2persformer = cam_extrinsics_persformer @ np.linalg.inv(maxtrix_camera2camera_w)
	cam_intrinsic = np.array(gt['intrinsic'])
	lanes = gt['lane_lines']
	matrix_IPM2ego = IPM2ego_matrix(
	ipm_center=(int(self.x_range[1] / self.meter_per_pixel), int(self.y_range[1] / self.meter_per_pixel)),
	m_per_pixel=self.meter_per_pixel) #
	image_gt = np.zeros((image_h, image_w), dtype=np.uint8)
	res_points_d = {}
	for idx in range(len(lanes)):
	lane1 = lanes[idx]
	lane_camera_w = np.array(lane1['xyz']).T[np.array(lane1['visibility']) == 1.0].T
	lane_camera_w = np.vstack((lane_camera_w, np.ones((1, lane_camera_w.shape[1]))))
	lane_ego_persformer = matrix_lane2persformer @ lane_camera_w #
	lane_ego_persformer[0], lane_ego_persformer[1] = lane_ego_persformer[1], -1 * lane_ego_persformer[0]
	lane_ego = cam_w_extrinsics @ lane_camera_w #
	''' plot uv '''
	uv1 = ego2image(lane_ego[:3], cam_intrinsic, cam_extrinsics)
	cv2.polylines(image_gt, [uv1[0:2, :].T.astype(np.int)], False, idx + 1, self.lane2d_thick)
	distance = np.sqrt((lane_ego_persformer[1][0] - lane_ego_persformer[1][-1]) ** 2 + (
	lane_ego_persformer[0][0] - lane_ego_persformer[0][-1]) ** 2)
	if distance < self.lane_length_threshold:
	continue
	y = lane_ego_persformer[1]
	x = lane_ego_persformer[0]
	z = lane_ego_persformer[2]
	if smooth:
	if len(x) < 2:
	continue
	elif len(x) == 2:
	curve = np.polyfit(x, y, 1)
	function2 = interp1d(x, z, kind='linear')
	elif len(x) == 3:
	curve = np.polyfit(x, y, 2)
	function2 = interp1d(x, z, kind='quadratic')
	else:
	curve = np.polyfit(x, y, 3)
	function2 = interp1d(x, z, kind='cubic')
	x_base = np.linspace(min(x), max(x), 20)
	y_pred = np.poly1d(curve)(x_base)
	ego_points = np.array([x_base, y_pred])
	z = function2(x_base)
	else:
	ego_points = np.array([x, y])
	ipm_points = np.linalg.inv(matrix_IPM2ego[:, :2]) @ (ego_points[:2] - matrix_IPM2ego[:, 2].reshape(2, 1))
	ipm_points_ = np.zeros_like(ipm_points)
	ipm_points_[0] = ipm_points[1]
	ipm_points_[1] = ipm_points[0]
	res_points = np.concatenate([ipm_points_, np.array([z])], axis=0)
	res_points_d[idx + 1] = res_points
	ipm_gt, offset_y_map, z_map = self.get_y_offset_and_z(res_points_d)
	''' virtual camera '''
	if self.use_virtual_camera:
	sc = Standard_camera(self.vc_intrinsic, self.vc_extrinsics, self.vc_image_shape,
	cam_intrinsic, cam_extrinsics, (image.shape[0], image.shape[1]))
	trans_matrix = sc.get_matrix(height=0)
	image = cv2.warpPerspective(image, trans_matrix, self.vc_image_shape)
	image_gt = cv2.warpPerspective(image_gt, trans_matrix, self.vc_image_shape)
	return image, image_gt, ipm_gt, offset_y_map, z_map, cam_extrinsics, cam_intrinsic

[qinjian623]

Hi,

You can return your category ground truth (gt) within the get_item function, but maintaining consistency with the spatial relationship of the BEV.

Please note that in our data generation process, the lane lines have undergone smoothing through fitting. It is important to ensure that your category ground truth remains consistent with this.

The remaining task is simply training the segmentation model by adding an additional head in the BEV space.

Based on our experience, background region does not need to calculate the loss, meaning that all background grids should be ignored. This will significantly improve the model's performance metrics.

Good luck.

[EnternalTwinkle]

@Yutong-gannis Can you share the source code? Thanks，1017094591@qq.com

[Angelfalling]

Hi,

You can return your category ground truth (gt) within the get_item function, but maintaining consistency with the spatial relationship of the BEV.

Please note that in our data generation process, the lane lines have undergone smoothing through fitting. It is important to ensure that your category ground truth remains consistent with this.

The remaining task is simply training the segmentation model by adding an additional head in the BEV space.

Based on our experience, background region does not need to calculate the loss, meaning that all background grids should be ignored. This will significantly improve the model's performance metrics.

Good luck.

Hello, very good classification suggestions, but I have some questions. Lane markings include solid lines and dashed lines, and there may also be multi-lane markings that can span multiple grid cells. In this situation, how can we use the grid for classification voting?
"Can I communicate with you via email?" this is my email: 923755330@qq.com

[qinjian623]

For the same lane line with different attributes, after clustering, you can segment and tally the category information.
Of course, this might require adding additional logic code.

For double lines,
they can be logically treated as a single lane line.

For lines that partially overlap,
this can be addressed by increasing the resolution of the BEV (Bird's Eye View).

We also recommend using the popular vit + mask segmentation approach, which is compatible with the current network backbone.

For more complex scenarios, you can use some of the latest heads designed for generating HD maps, such as map tr2.

Hi,
You can return your category ground truth (gt) within the get_item function, but maintaining consistency with the spatial relationship of the BEV.
Please note that in our data generation process, the lane lines have undergone smoothing through fitting. It is important to ensure that your category ground truth remains consistent with this.
The remaining task is simply training the segmentation model by adding an additional head in the BEV space.
Based on our experience, background region does not need to calculate the loss, meaning that all background grids should be ignored. This will significantly improve the model's performance metrics.
Good luck.

Hello, very good classification suggestions, but I have some questions. Lane markings include solid lines and dashed lines, and there may also be multi-lane markings that can span multiple grid cells. In this situation, how can we use the grid for classification voting? "Can I communicate with you via email?" this is my email: 923755330@qq.com