camera

real_robot.utils.camera.depth2xyz(depth_image: ndarray, intrinsics: ndarray, depth_scale: int | float | None = None) → ndarray

Use camera intrinsics to convert depth_image to xyz_image

Parameters:

• depth_image – [H, W] or [H, W, 1] np.uint16/np.floating np.ndarray For np.uint16, depth_scale is assumed to be 1000 (in millimeters). For np.floating, depth_scale is assumed to be 1 (in meters).

• intrinsics – [3, 3] camera intrinsics matrix or [fx, fy, cx, cy]

• depth_scale – depth scale units, in meters. If None, choose based on dtype.

Return xyz_image:

[H, W, 3] np.floating np.ndarray

real_robot.utils.camera.register_depth(depth_unaligned: ndarray, k_depth: ndarray, k_color: ndarray, T_color_depth: ndarray, color_im_size: tuple[int, int], dist_color: ndarray | None = None, depth_dilation: bool = False) → ndarray

Register depth to color frame (a.k.a., align_depth_to_color) This will align depth image to color frame (intrinsics, extrinsics, and resolution)

Parameters:

• depth_unaligned – unaligned depth image. [H, W] or [H, W, 1] np.uint16/np.floating np.ndarray

• k_depth – depth camera intrinsic matrix, [3, 3] np.floating np.ndarray

• k_color – color camera intrinsic matrix, [3, 3] np.floating np.ndarray

• T_color_depth – extrinsics from depth camera to color camera, following OpenCV frame convention. [4, 4] np.floating np.ndarray

• color_im_size – color image size, (width, height)

• dist_color – color camera distortion coefficients, [5,] np.floating np.ndarray

• depth_dilation – whether to dilate depth to avoid holes and occlusion errors.

Return depth:

aligned depth image, [H, W] np.uint16/np.floating np.ndarray

real_robot.utils.camera.resize_obs_image(rgb_image, depth_image, intr_params: tuple, new_size, interpolation=6)

Resize rgb/depth images into shape=(width, height) :param rgb_image: [H, W, 3] np.uint8 np.ndarray :param depth_image: [H, W] np.uint16 np.ndarray :param intr_params: (fx, fy, cx, cy) tuple :param new_size: (width, height) tuple

real_robot.utils.camera.transform_points(pts: ndarray, H: ndarray) → ndarray

Transform points by 4x4 transformation matrix H :return out: same shape as pts

real_robot.utils.camera.transform_points_batch(pts: ndarray, H: ndarray) → ndarray

Transform points by Bx4x4 transformation matrix H

[3,], [4, 4] => [3,] [P, 3], [4, 4] => [P, 3] [H, W, 3], [4, 4] => [H, W, 3] [N, H, W, 3], [4, 4] => [N, H, W, 3] [P, 3], [B, 4, 4] => [B, P, 3] [B, P, 3], [B, 4, 4] => [B, P, 3] [H, W, 3], [B, 4, 4] => [B, H, W, 3] # (H != B) [B, H, W, 3], [B, 4, 4] => [B, H, W, 3] [N, H, W, 3], [B, 4, 4] => [B, N, H, W, 3] # (N != B) [B, N, H, W, 3], [B, 4, 4] => [B, N, H, W, 3] [B, N, 3], [B, N, 4, 4] => [B, N, 3] [B, N, P, 3], [B, N, 4, 4] => [B, N, P, 3] [B, N, H, W, 3], [B, N, 4, 4] => [B, N, H, W, 3]