rsl-rl "Isaac-Lift-Cube-Franka-v0" franka only reach but doesn't lift

[zoctipus]

###############################################################################
                      Learning iteration 607/1500                       

                       Computation: 234626 steps/s (collection: 0.368s, learning 0.051s)
               Value function loss: 0.0056
                    Surrogate loss: 0.0017
             Mean action noise std: 0.31
                       Mean reward: -4.96
               Mean episode length: 248.09
    Episode Reward/reaching_object: 0.7675
     Episode Reward/lifting_object: 0.0000
Episode Reward/object_goal_tracking: 0.0000
Episode Reward/object_goal_tracking_fine_grained: 0.0000
        Episode Reward/action_rate: -0.2314
          Episode Reward/joint_vel: -1.5321
Metrics/object_pose/position_error: 0.3674
Metrics/object_pose/orientation_error: 2.6296
      Episode Termination/time_out: 15.3750
Episode Termination/object_dropping: 0.5833
--------------------------------------------------------------------------------
                   Total timesteps: 59768832
                    Iteration time: 0.42s
                        Total time: 242.68s
                               ETA: 356.4s

As you can see from the training script, franka lift reward is CONSTANTLY nearly zero
Episode Reward/lifting_object: 0.0000,

If disable the headless mode and inspect franka's learning progresss,
I will notice that franka first would try to close its finger randomly when it didn't learn how to reach to cube.
but once the cube is reached, it learned it needs to keep the fingers open, but it failed to learn that it needs to close the clow after cube is reached.

I will try to make it work in my best effort.
It will be nice if the demo are all working!

[zoctipus]

def grip_object(
    env: RLTaskEnv, object_cfg: SceneEntityCfg = SceneEntityCfg("object") , ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame")
)-> torch.Tensor:
    """Reward the agent for closing the gripper when object is within finger"""
    object: RigidObject = env.scene[object_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    # Target object position: (num_envs, 3)
    cube_pos_w = object.data.root_pos_w
    # End-effector position: (num_envs, 3)
    ee_w = ee_frame.data.target_pos_w[..., 0, :]
    # Distance of the end-effector to the object: (num_envs,)
    object_ee_distance = torch.norm(cube_pos_w - ee_w, dim=1)


    #add this line on top of existing object_ee_distance
    reward = torch.where((object_ee_distance < 0.02) & (env.scene['robot'].data.joint_pos_target[:,8] == 0), 1, -1)
    return reward

I added this reward to franka lift mdp and the training is now successful. Though I don't quite like this reward, I'd love to know if there's a better way to shape this reward.

[samibouziri]

I tried that environment but with IK relative input, and worked for me is to put more weight the lift (30/50) and to train for 15K episodes. The lift generally happens around 8K episodes. I also managed to lift within 1.5K episodes with a lift weight of 100 but I was not satisfied with the final motion of the arm.