don't install tests and other cruft in pip wheels

[RussTedrake]

This line was supposed to prevent that from happening. But after running pip install manipulation in a clean venv, I see

(venv) russt@TRI-179850-06:/tmp$ tree venv/lib/python3.10/site-packages/manipulation/
venv/lib/python3.10/site-packages/manipulation/
├── clutter.py
├── conf.py
├── create_sdf_from_mesh.py
├── directives_tree.py
├── drake_gym.py
├── envs
│   ├── box_flipup.py
│   ├── planar_gripper_pushing_a_box.py
│   └── __pycache__
│       ├── box_flipup.cpython-310.pyc
│       └── planar_gripper_pushing_a_box.cpython-310.pyc
├── exercises
│   ├── clutter
│   │   ├── normal_solution.npy
│   │   ├── __pycache__
│   │   │   ├── test_analytic_grasp.cpython-310.pyc
│   │   │   ├── test_grasp_candidate.cpython-310.pyc
│   │   │   └── test_normal.cpython-310.pyc
│   │   ├── test_analytic_grasp.py
│   │   ├── test_grasp_candidate.py
│   │   └── test_normal.py
│   ├── deep_perception
│   │   ├── __pycache__
│   │   │   └── test_contrastive.cpython-310.pyc
│   │   └── test_contrastive.py
│   ├── force
│   │   ├── __pycache__
│   │   │   └── test_hybrid.cpython-310.pyc
│   │   └── test_hybrid.py
│   ├── grader.py
│   ├── mobile
│   │   ├── __pycache__
│   │   │   └── test_mobile_base_ik.cpython-310.pyc
│   │   └── test_mobile_base_ik.py
│   ├── pick
│   │   ├── plot_planar_manipulator.py
│   │   ├── __pycache__
│   │   │   ├── plot_planar_manipulator.cpython-310.pyc
│   │   │   ├── test_differential_ik.cpython-310.pyc
│   │   │   ├── test_planar_manipulator.cpython-310.pyc
│   │   │   ├── test_rigid_transforms.cpython-310.pyc
│   │   │   ├── test_robot_painter.cpython-310.pyc
│   │   │   └── test_simple_qp.cpython-310.pyc
│   │   ├── test_differential_ik.py
│   │   ├── test_planar_manipulator.py
│   │   ├── test_rigid_transforms.py
│   │   ├── test_robot_painter.py
│   │   └── test_simple_qp.py
│   ├── pose
│   │   ├── __pycache__
│   │   │   ├── test_icp.cpython-310.pyc
│   │   │   ├── test_pose_estimation.cpython-310.pyc
│   │   │   └── test_ransac.cpython-310.pyc
│   │   ├── test_icp.py
│   │   ├── test_pose_estimation.py
│   │   └── test_ransac.py
│   ├── __pycache__
│   │   └── grader.cpython-310.pyc
│   ├── rl
│   │   ├── __pycache__
│   │   │   ├── test_stochastic_optimization.cpython-310.pyc
│   │   │   └── test_vpg.cpython-310.pyc
│   │   ├── test_stochastic_optimization.py
│   │   └── test_vpg.py
│   ├── robot
│   │   ├── __pycache__
│   │   │   ├── test_direct_joint_control.cpython-310.pyc
│   │   │   ├── test_hardware_station_io.cpython-310.pyc
│   │   │   ├── test_reflected_inertia.cpython-310.pyc
│   │   │   └── test_survey.cpython-310.pyc
│   │   ├── test_direct_joint_control.py
│   │   ├── test_hardware_station_io.py
│   │   ├── test_reflected_inertia.py
│   │   └── test_survey.py
│   ├── segmentation
│   │   ├── __pycache__
│   │   │   ├── test_mask.cpython-310.pyc
│   │   │   └── test_segmentation_and_grasp.cpython-310.pyc
│   │   ├── test_mask.py
│   │   └── test_segmentation_and_grasp.py
│   └── trajectories
│       ├── __pycache__
│       │   ├── test_door_opening.cpython-310.pyc
│       │   ├── test_rrt_planning.cpython-310.pyc
│       │   └── test_taskspace_iris.cpython-310.pyc
│       ├── rrt_planner
│       │   ├── geometry.py
│       │   ├── iiwa_rrt_problem.py
│       │   ├── __init__.py
│       │   ├── __pycache__
│       │   │   ├── geometry.cpython-310.pyc
│       │   │   ├── iiwa_rrt_problem.cpython-310.pyc
│       │   │   ├── __init__.cpython-310.pyc
│       │   │   ├── robot.cpython-310.pyc
│       │   │   └── rrt_planning.cpython-310.pyc
│       │   ├── robot.py
│       │   └── rrt_planning.py
│       ├── test_door_opening.py
│       ├── test_rrt_planning.py
│       └── test_taskspace_iris.py
├── icp.py
├── index.md
├── __init__.py
├── make_drake_compatible_model.md
├── make_drake_compatible_model.py
├── meshcat_cpp_utils.py
├── meshcat_utils.md
├── meshcat_utils.py
├── models
│   ├── 061_foam_brick_w_visual_contact_spheres.sdf
│   ├── book.sdf
│   ├── bunny
│   │   ├── bunny.npy
│   │   ├── bun_zipper_res2.ply
│   │   └── README
│   ├── camera_box.sdf
│   ├── clutter.dmd.yaml
│   ├── clutter_mustard.dmd.yaml
│   ├── clutter_planning.dmd.yaml
│   ├── clutter.scenarios.yaml
│   ├── clutter_w_cameras.dmd.yaml
│   ├── cupboard.scenario.yaml
│   ├── double_pendulum.urdf
│   ├── floating_joint.urdf
│   ├── floor.sdf
│   ├── hydro
│   │   ├── 003_cracker_box.sdf
│   │   ├── 004_sugar_box.sdf
│   │   ├── 005_tomato_soup_can.sdf
│   │   ├── 006_mustard_bottle.sdf
│   │   ├── 009_gelatin_box.sdf
│   │   ├── 010_potted_meat_can.sdf
│   │   ├── 061_foam_brick.sdf
│   │   ├── bin.sdf
│   │   ├── cupboard.sdf
│   │   ├── extra_heavy_duty_table_surface_only_collision.sdf
│   │   ├── README.md
│   │   ├── schunk_wsg_50_with_tip.sdf
│   │   └── ycb_mustard_bottle_mesh_pieces
│   │       ├── mesh_piece_000.obj
│   │       ├── mesh_piece_001.obj
│   │       ├── mesh_piece_002.obj
│   │       ├── mesh_piece_003.obj
│   │       ├── mesh_piece_004.obj
│   │       ├── mesh_piece_005.obj
│   │       ├── mesh_piece_006.obj
│   │       ├── mesh_piece_007.obj
│   │       └── mesh_piece_008.obj
│   ├── iiwa_and_wsg.dmd.yaml
│   ├── mecanum
│   │   ├── drake_obstacles.dmd.yaml
│   │   ├── ground.sdf
│   │   ├── mecanum_base.urdf
│   │   ├── obstacle_boxes.sdf
│   │   ├── obstacles.sdf
│   │   └── README.md
│   ├── mobile_iiwa14_primitive_collision.urdf
│   ├── mustard_w_cameras.dmd.yaml
│   ├── package.xml
│   ├── planar_bin.sdf
│   ├── planar_foam_brick_collision_as_visual.sdf
│   ├── planar_iiwa14_no_collision.urdf
│   ├── planar_joint.sdf
│   ├── planar_manipulation_station.scenario.yaml
│   ├── pr2_shelves.dmd.yaml
│   ├── rubiks_cube_2_by_2.sdf
│   ├── rubiks_cube.sdf
│   ├── schunk_wsg_50_welded_fingers.dmd.yaml
│   ├── schunk_wsg_50_welded_fingers.sdf
│   ├── schunk_wsg_50_welded_fingers_sphere.sdf
│   ├── segmentation_and_grasp_scene.dmd.yaml
│   ├── shelves.sdf
│   ├── shelves_w_ellipsoid_collision.sdf
│   ├── simple_2d_cspace.xml
│   ├── simple_2d_obstacle.obj
│   ├── spot
│   │   ├── __pycache__
│   │   │   └── read_spot_camera_intrinsics.cpython-310.pyc
│   │   ├── README.md
│   │   ├── read_spot_camera_intrinsics.py
│   │   ├── requirements.txt
│   │   ├── spot_with_arm_and_floating_base_actuators.scenario.yaml
│   │   ├── spot_with_arm_and_floating_base_actuators.urdf
│   │   └── spot_with_arm.urdf
│   ├── suction
│   │   ├── base.obj
│   │   ├── head.obj
│   │   ├── mid.obj
│   │   ├── README.md
│   │   ├── suction-base.urdf
│   │   ├── suction-head.urdf
│   │   └── tip.obj
│   ├── two_bins.dmd.yaml
│   ├── two_bins.sdf
│   ├── two_bins_w_cameras.dmd.yaml
│   ├── two_bins_w_cameras.sdf
│   └── two_link_iiwa14.urdf
├── mustard_depth_camera_example.py
├── pick.py
├── __pycache__
│   ├── clutter.cpython-310.pyc
│   ├── conf.cpython-310.pyc
│   ├── create_sdf_from_mesh.cpython-310.pyc
│   ├── directives_tree.cpython-310.pyc
│   ├── drake_gym.cpython-310.pyc
│   ├── icp.cpython-310.pyc
│   ├── __init__.cpython-310.pyc
│   ├── make_drake_compatible_model.cpython-310.pyc
│   ├── meshcat_cpp_utils.cpython-310.pyc
│   ├── meshcat_utils.cpython-310.pyc
│   ├── mustard_depth_camera_example.cpython-310.pyc
│   ├── pick.cpython-310.pyc
│   ├── scenarios.cpython-310.pyc
│   ├── station.cpython-310.pyc
│   ├── systems.cpython-310.pyc
│   └── utils.cpython-310.pyc
├── scenarios.py
├── _static
│   └── custom.css
├── station.md
├── station.py
├── systems.md
├── systems.py
├── test
│   ├── __pycache__
│   │   ├── test_create_sdf_from_mesh.cpython-310.pyc
│   │   ├── test_directives_tree.cpython-310.pyc
│   │   ├── test_gym.cpython-310.pyc
│   │   ├── test_hardware_station_interface.cpython-310.pyc
│   │   ├── test_meshcat_utils.cpython-310.pyc
│   │   ├── test_model_directives.cpython-310.pyc
│   │   ├── test_station_camera_ids.cpython-310.pyc
│   │   └── test_utils.cpython-310.pyc
│   ├── test_create_sdf_from_mesh.py
│   ├── test_directives_tree.py
│   ├── test_gym.py
│   ├── test_hardware_station_interface.py
│   ├── test_meshcat_utils.py
│   ├── test_model_directives.py
│   ├── test_station_camera_ids.py
│   └── test_utils.py
├── utils.md
└── utils.py

cc @nepfaff

[nepfaff]

I looked at this before and some seem to be of the opinion that one should distribute tests with the pip wheel (see here for example).

This way your module will be distributed with its tests and users can use them to verify that it works with their set up.

Otherwise, it might be cleaner to move the test directory outside of the manipulation package to make it clear that it isn't part of the package. But we are in a bit of a weird situations with the exercise tests being mixed in different folders so this wouldn't solve everything.

[nepfaff]

The docs are very short here but I'm not sure about asterix support for exclude. Moreover include doesn't seem to enable wheels by default.

Edit: Asterixes should work as here and here.

[RussTedrake]

some seem to be of the opinion that one should distribute tests with the pip wheel

that's interesting. but what if those test need, for instance, a pile of model files and assets in order to power them? i guess we can make those small enough.

But we are in a bit of a weird situations with the exercise tests being mixed in different folders so this wouldn't solve everything.

Right. We currently do need to ship the grader tests for exercises in the pip wheels -- that's essential to our current exercise deployment workflow.

At the very least, we should absolutely get rid of the pycache which is not getting properly excluded.

[RussTedrake]

At the very least, we should absolutely get rid of the pycache which is not getting properly excluded.

maybe i'm wrong about pycache

poetry build && unzip -l dist/manipulation-2025.1.4-py3-none-any.whl

doesn't have the pycache. maybe they're just getting created faster than I expected on install

[RussTedrake]

Ok, given the fact that I can't get poetry to respect the exclude, and the fact that some people say it's good practice to include the tests, I'll go ahead and call this WON'T FIX. I'll push a small fix that removes the spurious _static/custom.css that snuck in there.