Remove PIP_NO_DEPS from YamlTemplate Dockerfile

[Polber]

PIP_NO_DEPS is typically added to custom python templates to speedup launch, but since the YamlTemplate image already has all dependencies, it won't affect startup of native Beam YAML pipelines.

It does, however, prevent transitive dependencies from being installed for packages defined by a pythonPackage provider.

This PR removes the PIP_NO_DEPS to allow for proper compilation/expansion of custom pythonPackage provider transforms.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 42.57%. Comparing base (983c151) to head (d78bd3e).
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1748      +/-   ##
============================================
+ Coverage     42.15%   42.57%   +0.41%     
- Complexity     3278     3359      +81     
============================================
  Files           808      815       +7     
  Lines         47293    47672     +379     
  Branches       5053     5107      +54     
============================================
+ Hits          19938    20295     +357     
- Misses        25710    25719       +9     
- Partials       1645     1658      +13     

Components	Coverage Δ
spanner-templates	64.52% <ø> (+0.89%)	⬆️
spanner-import-export	64.32% <ø> (-0.13%)	⬇️
spanner-live-forward-migration	75.00% <ø> (ø)
spanner-live-reverse-replication	51.96% <ø> (ø)
spanner-bulk-migration	83.45% <ø> (+1.01%)	⬆️

see 18 files with indirect coverage changes

[damccorm]

Is there a reason we have this anywhere? I see we now only recommend this for Beam <=2.37.0, so should we remove it everywhere?

cc/ @tvalentyn who may have more context - https://github.com/GoogleCloudPlatform/python-docs-samples/pull/11277/files - I'm also concerned that we may no longer recommend this because we're already doing something similar, but I'm definitely missing context

[tvalentyn]

We don't need this, we already pass no-deps in https://github.com/apache/beam/blob/44792814ff303cd561a1fb3d0ff6cb1b4b4c3135/sdks/python/apache_beam/runners/portability/stager.py#L756

Context is that when beam tries to stage pipeline dependencies specified in a requirements.txt, if such a dependency uses pep-517 to package itself (e.g. with a toml file), then in order find its transitive dependencies, pip rebuilds a wheel, which may be slow and can fail. Hence we decided to only stage top-level dependencies, and we now pass no-deps in stager.py.

[Polber]

Only other place it is defined is the XLANG template, so I can also remove it from that one

[damccorm]

Sounds good. I think this a good change to make, so I'll approve. I doubt this will actually solve the problem it is targeting though given this information

[Polber]

@damccorm I tested a Yaml job that has a provider specified with dependencies which fails without this change. Tested that it works with this change.

The link @tvalentyn linked is for the worker container image. The problem here is during expansion of provided transforms in the template launcher. In the case of pythonPackage, a venv is cloned from the base venv, with extra packages defined in the provider spec being installed to the cloned venv. When PIP_NO_DEPS is defined, the transitive deps are not installed, meaning these custom transform environments do not contain all necessary packages.

An example (using first random package not in beam I could find):

pipeline:
  type: chain
  transforms:
    - type: Create
      config:
        elements:
          - id: 1
    - type: Calendar
    - type: LogForTesting

providers:
  - type: pythonPackage
    config:
      packages:
        - calendar-view
    transforms:
      Calendar: |
        from calendar_view.calendar import Calendar
        from calendar_view.core.event import EventStyles
        
        @beam.ptransform_fn
        def fn(pcoll):
          calendar = Calendar.build()
          calendar.add_event(day_of_week=0, start='08:00', end='17:00', style=EventStyles.GRAY)
          calendar.save("simple_view.png")
        
          return pcoll | beam.Map(lambda x: x)

In this example (which can be run locally), if PIP_NO_DEPS is passed, calendar-view is installed to the cloned venv during expansion, but the pipeline will fail with message ModuleNotFoundError: No module named 'PIL' since the Pillow transitive dep is not installed. Removing the PIP_NO_DEPS flag results in success