[BEAM-13314]Revise recommendations to manage Python pipeline dependencies.

website/www/site/content/en/documentation/runtime/environments.md

@@ -42,11 +42,11 @@ For optimal user experience, we also recommend you use the latest released versi
 
 ### Building and pushing custom containers
 
-Beam [SDK container images](https://hub.docker.com/search?q=apache%2Fbeam&type=image) are built from Dockerfiles checked into the [Github](https://github.com/apache/beam) repository and published to Docker Hub for every release. You can build customized containers in one of two ways:
+Beam [SDK container images](https://hub.docker.com/search?q=apache%2Fbeam&type=image) are built from Dockerfiles checked into the [Github](https://github.com/apache/beam) repository and published to Docker Hub for every release. You can build customized containers in one of three ways:
 
 1. **[Writing a new](#writing-new-dockerfiles) Dockerfile based on a released container image**. This is sufficient for simple additions to the image, such as adding artifacts or environment variables.
 2. **[Modifying](#modifying-dockerfiles) a source Dockerfile in [Beam](https://github.com/apache/beam)**. This method requires building from Beam source but allows for greater customization of the container (including replacement of artifacts or base OS/language versions).
-
+3. **[Modifying](#modify-existing-base-image) an existing container image to make it compatible with Apache Beam Runners**. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners.
 #### Writing a new Dockerfile based on an existing published container image {#writing-new-dockerfiles}
 
 1. Create a new Dockerfile that designates a base image using the [FROM instruction](https://docs.docker.com/engine/reference/builder/#from).
@@ -171,6 +171,48 @@ creates a Java 8 SDK image with appropriate licenses in `/opt/apache/beam/third_
 
 By default, no licenses/notices are added to the docker images.
 
+#### Modifying an existing container image to make it compatible with Apache Beam Runners {#modify-existing-base-image}
+Beam offers a way to provide your own custom container image. The easiest way to build a new custom image that is compatible with Apache Beam Runners is to use a [multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) process. This copies over the necessary artifacts from a default Apache Beam base image to build your custom container image.
+
+1. Copy necessary artifacts from Apache Beam base image to your image.
+  ```
+  # This can be any container image,
+ FROM python:3.7-bullseye
+
+ # Install SDK. (needed for Python SDK)
+ RUN pip install --no-cache-dir apache-beam[gcp]==2.35.0
+
+ # Copy files from official SDK image, including script/dependencies.
+ COPY --from=apache/beam_python3.7_sdk:2.35.0 /opt/apache/beam /opt/apache/beam
+
+ # Perform any additional customizations if desired
+
+ # Set the entrypoint to Apache Beam SDK launcher.
+ ENTRYPOINT ["/opt/apache/beam/boot"]
+
+  ```
+>**NOTE**: This example assumes necessary dependencies (in this case, Python 3.7 and pip) have been installed on the existing base image. Installing the Apache Beam SDK into the image will ensure that the image has the necessary SDK dependencies and reduce the worker startup time.
+>The version specified in the `RUN` instruction must match the version used to launch the pipeline.<br>
+>**Users need to make sure that whatever base image they use has the same Python/Java interpreter version that they used to run the pipeline**.
+
+
+2. [Build](https://docs.docker.com/engine/reference/commandline/build/) and [push](https://docs.docker.com/engine/reference/commandline/push/) the image using Docker.
+  ```
+    export BASE_IMAGE="apache/beam_python3.7_sdk:2.25.0"
+    export IMAGE_NAME="myremoterepo/mybeamsdk"
+    export TAG="latest"
+
+    # Optional - pull the base image into your local Docker daemon to ensure
+    # you have the most up-to-date version of the base image locally.
+    docker pull "${BASE_IMAGE}"
+
+    docker build -f Dockerfile -t "${IMAGE_NAME}:${TAG}" .
+ ```
+
+3. If your runner is running remotely, retag the image and [push](https://docs.docker.com/engine/reference/commandline/push/) the image to your repository.
+  ```
+  docker push "${IMAGE_NAME}:${TAG}"
+  ```
 
 ## Running pipelines with custom container images {#running-pipelines}
 

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

@@ -45,6 +45,16 @@ If your pipeline uses public packages from the [Python Package Index](https://py
     The runner will use the `requirements.txt` file to install your additional dependencies onto the remote workers.
 
 **Important:** Remote workers will install all packages listed in the `requirements.txt` file. Because of this, it's very important that you delete non-PyPI packages from the `requirements.txt` file, as stated in step 2. If you don't remove non-PyPI packages, the remote workers will fail when attempting to install packages from sources that are unknown to them.
+> **NOTE**: An alternative to `pip freeze` is to use a library like [pip-tools](https://github.com/jazzband/pip-tools) to compile the all the dependencies required for the pipeline from a `--requirements_file`, where only top-level dependencies are mentioned.
+## Custom Containers {#custom-containers}
+
+You can pass a [container](https://hub.docker.com/search?q=apache%2Fbeam&type=image) image with all the dependencies that are needed for the pipeline instead of `requirements.txt`. [Follow the instructions on how to run pipeline with Custom Container images](https://beam.apache.org/documentation/runtime/environments/#running-pipelines).
+
+1. If you are using a custom container image, we recommend that you install the dependencies from the `--requirements_file` directly into your image at build time. In this case, you do not need to pass `--requirements_file` option at runtime, which will reduce the pipeline startup time.
+
+       # Add these lines with the path to the requirements.txt to the Dockerfile
+       COPY <path to requirements.txt> /tmp/requirements.txt
+       RUN python -m pip install -r /tmp/requirements.txt
 
 
 ## Local or non-PyPI Dependencies {#local-or-nonpypi}
@@ -53,7 +63,7 @@ If your pipeline uses packages that are not available publicly (e.g. packages th
 
 1. Identify which packages are installed on your machine and are not public. Run the following command:
 
-        pip freeze
+      pip freeze
 
     This command lists all packages that are installed on your machine, regardless of where they were installed from.
 
@@ -123,3 +133,20 @@ If your pipeline uses non-Python packages (e.g. packages that require installati
         --setup_file /path/to/setup.py
 
 **Note:** Because custom commands execute after the dependencies for your workflow are installed (by `pip`), you should omit the PyPI package dependency from the pipeline's `requirements.txt` file and from the `install_requires` parameter in the `setuptools.setup()` call of your `setup.py` file.
+
+## Pre-building SDK container image
+
+In the pre-building step, we install pipeline dependencies on the container image prior to the job submission. This would speed up the pipeline execution.\
+To use pre-building the dependencies from `requirements.txt` on the container image. Follow the steps below.
+1. Provide the container engine. We support `local_docker` and `cloud_build`(requires a GCP project with Cloud Build API enabled).
+
+       --prebuild_sdk_container_engine <execution_environment>
+2. To pass a base image for pre-building dependencies, enable this flag. If not, apache beam's base image would be used.
+
+       --sdk_container_image <location_to_base_image>
+3. To push the container image, pre-built locally with `local_docker` , to a remote repository(eg: docker registry), provide URL to the remote registry by passing
+
+       --docker_registry_push_url <IMAGE_URL>
+> To use Docker, the `--sdk_container_image` should be compatible with Apache Beam Runner. Please follow the [instructions](https://beam.apache.org/documentation/runtime/environments/#building-and-pushing-custom-containers) on how to build a base container image compatible with Apache Beam.
+
+**NOTE**: This feature is available only for the `DataflowRunner`.