Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: FeastExtrasDependencyImportError when using SparkOfflineStore without S3 #4594

Conversation

aloysius-lim
Copy link
Contributor

What this PR does / why we need it:

When using SparkOfflineStore without S3, the following error is raised:

❯ python -c "from feast.infra.offline_stores.contrib.spark_offline_store.spark import SparkOfflineStore"

Traceback (most recent call last):
  File "[REDACTED]/python3.10/site-packages/feast/infra/utils/aws_utils.py", line 28, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "[REDACTED]/python3.10/site-packages/feast/infra/offline_stores/contrib/spark_offline_store/spark.py", line 34, in <module>
    from feast.infra.utils import aws_utils
  File "[REDACTED]/python3.10/site-packages/feast/infra/utils/aws_utils.py", line 34, in <module>
    raise FeastExtrasDependencyImportError("aws", str(e))
feast.errors.FeastExtrasDependencyImportError: No module named 'boto3'
You may need run pip install 'feast[aws]'

Installing the AWS libraries should not be required when not using AWS, to avoid unnecessary bloat. Since Spark is commonly used in many different environments (e.g. GCS, Azure or on premise), this change allows SparkOfflineStore to be used without installing the AWS libraries.

Signed-off-by: Aloysius Lim <aloysius.lim@cuezen.com>
Copy link
Member

@franciscojavierarceo franciscojavierarceo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Signed-off-by: Aloysius Lim <aloysius.lim@cuezen.com>
@franciscojavierarceo franciscojavierarceo merged commit 1ba94f7 into feast-dev:master Oct 2, 2024
18 checks passed
@aloysius-lim aloysius-lim deleted the do-not-require-aws-for-spark branch October 2, 2024 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants