feat: Feature/upgrade 3.11 and remove bytewax #4046

lokeshrangineni · 2024-03-27T16:43:05Z

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

…pendencies. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

…hould fix the failure of python unit test cases. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

…-bytewax # Conflicts: # sdk/python/feast/infra/materialization/contrib/bytewax/bytewax_materialization_engine.py

lokeshrangineni · 2024-04-03T04:09:32Z

I am looking at the PR again to modify the the logic without actually removing the bytewax engine. If this approach works then probably i will update the PR otherwise we can go ahead with this approach.

…1-remove-bytewax # Conflicts: # .github/workflows/build_wheels.yml # .github/workflows/nightly-ci.yml # .github/workflows/unit_tests.yml # sdk/python/feast/infra/materialization/kubernetes/Dockerfile # sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_engine.py # sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_job.py # sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_task.py # sdk/python/feast/infra/materialization/kubernetes/main.py # setup.py

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> This reverts commit 4bec6b0.

…dev#4140) * chore: Move feast install to docker build in java it tests (feast-dev#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove commented out lines in compose file Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * make local compose mode default Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * limit COPY contents Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * include pyproject.toml in dockerfile Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change links to depends_on Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try updating setup-python to v5 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * pin macos image to macos-12 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * force rerun Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * chore: Incorporate release 0.37.1 to master (feast-dev#4127) * chore(release): release 0.37.1 ## [0.37.1](feast-dev/feast@v0.37.0...v0.37.1) (2024-04-17) ### Bug Fixes * Pgvector patch ([feast-dev#4108](feast-dev#4108)) ([1a1f0b1](feast-dev@1a1f0b1)) ### Reverts * Reverts "fix: Using version args to install the correct feast version" ([feast-dev#4112](feast-dev#4112)) ([d5ded69](feast-dev@d5ded69)), closes [feast-dev#3953](feast-dev#3953) * chore: Move feast install to docker build in java it tests (feast-dev#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove commented out lines in compose file Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * make local compose mode default Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * limit COPY contents Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * include pyproject.toml in dockerfile Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change links to depends_on Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try updating setup-python to v5 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * pin macos image to macos-12 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * force rerun Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co> Co-authored-by: Tornike Gurgenidze <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * chore: Install python dependencies with uv in workflows (feast-dev#4086) * install dependencies in unit-tests with uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * install dependencies in unit-tests with uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * enable caching, change linter job Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change local integration tests to uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change all installs to uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try adding uv cache Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * fix lambda cache step name Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * reenable caches for uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove dangling cache step Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * feat: Make arrow primary interchange for online ODFV execution (feast-dev#4143) * rewrite online flow to use transform_arrow Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * fix transformation server Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * feat: Feast/IKV online store documentation (feast-dev#4146) * feat: Feast/IKV online store documentation Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * functionality matric Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * more changes Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * mount dir Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> --------- Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Default value is not set in Redis connection string using environment variable (feast-dev#4136) Removed documentation of Redis connection string supporting default values when using environment variables as it isn't supported Fixes feast-dev#3669 Signed-off-by: Theodor Mihalache <tmihalac@redhat.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Make sure schema is used when calling `get_table_query_string` method for Snowflake datasource (feast-dev#4131) * Fix get_table_query_string method for Snowflake datasource Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Add quotes to table string Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> --------- Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Change checkout action back to v3 from v5 which isn't released yet (feast-dev#4147) Signed-off-by: Jeremy Ary <jary@redhat.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * changed the code the way mysql container is initialized. Trying to fix the issue - feast-dev#4128 Also going to check if this change will be resolved in the github actions as well. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> Signed-off-by: Theodor Mihalache <tmihalac@redhat.com> Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> Signed-off-by: Jeremy Ary <jary@redhat.com> Co-authored-by: Tornike Gurgenidze <togurg14@freeuni.edu.ge> Co-authored-by: Jeremy Ary <jary@redhat.com> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co> Co-authored-by: Pushkar Gupta <pushkar.moi@gmail.com> Co-authored-by: Theodor Mihalache <84387487+tmihalac@users.noreply.github.com> Co-authored-by: Tom Steenbergen <41334387+TomSteenbergen@users.noreply.github.com>

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> This reverts commit 8d77135.

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

jeremyary · 2024-04-26T12:29:07Z

.github/workflows/unit_tests.yml

-        python-version: [ "3.9", "3.10" ]
-        os: [ ubuntu-latest, macos-12 ]
+        python-version: [ "3.9", "3.10", "3.11"]
+        os: [ ubuntu-latest, macOS-12 ]


SUPER nitpicky of me, but casing here is fine in lowercase which is how GitHub references them too

jeremyary · 2024-04-26T12:29:41Z

sdk/python/requirements/py3.9-requirements.txt

@@ -144,10 +144,12 @@ six==1.16.0
    # via python-dateutil
 sniffio==1.3.1
    # via anyio
-sqlalchemy[mypy]==2.0.29
+sqlalchemy[mypy]==1.4.52


not saying it's wrong, more curious, can you please share what's causing the downgrade of sqlalchemy?

I have just re-ran the command to regenerate this file but not sure why this was modified. I have not modified the file manually. I will pay attention to it.

jeremyary · 2024-04-26T12:31:40Z

setup.py

@@ -65,15 +65,15 @@
    "fastapi>=0.68.0",
    "uvicorn[standard]>=0.14.0,<1",
    "gunicorn; platform_system != 'Windows'",
-    "dask[dataframe]>=2021.1.0",
+    "dask[dataframe]>=2024.4.2",


I feel like I saw an answer in slack somewhere that I can't find at the moment, but this is a pretty big version jump, was Dask having issues with new Python version?

We had a issue with Dask with new python version. Build is failing so had to upgrade but i have not tried the immediate version so let me give a shot with the version released in 2022.

you're good! I was just looking for reason - since we need to upgrade we might as well catch up

jeremyary · 2024-04-26T12:33:08Z

setup.py

    "bowler",  # Needed for automatic repo upgrades
 ]

 GCP_REQUIRED = [
    "google-api-core>=1.23.0,<3",
    "googleapis-common-protos>=1.52.0,<2",
    "google-cloud-bigquery[pandas]>=2,<3.13.0",
-    "google-cloud-bigquery-storage >= 2.0.0,<3",
+    "google-cloud-bigquery-storage >= 2.23.0,<3",


Did we encounter an issue w/BigQuery dependency too?

Thank you for pointing it out. I had modified this version in between to resolve an issue with dask library. probably we don't need this change. i will remove this upgrade and see what happens.

jeremyary · 2024-04-26T13:17:29Z

@lokeshrangineni looking really good, appreciate the diligence, I know you've been sticking with this one for a while!

update doc Signed-off-by: cmuhao <sduxuhao@gmail.com>

…pendencies. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> This reverts commit 8d77135.

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

…e/upgrade-3.11-remove-bytewax

jeremyary · 2024-04-30T14:10:46Z

work continuing here due to conflict complexity

lokeshrangineni added 3 commits March 27, 2024 00:56

Upgrading the python version to 3.11 and also removing the bytewax de…

408f400

…pendencies. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Removing missing bytewax dependencies from the previous commit.

1214f71

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Merge branch 'feast-dev:master' into feature/upgrade-3.11-remove-bytewax

2dd2051

lokeshrangineni changed the title ~~Feature/upgrade 3.11 remove bytewax~~ Feature/upgrade 3.11 and remove bytewax Mar 27, 2024

lokeshrangineni added 4 commits March 27, 2024 13:51

Merge branch 'feast-dev:master' into feature/upgrade-3.11-remove-bytewax

228e3b4

Removing duplicate dependency ibis which was added by mistake. this s…

f0971a8

…hould fix the failure of python unit test cases. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Removing the bytewax materialization engine.

1cfa4d5

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Removing the bytewax documentation.

822cd67

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

lokeshrangineni changed the title ~~Feature/upgrade 3.11 and remove bytewax~~ feat: Feature/upgrade 3.11 and remove bytewax Apr 1, 2024

lokeshrangineni added 3 commits April 1, 2024 15:51

Merge branch 'feast-dev:master' into feature/upgrade-3.11-remove-bytewax

43ef03b

Adding missed duckdb dependencies.

10ddd01

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Merge branch 'feature/origin-master' into feature/upgrade-3.11-remove…

68da3f0

…-bytewax # Conflicts: # sdk/python/feast/infra/materialization/contrib/bytewax/bytewax_materialization_engine.py

lokeshrangineni marked this pull request as ready for review April 2, 2024 17:12

lokeshrangineni requested a review from whoahbot as a code owner April 2, 2024 17:12

jeremyary added the ok-to-test label Apr 2, 2024

lokeshrangineni changed the title ~~feat: Feature/upgrade 3.11 and remove bytewax~~ HOLD - feat: Feature/upgrade 3.11 and remove bytewax Apr 3, 2024

This was referenced Apr 9, 2024

feat: Kubernetes materialization engine written based on bytewax #4087

Merged

chore: Omit bytewax deps and materialization engine #4098

Merged

jeremyary changed the title ~~HOLD - feat: Feature/upgrade 3.11 and remove bytewax~~ feat: Feature/upgrade 3.11 and remove bytewax Apr 23, 2024

lokeshrangineni and others added 10 commits April 25, 2024 11:51

updating the requirements.txt files as per the latest code.

b8fdfbd

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

updating the requirements.txt files as per the latest code.

4bec6b0

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Revert "updating the requirements.txt files as per the latest code."

fb6d0e5

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> This reverts commit 4bec6b0.

just checking if this resolves the issue with dask incompatibility.

d9d59af

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

updating 3.9 dependencies to make it work with 3.11 as well.

89ba1d8

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

updating 3.10 dependencies to make it work with 3.11 as well.

2bfedcf

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Revert "updating 3.10 dependencies to make it work with 3.11 as well."

29585f2

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> This reverts commit 8d77135.

updating dask version so that it can work on python 3.11

bada729

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

lokeshrangineni force-pushed the feature/upgrade-3.11-remove-bytewax branch from 279b043 to bada729 Compare April 25, 2024 21:07

jeremyary reviewed Apr 26, 2024

View reviewed changes

HaoXuAI and others added 5 commits April 26, 2024 15:16

fix: Update doc (feast-dev#4153)

95355c4

update doc Signed-off-by: cmuhao <sduxuhao@gmail.com>

Upgrading the python version to 3.11 and also removing the bytewax de…

e7dddd3

…pendencies. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

updating 3.10 dependencies to make it work with 3.11 as well.

5d4c322

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

Revert "updating 3.10 dependencies to make it work with 3.11 as well."

e7b9c13

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> This reverts commit 8d77135.

updating dask version so that it can work on python 3.11

3d005b0

Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>

lokeshrangineni changed the base branch from master to woop-patch-2 April 29, 2024 16:08

lokeshrangineni requested review from sfc-gh-madkins, DvirDukhan, sudohainguyen, niklasvm, kevjumba, yelo-blood and a team as code owners April 29, 2024 16:08

lokeshrangineni changed the base branch from woop-patch-2 to master April 29, 2024 16:08

Merge remote-tracking branch 'refs/remotes/origin/master' into featur…

61d3319

…e/upgrade-3.11-remove-bytewax

jeremyary closed this Apr 30, 2024

lokeshrangineni deleted the feature/upgrade-3.11-remove-bytewax branch June 20, 2024 19:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Feature/upgrade 3.11 and remove bytewax #4046

feat: Feature/upgrade 3.11 and remove bytewax #4046

lokeshrangineni commented Mar 27, 2024

lokeshrangineni commented Apr 3, 2024

jeremyary Apr 26, 2024

jeremyary Apr 26, 2024

lokeshrangineni Apr 26, 2024 •

edited

Loading

jeremyary Apr 26, 2024

lokeshrangineni Apr 26, 2024 •

edited

Loading

jeremyary Apr 26, 2024

jeremyary Apr 26, 2024

lokeshrangineni Apr 26, 2024

jeremyary commented Apr 26, 2024

jeremyary commented Apr 30, 2024

feat: Feature/upgrade 3.11 and remove bytewax #4046

feat: Feature/upgrade 3.11 and remove bytewax #4046

Conversation

lokeshrangineni commented Mar 27, 2024

lokeshrangineni commented Apr 3, 2024

jeremyary Apr 26, 2024

Choose a reason for hiding this comment

jeremyary Apr 26, 2024

Choose a reason for hiding this comment

lokeshrangineni Apr 26, 2024 • edited Loading

Choose a reason for hiding this comment

jeremyary Apr 26, 2024

Choose a reason for hiding this comment

lokeshrangineni Apr 26, 2024 • edited Loading

Choose a reason for hiding this comment

jeremyary Apr 26, 2024

Choose a reason for hiding this comment

jeremyary Apr 26, 2024

Choose a reason for hiding this comment

lokeshrangineni Apr 26, 2024

Choose a reason for hiding this comment

jeremyary commented Apr 26, 2024

jeremyary commented Apr 30, 2024

lokeshrangineni Apr 26, 2024 •

edited

Loading

lokeshrangineni Apr 26, 2024 •

edited

Loading