feat: Instrument Feast using Prometheus and OpenTelemetry #4300

tsisodia10 · 2024-06-20T23:42:58Z

What this PR does / why we need it:

This draft pull request introduces monitoring functionality for the Feast feature store(python server). It instruments the feature store using Prometheus and OpenTelemetry to capture system-level metrics.

tchughesiv

it would be a good idea to make this optional by adding something like a --metrics flag to the CLI and a similar param to the helm chart as well. imo, this should not be started / configured unless the user chooses to enable it.

infra/charts/feast-feature-server/templates/instrumentation.yml

infra/charts/feast-feature-server/templates/otel-collector-ls.yml

infra/charts/feast-feature-server/templates/otel-collector.yml

infra/charts/feast-feature-server/templates/pod-monitor.yml

infra/charts/feast-feature-server/templates/rbac.yml

infra/charts/feast-feature-server/templates/deployment.yaml

infra/charts/feast-feature-server/templates/instrumentation.yml

sdk/python/feast/feature_store.py

redhatHameed · 2024-06-21T21:10:03Z

sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile

@@ -0,0 +1,34 @@
+FROM python:3.11-slim


why needed new Dockerfile why not using existing one.

+1 , do we need a separate image?

I am working on merging this Dockerfile with the existing ones, the downside I am facing is that the new image build is not picking the updated code with the existing Dockerfiles.

tsisodia10 · 2024-06-24T14:12:22Z

it would be a good idea to make this optional by adding something like a --metrics flag to the CLI and a similar param to the helm chart as well. imo, this should not be started / configured unless the user chooses to enable it.

@tchughesiv makes sense, I will modify it accordingly.

sdk/python/feast/feature_store.py

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

tsisodia10 · 2024-06-26T14:51:23Z

it would be a good idea to make this optional by adding something like a --metrics flag to the CLI and a similar param to the helm chart as well. imo, this should not be started / configured unless the user chooses to enable it.

Done, PTAL!

tchughesiv

getting there...
now, a similar bool metrics param should be added to the feast-feature-server helm chart. this param should be used to enable all metrics related configs when set. it should default to false. be sure to run make build-helm-docs afterward.

sdk/python/feast/cli.py

tchughesiv · 2024-06-26T15:00:25Z

infra/charts/feast-feature-server/templates/deployment.yaml

@@ -52,7 +57,8 @@ spec:
            - "serve"
            - "-h"
            - "0.0.0.0"
-            {{- end }}
+            {{- end }} 
+          command: ["feast", "serve", "-h", "0.0.0.0"]


this should likely turn into something like -

{{- if .Values.metrics }} command: ["feast", "serve", "--metrics", "-h", "0.0.0.0"] {{- else }} command: ["feast", "serve", "-h", "0.0.0.0"]

Done, please take a look!

@tchughesiv I think we would not need the check here, the --metric flag is working perfectly fine without it being added in the deployment. The user can pass the bool value during helm deployment.

@tsisodia10 if the --metrics flag is NOT specified, prom and telemetry should NOT be enabled. Metrics should only enabled if the flag is used. Based on this understanding, we would need this helm check. Is this not currently how the cli flag is functioning?

i don't see it currently being passed anywhere in the helm chart's deployment.yaml... how it is currently being enabled?

@dmartinol related thread

That's strange to have the metrics flag work even if it's not present in the deployment manifest. On enabling it during feast deployment, the monitoring manifests gets deployed and vice versa.
Will add the check in the deployment file then.

infra/charts/feast-feature-server/templates/instrumentation.yml

infra/charts/feast-feature-server/templates/otel-collector-ls.yml

infra/charts/feast-feature-server/templates/otel-collector.yml

infra/charts/feast-feature-server/values.yaml

tchughesiv · 2024-06-26T15:04:38Z

sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile

@@ -0,0 +1,34 @@
+FROM python:3.11-slim


+1 , do we need a separate image?

tokoko · 2024-06-26T18:42:39Z

@tsisodia10 prometheus-client should be added to setup.py explicitly under an extra, I think (maybe metrics or prometheus). (Right now it's part of our ci dependencies only under ge extra "thanks to" a whole mess of unnecessary dependencies great-expectations library likes to drag along wherever it goes)

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

sdk/python/feast/cli.py

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

infra/charts/feast-feature-server/values.yaml

sdk/python/feast/cli.py

infra/charts/feast-feature-server/templates/service.yaml

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

tchughesiv

@tsisodia10 it might be a good idea to add a couple tests... maybe in sdk/python/tests/integration/online_store? We should make sure that the metrics components are only starting when the --metrics flag is present.

tokoko · 2024-07-16T12:41:14Z

@tsisodia10 Can you take care of a DCO check? I think the easiest way is to rebase all commits and force push a single signed commit (git rebase HEAD~49 --signoff and git push --force-with-lease origin inst-stable).

Also please run make lint-python and make format-python before pushing the changes.

tchughesiv · 2024-07-16T14:15:52Z

infra/charts/feast-feature-server/templates/deployment.yaml

+            # - "--metrics"
+            # - "-h"
+            # - "0.0.0.0"
+            # {{- else }}


shouldn't this block of configs be uncommented and enabled only when metrics are configured in the chart?

these chart changes should all be fully vetted in a cluster to ensure it continues to work as expected with or without metrics configured.

sdk/python/feast/infra/feature_servers/multicloud/Dockerfile

tchughesiv · 2024-07-16T14:43:59Z

infra/charts/feast-feature-server/templates/deployment.yaml

@@ -14,6 +14,7 @@ spec:
    {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
+        # instrumentation.opentelemetry.io/inject-python: "true"


does this need to stay? a check added?

I think good to have for reference rather than lost in the README. If the user wants to enable monitoring, he just needs to uncomment this statement.

for sure! i was thinking it made more sense though to incorporate it into the metrics param chart checks. e.g.

{{- if .Values.metrics }}

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

remove providers Signed-off-by: tokoko <togurg14@freeuni.edu.ge>

…v#4303) * Makefile: Formatting Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Makefile: Exclude Snowflake tests for postgres offline store tests Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Bootstrap: Use conninfo Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Tests: Make connection string compatible with psycopg3 Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Tests: Test connection type pool and singleton Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Global: Replace conn.set_session() calls to be psycopg3 compatible Set connection read only Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Offline: Use psycopg3 Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Online: Use psycopg3 Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Online: Restructure online_write_batch Addition Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Online: Use correct placeholder Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Online: Handle bytes properly in online_read() Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Online: Whitespace Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Online: Open ConnectionPool Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Online: Add typehint Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Utils: Use psycopg3 Use new ConnectionPool Pass kwargs as named argument Use executemany over execute_values Remove not-required open argument in psycopg.connect Improve Use SpooledTemporaryFile Use max_size and add docstring Properly write with StringIO Utils: Use SpooledTemporaryFile over StringIO object Add replace Fix df_to_postgres_table Remove import Utils Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Lint: Raise exceptions if cursor returned no columns or rows Add log statement Lint: Fix _to_arrow_internal Lint: Fix _get_entity_df_event_timestamp_range Update exception Use ZeroColumnQueryResult Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Add comment on +psycopg string Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Docs: Remove mention of psycopg2 Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Lint: Fix Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Default to postgresql+psycopg and log warning Update warning Fix Format warning Add typehints Use better variable name Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> * Solve merge conflicts Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com> --------- Signed-off-by: Job Almekinders <job.almekinders@teampicnic.com>

* move get_online_features to OnlineStore interface Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * fix pydantic warnings Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * run ruff format Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge>

adding try and except block for import Co-authored-by: Francisco Javier Arceo <franciscojavierarceo@users.noreply.github.com>

Signed-off-by: Shuchu Han <shuchu.han@gmail.com>

Signed-off-by: Francisco Javier Arceo <franciscojavierarceo@users.noreply.github.com> Co-authored-by: Francisco Javier Arceo <franciscojavierarceo@users.noreply.github.com>

…batch and fix progress bar (feast-dev#4331) * Remove batching and fix tqdm progress bar Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Comment Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Remove test changes Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Update comment Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> --------- Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com>

…v#4327) * Add async retrieval for postgres Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Format Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Update _prepare_keys method Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Fix typo Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> --------- Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com>

* Add sql registry async refresh Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * make refresh code a daemon thread Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * Change RegistryConfig to cacheMode Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * Only run async when ttl > 0 Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * make refresh async run in a loop Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * make refresh async run in a loop Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * Reorder async refresh call Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * Add documentation Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * Update test_universal_registry.py Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * Force rerun of tests Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * Force rerun of tests Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> * Format repo config file Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> --------- Signed-off-by: Stanley Opara <a-sopara@expediagroup.com> Co-authored-by: Stanley Opara <a-sopara@expediagroup.com>

* Update maintainers.md * Update maintainers.md * Update maintainers.md

upgrade ibis version Signed-off-by: tokoko <togurg14@freeuni.edu.ge>

* update dask version to support pandas 1.x Signed-off-by: cmuhao <sduxuhao@gmail.com> * update dask version to support pandas 1.x Signed-off-by: cmuhao <sduxuhao@gmail.com> * update dask version to support pandas 1.x Signed-off-by: cmuhao <sduxuhao@gmail.com> --------- Signed-off-by: cmuhao <sduxuhao@gmail.com>

Fix OnDemandFeatureView type inference for array types Signed-off-by: Alex Mirrington <alex.mirrington@rokt.com>

…east-dev#4337) Bumps [zipp](https://github.com/jaraco/zipp) from 3.18.1 to 3.19.1. - [Release notes](https://github.com/jaraco/zipp/releases) - [Changelog](https://github.com/jaraco/zipp/blob/main/NEWS.rst) - [Commits](jaraco/zipp@v3.18.1...v3.19.1) --- updated-dependencies: - dependency-name: zipp dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…-dev#4287)

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

tsisodia10 · 2024-07-23T14:28:29Z

I encountered issues while rebasing and squashing commits from the master branch, resulting in an incorrect commit history for this PR. To resolve this, I will close this PR and open a new one after manually selecting and adding the necessary changes. This process might take some time, so I appreciate your understanding and patience during this period. Thank you.

tsisodia10 changed the title ~~Instrument Feast using Prometheus and OpenTelemetry~~ feat: Instrument Feast using Prometheus and OpenTelemetry Jun 20, 2024

tchughesiv suggested changes Jun 21, 2024

View reviewed changes

tsisodia10 changed the base branch from v0.38-branch to master June 21, 2024 20:02

tsisodia10 force-pushed the inst-stable branch from 781d1a9 to d341cf5 Compare June 21, 2024 20:23

redhatHameed reviewed Jun 21, 2024

View reviewed changes

tokoko reviewed Jun 25, 2024

View reviewed changes

sdk/python/feast/feature_store.py Outdated Show resolved Hide resolved

franciscojavierarceo and others added 8 commits June 26, 2024 10:33

Update SUMMARY.md

c73f633

Add

17d6978

Instrument Feast using Prometheus and OpenTelemetry

8cbd893

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Remove conflict

7ddec2c

Add newline and refine manifests

c1cd375

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Refine manifests

8e08b96

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Refine manifests

3062e38

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Add metrics flag

70ae69a

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

tsisodia10 force-pushed the inst-stable branch from 86a846f to 70ae69a Compare June 26, 2024 14:46

tchughesiv suggested changes Jun 26, 2024

View reviewed changes

tsisodia10 added 2 commits June 26, 2024 16:18

Revert values.yaml

7ec5207

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Refactor metric flag

def7c1b

tchughesiv reviewed Jul 1, 2024

View reviewed changes

sdk/python/feast/cli.py Outdated Show resolved Hide resolved

tsisodia10 added 4 commits July 9, 2024 16:55

Refinement

f6fbbbb

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Modify metric value in values.yaml

4e9ffb7

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Refine

b598615

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Remove sample metric

dc821ae

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

redhatHameed reviewed Jul 9, 2024

View reviewed changes

infra/charts/feast-feature-server/values.yaml Outdated Show resolved Hide resolved

sdk/python/feast/cli.py Show resolved Hide resolved

infra/charts/feast-feature-server/templates/service.yaml Show resolved Hide resolved

tsisodia10 added 2 commits July 9, 2024 19:09

Add metric check in service.yaml

ca97429

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

Remove namespace from values.yaml

7ae4e1e

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

tchughesiv suggested changes Jul 9, 2024

View reviewed changes

tchughesiv suggested changes Jul 16, 2024

View reviewed changes

tsisodia10 force-pushed the inst-stable branch from 844afb6 to f07eddb Compare July 23, 2024 12:15

tsisodia10 and others added 19 commits July 23, 2024 17:52

Modify deployment.yaml

22edf06

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

chore: Remove existing providers from the repo (feast-dev#4298)

e2d3576

remove providers Signed-off-by: tokoko <togurg14@freeuni.edu.ge>

Instrument feature store using opentelemetry and prometheus

54b2156

fix: CGO Memory leak issue in GO Feature server (feast-dev#4291)

cbe8491

fix: Fix SQLite import issue (feast-dev#4294)

d165f4b

adding try and except block for import Co-authored-by: Francisco Javier Arceo <franciscojavierarceo@users.noreply.github.com>

fix: Using one single function call for utcnow(). (feast-dev#4307)

f0d632b

Signed-off-by: Shuchu Han <shuchu.han@gmail.com>

chore: Upgrading sqlite_vec to latest package (feast-dev#4332)

8de8f78

Signed-off-by: Francisco Javier Arceo <franciscojavierarceo@users.noreply.github.com> Co-authored-by: Francisco Javier Arceo <franciscojavierarceo@users.noreply.github.com>

feat: Add Tornike to maintainers.md (feast-dev#4339)

97e0106

* Update maintainers.md * Update maintainers.md * Update maintainers.md

chore: Upgrade ibis to 9.0 (feast-dev#4330)

c280c4a

upgrade ibis version Signed-off-by: tokoko <togurg14@freeuni.edu.ge>

fix: OnDemandFeatureView type inference for array types (feast-dev#4310)

e4edc11

Fix OnDemandFeatureView type inference for array types Signed-off-by: Alex Mirrington <alex.mirrington@rokt.com>

chore: Bump braces from 3.0.2 to 3.0.3 in /sdk/python/feast/ui (feast…

c78a9de

…-dev#4287)

Instrument feast using Opentelemetry and Prometheus

a4e7005

Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com>

tsisodia10 force-pushed the inst-stable branch from f07eddb to a4e7005 Compare July 23, 2024 12:26

tsisodia10 requested review from sudohainguyen, DvirDukhan and a team as code owners July 23, 2024 12:26

tsisodia10 closed this Jul 23, 2024

tchughesiv mentioned this pull request Jul 30, 2024

feat: Instrument Feast using Prometheus and OpenTelemetry #4366

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Instrument Feast using Prometheus and OpenTelemetry #4300

feat: Instrument Feast using Prometheus and OpenTelemetry #4300

tsisodia10 commented Jun 20, 2024

tchughesiv left a comment

redhatHameed Jun 21, 2024 •

edited

Loading

tchughesiv Jun 26, 2024

tsisodia10 Jul 9, 2024 •

edited

Loading

tsisodia10 commented Jun 24, 2024

tsisodia10 commented Jun 26, 2024

tchughesiv left a comment

tchughesiv Jun 26, 2024

tsisodia10 Jul 9, 2024

tsisodia10 Jul 9, 2024 •

edited

Loading

tchughesiv Jul 9, 2024

tchughesiv Jul 9, 2024 •

edited

Loading

tchughesiv Jul 9, 2024

tsisodia10 Jul 10, 2024

tchughesiv Jun 26, 2024

tokoko commented Jun 26, 2024

tchughesiv left a comment •

edited

Loading

tokoko commented Jul 16, 2024

tchughesiv Jul 16, 2024

tsisodia10 Jul 16, 2024

tchughesiv Jul 16, 2024

tchughesiv Jul 16, 2024

tsisodia10 Jul 23, 2024

tchughesiv Jul 23, 2024

tsisodia10 commented Jul 23, 2024

feat: Instrument Feast using Prometheus and OpenTelemetry #4300

feat: Instrument Feast using Prometheus and OpenTelemetry #4300

Conversation

tsisodia10 commented Jun 20, 2024

What this PR does / why we need it:

tchughesiv left a comment

Choose a reason for hiding this comment

redhatHameed Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsisodia10 Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

tsisodia10 commented Jun 24, 2024

tsisodia10 commented Jun 26, 2024

tchughesiv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsisodia10 Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchughesiv Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tokoko commented Jun 26, 2024

tchughesiv left a comment • edited Loading

Choose a reason for hiding this comment

tokoko commented Jul 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsisodia10 commented Jul 23, 2024

redhatHameed Jun 21, 2024 •

edited

Loading

tsisodia10 Jul 9, 2024 •

edited

Loading

tsisodia10 Jul 9, 2024 •

edited

Loading

tchughesiv Jul 9, 2024 •

edited

Loading

tchughesiv left a comment •

edited

Loading