Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kf 3597 feat base class charm - kfp persistence #268

Merged
merged 44 commits into from
Aug 3, 2023
Merged

Kf 3597 feat base class charm - kfp persistence #268

merged 44 commits into from
Aug 3, 2023

Conversation

i-chvets
Copy link
Contributor

@i-chvets i-chvets commented Jul 23, 2023 •
edited
Loading

Initial version of base-class based KFP Persistence Agent charm.

Summary of changes:

  • Re-wrote charm using base-class framework.
  • Added integration tests local to the charm (minimal). Will rely on bundle level (top level integration tests).
  • Modified unit tests to be aligned with new framework.

NOTE: Due to deployment parameter in PodSpec version, kfp-persistence cannot be upgraded using Juju refresh:
juju.errors.JujuAPIError: Juju on containers does not support updating deployment info for services.
Upgrade will be done by re-deploying the charm.

@i-chvets i-chvets requested a review from a team as a code owner July 23, 2023 19:31
@i-chvets
Copy link
Contributor Author

PR fails in CI due to K8S resources:


App                           Version                  Status   Scale  Charm                         Channel         Rev  Address         Exposed  Message
grafana-k8s                   9.2.1                    active       1  grafana-k8s                   latest/stable    81  10.152.183.236  no       
kfp-api                                                waiting      1  kfp-api                                         0  10.152.183.16   no       installing agent
kfp-db                        mariadb/server:10.3      active       1  charmed-osm-mariadb-k8s       latest/stable    35  10.152.183.111  no       ready
kfp-viz                       res:oci-image@3de6f3c    blocked    0/1  kfp-viz                       2.0/stable      476  10.152.183.46   no       0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes ...
minio                         res:oci-image@1755999    blocked    0/1  minio                         ckf-1.7/stable  186  10.152.183.254  no       0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes ...
mysql-k8s                     8.0.32-0ubuntu0.22.04.2  waiting      1  mysql-k8s                     8.0/stable       75  10.152.183.202  no       waiting for units to settle down
prometheus-k8s                2.33.5                   active       1  prometheus-k8s                latest/stable   103  10.152.183.70   no       
prometheus-scrape-config-k8s  n/a                      active       1  prometheus-scrape-config-k8s  latest/stable    39  10.152.183.72   no

ca-scribner
ca-scribner reviewed Jul 24, 2023
View reviewed changes
Copy link
Contributor

@ca-scribner ca-scribner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking interesting! Overall looking great. Pretty much everything commented on was just something that'll be fun to discuss with others as they try this out, not a right/wrong thing. Figuring out the best feel for the patterns will be fun

@i-chvets
Copy link
Contributor Author

i-chvets commented Aug 1, 2023

Local integration tests pass:

$ tox -e integration
integration: commands[0]> pytest -vv --tb native --asyncio-mode=auto /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/tests/integration --log-cli-level=INFO -s
============================================ test session starts =============================================
platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.2.0 -- /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api
configfile: pyproject.toml
plugins: operator-0.28.0, asyncio-0.21.1
asyncio: mode=auto
collected 5 items                                                                                            

tests/integration/test_charm.py::TestCharm::test_build_and_deploy 
----------------------------------------------- live log setup -----------------------------------------------
INFO     pytest_operator.plugin:plugin.py:647 Adding model microk8s-localhost:test-charm-c4kt on cloud microk8s
----------------------------------------------- live log call ------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/tmp/pytest/test-charm-c4kt0
INFO     pytest_operator.plugin:plugin.py:982 Building charm kfp-api
INFO     pytest_operator.plugin:plugin.py:987 Built charm kfp-api in 17.09s
INFO     test_charm:test_charm.py:30 Built charm /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/tmp/pytest/test-charm-c4kt0/charms/kfp-api_ubuntu-20.04-amd64.charm
INFO     juju.model:model.py:1981 Deploying local:kfp-api-0
INFO     juju.model:model.py:1981 Deploying ch:amd64/focal/charmed-osm-mariadb-k8s-35
INFO     juju.model:model.py:1981 Deploying ch:amd64/focal/minio-186
INFO     juju.model:model.py:1981 Deploying ch:amd64/focal/kfp-viz-476
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [allocating] waiting: agent initializing
  kfp-viz/0 [allocating] waiting: installing agent
  kfp-db/0 [allocating] waiting: installing agent
  minio/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [allocating] waiting: agent initializing
  kfp-viz/0 [allocating] waiting: installing agent
  kfp-db/0 [executing] active: ready
  minio/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [idle] waiting: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
  kfp-viz/0 [allocating] waiting: agent initializing
  kfp-db/0 [idle] active: ready
  minio/0 [allocating] waiting: agent initializing
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [executing] waiting: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
  kfp-viz/0 [allocating] waiting: agent initializing
  minio/0 [executing] active: 
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [idle] active: 
  kfp-viz/0 [idle] active: 
  minio/0 [idle] active: 
PASSED
tests/integration/test_charm.py::TestCharm::test_relational_db_relation_with_mysql_relation 
----------------------------------------------- live log call ------------------------------------------------
INFO     juju.model:model.py:1981 Deploying ch:amd64/jammy/mysql-k8s-88
INFO     juju.model:model.py:2618 Waiting for model:
  mysql-k8s/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2618 Waiting for model:
  mysql-k8s/0 [allocating] waiting: agent initializing
INFO     juju.model:model.py:2618 Waiting for model:
  mysql-k8s/0 [allocating] waiting: agent initializing
INFO     juju.model:model.py:2618 Waiting for model:
  mysql-k8s/0 [executing] maintenance: Initialising mysqld
INFO     juju.model:model.py:2618 Waiting for model:
  mysql-k8s/0 [idle] active: Primary
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [idle] active: 
PASSED
tests/integration/test_charm.py::TestCharm::test_relational_db_relation_with_mysql_k8s 
----------------------------------------------- live log call ------------------------------------------------
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [idle] blocked: Remove deprecated mysql relation to unblock. See logs
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [idle] active: 
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [idle] blocked: Please add required database relation: eg. relational-db
PASSED
tests/integration/test_charm.py::TestCharm::test_msql_relation_with_relational_db_relation 
----------------------------------------------- live log call ------------------------------------------------
INFO     juju.model:model.py:2618 Waiting for model:
  kfp-api/0 [idle] active: 
PASSED
tests/integration/test_charm.py::TestCharm::test_prometheus_grafana_integration 
----------------------------------------------- live log call ------------------------------------------------
INFO     juju.model:model.py:1981 Deploying ch:amd64/focal/prometheus-k8s-103
INFO     juju.model:model.py:1981 Deploying ch:amd64/focal/grafana-k8s-81
INFO     juju.model:model.py:1981 Deploying ch:amd64/focal/prometheus-scrape-config-k8s-39
INFO     juju.model:model.py:2618 Waiting for model:
  grafana-k8s/0 [allocating] waiting: installing agent
  prometheus-k8s/0 [allocating] waiting: installing agent
  prometheus-scrape-config-k8s/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2618 Waiting for model:
  grafana-k8s/0 [allocating] waiting: agent initializing
  prometheus-k8s/0 [executing] maintenance: installing charm software
  prometheus-scrape-config-k8s/0 [executing] active: 
INFO     juju.model:model.py:2618 Waiting for model:
  grafana-k8s/0 [allocating] waiting: agent initializing
  prometheus-k8s/0 [executing] maintenance: installing charm software
INFO     juju.model:model.py:2618 Waiting for model:
  grafana-k8s/0 [executing] maintenance: installing charm software
  prometheus-k8s/0 [idle] active: 
INFO     juju.model:model.py:2618 Waiting for model:
  grafana-k8s/0 [executing] maintenance: installing charm software
INFO     juju.model:model.py:2618 Waiting for model:
  grafana-k8s/0 [executing] maintenance: installing charm software
INFO     juju.model:model.py:2618 Waiting for model:
  grafana-k8s/0 [idle] active: 
  prometheus-k8s/0 [idle] active: 
INFO     test_charm:test_charm.py:217 Prometheus available at http://10.1.59.108:9090
INFO     test_charm:test_charm.py:220 Testing prometheus deployment (attempt 1)
INFO     test_charm:test_charm.py:230 Response status is success
PASSED
--------------------------------------------- live log teardown ----------------------------------------------
INFO     pytest_operator.plugin:plugin.py:790 Model status:

Model            Controller          Cloud/Region        Version  SLA          Timestamp
test-charm-c4kt  microk8s-localhost  microk8s/localhost  2.9.44   unsupported  13:49:48-04:00

App                           Version                  Status  Scale  Charm                         Channel         Rev  Address         Exposed  Message
grafana-k8s                   9.2.1                    active      1  grafana-k8s                   latest/stable    81  10.152.183.204  no       
kfp-api                                                active      1  kfp-api                                         0  10.152.183.9    no       
kfp-db                        mariadb/server:10.3      active      1  charmed-osm-mariadb-k8s       latest/stable    35  10.152.183.235  no       ready
kfp-viz                       res:oci-image@3de6f3c    active      1  kfp-viz                       2.0/stable      476  10.152.183.252  no       
minio                         res:oci-image@1755999    active      1  minio                         ckf-1.7/stable  186  10.152.183.107  no       
mysql-k8s                     8.0.32-0ubuntu0.22.04.2  active      1  mysql-k8s                     8.0/edge         88  10.152.183.112  no       
prometheus-k8s                2.33.5                   active      1  prometheus-k8s                latest/stable   103  10.152.183.233  no       
prometheus-scrape-config-k8s  n/a                      active      1  prometheus-scrape-config-k8s  latest/stable    39  10.152.183.250  no       

Unit                             Workload  Agent  Address      Ports              Message
grafana-k8s/0*                   active    idle   10.1.59.110                     
kfp-api/0*                       active    idle   10.1.59.72                      
kfp-db/0*                        active    idle   10.1.59.94   3306/TCP           ready
kfp-viz/0*                       active    idle   10.1.59.89   8888/TCP           
minio/0*                         active    idle   10.1.59.98   9000/TCP,9001/TCP  
mysql-k8s/0*                     active    idle   10.1.59.97                      Primary
prometheus-k8s/0*                active    idle   10.1.59.108                     
prometheus-scrape-config-k8s/0*  active    idle   10.1.59.101                     


INFO     pytest_operator.plugin:plugin.py:796 Juju error logs:

unit-kfp-api-0: 13:40:43 ERROR unit.kfp-api/0.juju-log Failed to handle <LeaderElectedEvent via KfpApiOperator/on/leader_elected[13]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 13:40:44 ERROR unit.kfp-api/0.juju-log Failed to handle <PebbleReadyEvent via KfpApiOperator/on/ml_pipeline_api_server_pebble_ready[16]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 13:40:45 ERROR unit.kfp-api/0.juju-log Failed to handle <ConfigChangedEvent via KfpApiOperator/on/config_changed[19]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 13:40:47 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to handle <RelationJoinedEvent via KfpApiOperator/on/mysql_relation_joined[25]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 13:40:48 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to handle <RelationChangedEvent via KfpApiOperator/on/mysql_relation_changed[28]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 13:41:28 ERROR unit.kfp-api/0.juju-log object-storage:1: Failed to handle <RelationChangedEvent via KfpApiOperator/on/object_storage_relation_changed[34]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 13:41:31 ERROR unit.kfp-api/0.juju-log object-storage:1: Failed to handle <RelationChangedEvent via KfpApiOperator/on/object_storage_relation_changed[37]> with error: List of <ops.model.Relation kfp-viz:2> versions not found for apps: kfp-viz
unit-kfp-api-0: 13:41:34 ERROR unit.kfp-api/0.juju-log object-storage:1: Failed to handle <RelationChangedEvent via KfpApiOperator/on/object_storage_relation_changed[40]> with error: List of <ops.model.Relation kfp-viz:2> versions not found for apps: kfp-viz
unit-kfp-api-0: 13:41:37 ERROR unit.kfp-api/0.juju-log kfp-viz:2: Failed to handle <RelationChangedEvent via KfpApiOperator/on/kfp_viz_relation_changed[46]> with error: List of <ops.model.Relation kfp-viz:2> versions not found for apps: kfp-viz
unit-kfp-api-0: 13:44:38 ERROR unit.kfp-api/0.juju-log relational-db:4: Relation mysql is deprecated. Remove deprecated mysql relation to unblock.
unit-kfp-api-0: 13:44:39 ERROR unit.kfp-api/0.juju-log relational-db:4: Relation mysql is deprecated. Remove deprecated mysql relation to unblock.
unit-kfp-api-0: 13:44:40 ERROR unit.kfp-api/0.juju-log relational-db:4: Relation mysql is deprecated. Remove deprecated mysql relation to unblock.
unit-kfp-api-0: 13:44:40 ERROR unit.kfp-api/0.juju-log relational-db:4: Relation mysql is deprecated. Remove deprecated mysql relation to unblock.
unit-kfp-api-0: 13:45:12 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to generate container configuration.
unit-kfp-api-0: 13:45:12 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to handle <RelationDepartedEvent via KfpApiOperator/on/mysql_relation_departed[74]> with error: Please add required database relation: eg. relational-db
unit-kfp-api-0: 13:45:13 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to generate container configuration.
unit-kfp-api-0: 13:45:13 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to handle <RelationBrokenEvent via KfpApiOperator/on/mysql_relation_broken[77]> with error: Please add required database relation: eg. relational-db
unit-kfp-api-0: 13:45:31 ERROR unit.kfp-api/0.juju-log relational-db:5: Failed to generate container configuration.
unit-kfp-api-0: 13:45:31 ERROR unit.kfp-api/0.juju-log relational-db:5: Failed to handle <RelationJoinedEvent via KfpApiOperator/on/relational_db_relation_joined[83]> with error: Waiting for relational-db data
unit-kfp-api-0: 13:45:32 ERROR unit.kfp-api/0.juju-log relational-db:5: Failed to generate container configuration.
unit-kfp-api-0: 13:45:32 ERROR unit.kfp-api/0.juju-log relational-db:5: Failed to handle <RelationChangedEvent via KfpApiOperator/on/relational_db_relation_changed[86]> with error: Waiting for relational-db data
unit-kfp-api-0: 13:45:57 ERROR unit.kfp-api/0.juju-log mysql:6: Relation mysql is deprecated. Remove deprecated mysql relation to unblock.
unit-kfp-api-0: 13:45:58 ERROR unit.kfp-api/0.juju-log mysql:6: Relation mysql is deprecated. Remove deprecated mysql relation to unblock.

INFO     pytest_operator.plugin:plugin.py:884 Resetting model test-charm-c4kt...
INFO     pytest_operator.plugin:plugin.py:873    Destroying applications kfp-api
INFO     pytest_operator.plugin:plugin.py:873    Destroying applications kfp-db
INFO     pytest_operator.plugin:plugin.py:873    Destroying applications minio
INFO     pytest_operator.plugin:plugin.py:873    Destroying applications kfp-viz
INFO     pytest_operator.plugin:plugin.py:873    Destroying applications mysql-k8s
INFO     pytest_operator.plugin:plugin.py:873    Destroying applications prometheus-k8s
INFO     pytest_operator.plugin:plugin.py:873    Destroying applications grafana-k8s
INFO     pytest_operator.plugin:plugin.py:873    Destroying applications prometheus-scrape-config-k8s
INFO     pytest_operator.plugin:plugin.py:889 Not waiting on reset to complete.
INFO     pytest_operator.plugin:plugin.py:862 Forgetting main...


============================================== warnings summary ==============================================
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
tests/integration/test_charm.py::TestCharm::test_build_and_deploy
  /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:218: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._drain_lock = asyncio.Lock(loop=loop)

tests/integration/test_charm.py: 68 warnings
  /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:977: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.sleep(self.ping_interval, loop=self.loop)

tests/integration/test_charm.py: 3204 warnings
  /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:911: DeprecationWarning: 'with (yield from lock)' is deprecated use 'async with lock' instead
    with (yield from self._drain_lock):

tests/integration/test_charm.py: 3135 warnings
  /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:416: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait(

tests/integration/test_charm.py: 66 warnings
  /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:988: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::TestCharm::test_prometheus_grafana_integration
tests/integration/test_charm.py::TestCharm::test_prometheus_grafana_integration
  /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:532: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::TestCharm::test_prometheus_grafana_integration
tests/integration/test_charm.py::TestCharm::test_prometheus_grafana_integration
  /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:554: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::TestCharm::test_prometheus_grafana_integration
tests/integration/test_charm.py::TestCharm::test_prometheus_grafana_integration
  /home/ichvets/cw/dev/base-charm/kfp-operators/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:1077: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ 5 passed, 6481 warnings in 669.16s (0:11:09) ================================
  integration: OK (669.78=setup[0.04]+cmd[669.74] seconds)
  congratulations :) (669.84 seconds)

@i-chvets
Copy link
Contributor Author

i-chvets commented Aug 1, 2023

Another random general comment, should we have a convention to include components in a src/components directory?

Done.

Ivan Chvets added 4 commits August 2, 2023 10:33
fix: added get_status to pebble component
c32acf6
fix: updated requirements
eb7f7a1
fix: linting
a2fa34f
fix: updated according to comments
d9a0099 
Summary of changes:
- Removed relation component and use imported one from chisme.
- Modified compoent names according to new pattern.
- Use MySQL DB in integration tests.
fix: updated according to comments
b6bd936 
Summary of changes:
- Using status from parent class.
- Removed skipped bundle tests.
- Modified code to use chisme unit testing.
@ca-scribner ca-scribner mentioned this pull request Aug 3, 2023
Merged
@ca-scribner ca-scribner merged commit 4330b3a into main Aug 3, 2023
@ca-scribner ca-scribner deleted the kf-3597-feat-base-class-charm branch August 3, 2023 23:27
@i-chvets i-chvets mentioned this pull request Aug 21, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@orfeas-k orfeas-k orfeas-k left review comments

@ca-scribner ca-scribner ca-scribner approved these changes

Assignees
No one assigned
Labels
Libraries: OK
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@i-chvets @orfeas-k @ca-scribner