Job db #30

bertsky · 2023-07-06T12:38:27Z

Builds on #29, adds using the database for job monitoring as well – depends on slub/ocrd_manager#62

…rd_monitor into feature/browser-db

…s Browser behavior in tests

…nction.

bertsky · 2023-07-06T13:01:55Z

Minimal diff (against #29 directly) is here

SvenMarcus · 2023-07-25T07:01:44Z

For some reason, this PR is missing a commit, despite being pushed to the job-db branch. Mypy fails, although the attribute name has already been fixed in the latest commit. Link to code line

Adding another commit seems to have resolved the issue.

markusweigelt · 2023-08-04T11:51:40Z

@SvenMarcus When I start the monitor via the base repo I get the following error. Is there something wrong with the configuration or is this still a problem in the monitor.

Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 416, in main
    run(
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 587, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve
    config.load()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 467, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/ocrd-monitor/ocrdmonitor/main.py", line 5, in <module>
    settings = Settings()
               ^^^^^^^^^^
  File "pydantic/env_settings.py", line 40, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
monitor_db_connection_string
  field required (type=value_error.missing)

ATM I checked out the open pr branches in the submodules without testing the monitor individually.

SvenMarcus · 2023-08-04T12:00:28Z

@SvenMarcus When I start the monitor via the base repo I get the following error. Is there something wrong with the configuration or is this still a problem in the monitor.

Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 416, in main
    run(
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 587, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve
    config.load()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 467, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/ocrd-monitor/ocrdmonitor/main.py", line 5, in <module>
    settings = Settings()
               ^^^^^^^^^^
  File "pydantic/env_settings.py", line 40, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
monitor_db_connection_string
  field required (type=value_error.missing)

ATM I checked out the open pr branches in the submodules without testing the monitor individually.

The meta repository doesn't contain the necessary configuration yet. The monitor has some renamed and new variables (see docker-compose.yml).

markusweigelt · 2023-08-04T12:24:27Z

Ok i will check this. I thought that all changes regarding Monitor adjustments were made in this Base Repository PR

markusweigelt · 2023-08-04T12:41:08Z

In docker-compose.yml we extend from the submodule Docker Compose file. The add-database branch of @bertsky is linked to the changes of this PR

https://github.com/slub/ocrd_kitodo/blob/745f5d7b686eaf067e85a4d7c55a57e515f30e6d/docker-compose.yml#L13C1-L19
and
https://github.com/slub/ocrd_kitodo/blob/745f5d7b686eaf067e85a4d7c55a57e515f30e6d/docker-compose.yml#L26-L37

So the MONITOR_DB_CONNECTION env should be available.

markusweigelt · 2023-08-04T12:48:50Z

Docker inspect of not running Monitor container outputs following env`s

"Env": [
    "MONITOR_PORT_LOG=8088",
    "CONTROLLER=ocrd-controller:22",
    "MONITOR_DB_CONNECTION=mongodb://root:root_password@ocrd-database:27017",
    "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "LANG=C.UTF-8",
    "GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D",
    "PYTHON_VERSION=3.11.4",
    "PYTHON_PIP_VERSION=23.1.2",
    "PYTHON_SETUPTOOLS_VERSION=65.5.1",
    "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py",
    "PYTHON_GET_PIP_SHA256=96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207"
],

markusweigelt · 2023-08-04T14:43:39Z

Problem solved with renaming DB_CONNECTION_STRING to MONITOR_DB_CONNECTION_STRING and DB_CONNECTION to MONITOR_DB_CONNECTION in init.sh.

SvenMarcus · 2023-08-08T17:15:06Z

Took me ages, but the settings finally work with pydantic v2. Maybe I'm using it wrong, but pydantic-settings for v2 seems way more unintuitive than the previous version, so we should probably keep it in mind in case we can simplify some things.

markusweigelt · 2023-08-10T08:14:09Z

A small thing caught my attention for which I have created an issue. #31

SvenMarcus added 26 commits May 2, 2023 17:02

Initial failing browser db test

c38a13b

Passes first DB test. Needs refactoring!

ca7f4df

Begin implementing repository

5542972

Initial failing browser db test

a525a1d

Passes first DB test. Needs refactoring!

3f0e307

Begin implementing repository

c908ad8

Refactor DB test. Make repository setup async

aaf6fec

Merge branch 'feature/browser-db' of https://github.com/SvenMarcus/oc…

0801e99

…rd_monitor into feature/browser-db

Work on test setup

4e625e8

Refactor fixtures. Begin refactor towards running browser

91ca4df

Factory responsible for starting browsers.

6a321eb

Remove RedirectMap from workspace endpoint. Restoring factory control…

9e57fee

…s Browser behavior in tests

Use FastAPI dependency injection

1406a15

Update docker-compose.yml and init.sh for MongoDB

a0877cf

Working Docker Browsers. Subprocesses are crashing

25683e0

Working subprocess browsers

9124bf3

Try ports instead of taking from set

fe176f1

Mark launch tests as integration

59ae366

Add simple test to ping browser. Remove unused Port class

f7cdf42

Build browse-ocrd docker image in CI

f16cff5

Introduce first method for repository

d1a0167

Fix mypy failure

fd4767b

Clean unreachable browsers on startup. Refactor workspace endpoint

94a5966

Catch exception when killing browser fails

546aecd

Pass SubProcessBrowser Tests. Refactor towards common port binding fu…

98d03ce

…nction.

Better, but still insufficient, handling of native browser launches

a8bf2b8

bertsky requested review from SvenMarcus and markusweigelt July 6, 2023 12:38

bertsky mentioned this pull request Jul 6, 2023

Add database slub/ocrd_kitodo#89

Merged

SvenMarcus approved these changes Jul 24, 2023

View reviewed changes

SvenMarcus mentioned this pull request Jul 24, 2023

Feature/browser db #29

Closed

3 tasks

Fix long line in logs

54126b8

Add devcontainer config

8538894

Fix monitor db connection string, rename mongodb express service

2d74cca

markusweigelt and others added 4 commits August 7, 2023 12:57

using setting in pydantic v2

8467cf2

Add pydantic settings module

f84999c

Add pydantic settings module and update dependencies to run pydantic

c084c60

Migrate settings to pydantic v2

115dbc5

markusweigelt added 8 commits August 9, 2023 17:36

Add ocrd browser

82786b0

Add missing newlines

e44af1d

Install necessary deps in test ci

604bfec

Install necessary deb in test ci

3dca921

Install and using pdm

8a9a98a

Install necessary deps

5982a4c

Install using dev group

a9ae2d3

Change installation of deps in test ci

64ebfaf

markusweigelt approved these changes Aug 10, 2023

View reviewed changes

markusweigelt merged commit da26477 into main Aug 10, 2023

bertsky mentioned this pull request Aug 14, 2023

Configurable Manager endpoint #31

Closed

markusweigelt deleted the job-db branch August 29, 2023 14:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Job db #30

Job db #30

bertsky commented Jul 6, 2023

bertsky commented Jul 6, 2023

SvenMarcus commented Jul 25, 2023 •

edited

Loading

markusweigelt commented Aug 4, 2023

SvenMarcus commented Aug 4, 2023

markusweigelt commented Aug 4, 2023 •

edited

Loading

markusweigelt commented Aug 4, 2023

markusweigelt commented Aug 4, 2023

markusweigelt commented Aug 4, 2023

SvenMarcus commented Aug 8, 2023

markusweigelt commented Aug 10, 2023

Job db #30

Job db #30

Conversation

bertsky commented Jul 6, 2023

bertsky commented Jul 6, 2023

SvenMarcus commented Jul 25, 2023 • edited Loading

markusweigelt commented Aug 4, 2023

SvenMarcus commented Aug 4, 2023

markusweigelt commented Aug 4, 2023 • edited Loading

markusweigelt commented Aug 4, 2023

markusweigelt commented Aug 4, 2023

markusweigelt commented Aug 4, 2023

SvenMarcus commented Aug 8, 2023

markusweigelt commented Aug 10, 2023

SvenMarcus commented Jul 25, 2023 •

edited

Loading

markusweigelt commented Aug 4, 2023 •

edited

Loading