Add exception handler for model server and Add ability to specify custom handler

[sivanantha321]

What this PR does / why we need it:

• Handles Model server shutdown properly when exception occured.
• Provides a way to add custom exception handler to the event loop

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3404

Type of changes
Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

• Test A
  Before:

Traceback (most recent call last):
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/sklearnserver/sklearnserver/__main__.py", line 41, in <module>
    kserve.ModelServer(registered_models=SKLearnModelRepository(args.model_dir), workers=6).start(
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 229, in start
    asyncio.run(servers_task())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 227, in servers_task
    await asyncio.gather(*servers)
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 214, in serve
    raise RuntimeError("Some Exception")
RuntimeError: Some Exception
Exception ignored in: <function Server.__del__ at 0x7f90add16170>
Traceback (most recent call last):
  File "/home/ubuntu/go/src/github.com/kserve/kserve/.venv/lib/python3.10/site-packages/grpc/aio/_server.py", line 185, in __del__
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 118, in grpc._cython.cygrpc.schedule_coro_threadsafe
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 110, in grpc._cython.cygrpc.schedule_coro_threadsafe
  File "/usr/lib/python3.10/asyncio/base_events.py", line 436, in create_task
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'AioServer.shutdown' was never awaited

After:

Traceback (most recent call last):
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/sklearnserver/sklearnserver/__main__.py", line 41, in <module>
    kserve.ModelServer(registered_models=SKLearnModelRepository(args.model_dir), workers=6).start(
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 224, in start
    asyncio.run(servers_task())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 222, in servers_task
    await asyncio.gather(*servers)
  File "/home/ubuntu/go/src/github.com/kserve/kserve/python/kserve/kserve/model_server.py", line 209, in serve
    raise RuntimeError("Some Exception")
RuntimeError: Some Exception

• Logs

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

• Have you added unit/e2e tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Release note:

Add exception handler for model server and Add ability to specify custom exception handler

[rachitchauhan43]

@sivanantha321 : Possible to get a unit test case around this feature to test that custom exception handler is indeed getting added ?

[rachitchauhan43]

Left a comment regarding unit test , rest LGTM

[sivanantha321]

@sivanantha321 : Possible to get a unit test case around this feature to test that custom exception handler is indeed getting added ?

I tried. But I am not able to get it to work.

[oss-prow-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rachitchauhan43, sivanantha321

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• python/kserve/OWNERS [rachitchauhan43]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment