Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add UI/SDK support to pyfunc ensemblers #171

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
59cd9b7
Move getensemblerdirectory method from ensembling_job_service to ense…
deadlycoconuts Feb 17, 2022
e8af889
Add holders in appcontext to carry an ensembler service builder
deadlycoconuts Feb 18, 2022
7abc204
Make deployment controller utilise pyfunc configs to generate docker …
deadlycoconuts Feb 18, 2022
c695770
Refactor tests with changes to ensembler config schema
deadlycoconuts Feb 21, 2022
f8d6871
Revert changes to get ensembler directory function
deadlycoconuts Feb 21, 2022
4de511e
Create config class for resource request and timeout
deadlycoconuts Feb 22, 2022
cc5cf83
Fix migration script for new py_func_ref_config
deadlycoconuts Feb 22, 2022
272dc24
Fix lint comments
deadlycoconuts Feb 22, 2022
3288a50
Fix port for ensembler in turing api
deadlycoconuts Feb 24, 2022
42936bd
Update e2e test data
deadlycoconuts Feb 25, 2022
7f73b76
Revert combination of resource request and timeout into a single object
deadlycoconuts Feb 25, 2022
f8dbf5c
Revert changes to e2e test data
deadlycoconuts Feb 25, 2022
f04862d
Rename py_func_ref_config to pyfunc_config
deadlycoconuts Feb 28, 2022
fe8eb1f
Rename pyfunc_config to make it consistent with camelcase PyfuncConfig
deadlycoconuts Feb 28, 2022
ab669e2
Remove go build tags
deadlycoconuts Mar 1, 2022
4e00dbf
Remove go build tag from integration_config
deadlycoconuts Mar 1, 2022
848c2ab
Refactor image building step into DeployRouterVersion process
deadlycoconuts Mar 2, 2022
901beb9
Rework image/docker naming convention to include run ids
deadlycoconuts Mar 2, 2022
cdfe563
Add new PyFuncEnsembler class
deadlycoconuts Feb 28, 2022
8d9682f
Fix typo of file name for ServiceAccountCombobox
deadlycoconuts Feb 28, 2022
3ebc628
Add PyFunc ensembler config form
deadlycoconuts Feb 28, 2022
3c2cda3
Add PyFunc ensembler config view
deadlycoconuts Feb 28, 2022
81d3c31
Remove redundant passing of props
deadlycoconuts Feb 28, 2022
980d6ba
Refactor new name of pyfunc_config
deadlycoconuts Feb 28, 2022
c31c898
Update SDK with latest addition of PyfuncEnsemblers
deadlycoconuts Mar 3, 2022
958edc0
Add pytests for new pyfunc ensembler config
deadlycoconuts Mar 3, 2022
b385cff
Add leftover openapi generated files
deadlycoconuts Mar 3, 2022
73644c5
Clean up rebased files
deadlycoconuts Mar 7, 2022
bc28f3a
Add type casting for project_id
deadlycoconuts Mar 9, 2022
adf02da
Add optional docker config display to pyfunc view
deadlycoconuts Mar 9, 2022
a8d3ba5
Fix optional docker config display to pyfunc view
deadlycoconuts Mar 9, 2022
ed9999a
Fix OpenAPI return value inconsistencies
deadlycoconuts Mar 9, 2022
41d0ef4
Add missing openapi file
deadlycoconuts Mar 9, 2022
38d8e9a
Fix OpenAPI specs and classes
deadlycoconuts Mar 9, 2022
993e6e5
Replace EuiSuperSelect with EuiComboBox
deadlycoconuts Mar 10, 2022
7707a6a
Rearrange statements for clarity
deadlycoconuts Mar 10, 2022
0886907
Remove default ensembler_id and cleanup naming irregularities
deadlycoconuts Mar 10, 2022
4ec891c
Add version field to Router SDK class
deadlycoconuts Mar 10, 2022
cd2ba25
Refactor wait for status to update current router with latest configs
deadlycoconuts Mar 10, 2022
1225893
Refactor wait for router version back to original
deadlycoconuts Mar 10, 2022
99a077c
Fix quotation marks
deadlycoconuts Mar 10, 2022
d6cde02
Add additional precision to pyfunc ensembler description
deadlycoconuts Mar 11, 2022
ec080a3
Refactor SDK and OpenAPI specs
deadlycoconuts Mar 11, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 39 additions & 7 deletions api/api/openapi.bundle.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/IdObject'
$ref: '#/components/schemas/JobId'
description: Accepted.
"400":
description: Invalid ensembling job
Expand Down Expand Up @@ -551,7 +551,7 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/IdObject_1'
$ref: '#/components/schemas/RouterId'
description: OK
"400":
description: Invalid project_id or router_id
Expand Down Expand Up @@ -682,7 +682,7 @@ paths:
content:
application/json:
schema:
$ref: '#/components/schemas/IdObject_1'
$ref: '#/components/schemas/RouterIdObject'
description: OK
"400":
description: Invalid project_id or router_id
Expand Down Expand Up @@ -863,9 +863,7 @@ paths:
content:
application/json:
schema:
items:
$ref: '#/components/schemas/Event'
type: array
$ref: '#/components/schemas/RouterEvents'
description: Get events
"400":
description: Invalid project_id or router_id
Expand Down Expand Up @@ -1694,13 +1692,16 @@ components:
- failed_submission
- failed_building
type: string
JobId:
$ref: '#/components/schemas/IdObject'
IdObject:
example:
id: 0
properties:
id:
format: int32
type: integer
type: object
Id:
format: int32
type: integer
Expand Down Expand Up @@ -2520,7 +2521,7 @@ components:
- environment_name
- name
type: object
IdObject_1:
RouterId:
$ref: '#/components/schemas/IdObject'
RouterIdAndVersion:
example:
Expand All @@ -2534,6 +2535,37 @@ components:
format: int32
type: integer
type: object
RouterIdObject:
example:
router_id: 0
properties:
router_id:
format: int32
type: integer
type: object
RouterEvents:
example:
events:
- event_type: info
updated_at: 2000-01-23T04:56:07.000+00:00
stage: stage
created_at: 2000-01-23T04:56:07.000+00:00
id: 0
message: message
version: 6
- event_type: info
updated_at: 2000-01-23T04:56:07.000+00:00
stage: stage
created_at: 2000-01-23T04:56:07.000+00:00
id: 0
message: message
version: 6
properties:
events:
items:
$ref: '#/components/schemas/Event'
type: array
type: object
Event:
example:
event_type: info
Expand Down
1 change: 1 addition & 0 deletions api/api/specs/common.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ components:
format: "int32"

IdObject:
type: "object"
properties:
id:
$ref: "#/components/schemas/Id"
Expand Down
4 changes: 2 additions & 2 deletions api/api/specs/jobs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -125,15 +125,15 @@ paths:
content:
application/json:
schema:
$ref: "#/components/schemas/IdObject"
$ref: "#/components/schemas/JobId"
400:
description: "Invalid ensembling job"
404:
description: "Ensembling job not found"

components:
schemas:
IdObject:
JobId:
$ref: "common.yaml#/components/schemas/IdObject"
EnsemblingJob:
description: A JSON object that represents an ensembling job for batch experiment use cases
Expand Down
26 changes: 19 additions & 7 deletions api/api/specs/routers.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ paths:
content:
application/json:
schema:
$ref: "#/components/schemas/IdObject"
$ref: "#/components/schemas/RouterId"
400:
description: "Invalid project_id or router_id"
404:
Expand Down Expand Up @@ -212,7 +212,7 @@ paths:
content:
application/json:
schema:
$ref: "#/components/schemas/IdObject"
$ref: "#/components/schemas/RouterIdObject"
400:
description: "Invalid project_id or router_id"
404:
Expand Down Expand Up @@ -380,18 +380,30 @@ paths:
content:
application/json:
schema:
type: "array"
items:
$ref: "#/components/schemas/Event"
$ref: "#/components/schemas/RouterEvents"
400:
description: "Invalid project_id or router_id"
404:
description: "No router version found"

components:
schemas:
IdObject:
$ref: "common.yaml#/components/schemas/IdObject"
RouterId:
$ref: "./common.yaml#/components/schemas/IdObject"

RouterIdObject:
type: object
properties:
router_id:
$ref: "common.yaml#/components/schemas/Id"

RouterEvents:
type: object
properties:
events:
type: "array"
items:
$ref: "#/components/schemas/Event"

RouterIdAndVersion:
type: object
Expand Down
1 change: 0 additions & 1 deletion sdk/samples/router/create_from_existing_router.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,6 @@ def main(turing_api: str, project: str):

# Create an ensembler for the router
ensembler = DockerRouterEnsemblerConfig(
id=1,
image="ealen/echo-server:0.5.1",
resource_request=ResourceRequest(
min_replica=1,
Expand Down
1 change: 0 additions & 1 deletion sdk/samples/router/general.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,6 @@ def main(turing_api: str, project: str):

# Create an ensembler for the router
ensembler = DockerRouterEnsemblerConfig(
id=1,
image="ealen/echo-server:0.5.1",
resource_request=ResourceRequest(
min_replica=1,
Expand Down
27 changes: 22 additions & 5 deletions sdk/tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -333,14 +333,24 @@ def generic_ensembler_docker_config(generic_resource_request, generic_env_var):
)


@pytest.fixture(params=["standard", "docker"])
def ensembler(request, generic_ensembler_standard_config, generic_ensembler_docker_config):
@pytest.fixture
def generic_ensembler_pyfunc_config(generic_resource_request):
return turing.generated.models.EnsemblerPyfuncConfig(
project_id=77,
ensembler_id=11,
resource_request=generic_resource_request,
timeout="500ms"
)


@pytest.fixture(params=["standard", "docker", "pyfunc"])
def ensembler(request, generic_ensembler_standard_config, generic_ensembler_docker_config, generic_ensembler_pyfunc_config):
ensembler_type = request.param
return turing.generated.models.RouterEnsemblerConfig(
id=1,
type=ensembler_type,
standard_config=generic_ensembler_standard_config,
docker_config=generic_ensembler_docker_config,
pyfunc_config=generic_ensembler_pyfunc_config,
created_at=datetime.now() + timedelta(seconds=10),
updated_at=datetime.now() + timedelta(seconds=10)
)
Expand All @@ -362,6 +372,14 @@ def generic_docker_router_ensembler_config(generic_ensembler_docker_config):
)


@pytest.fixture
def generic_pyfunc_router_ensembler_config(generic_ensembler_pyfunc_config):
return turing.generated.models.RouterEnsemblerConfig(
type="pyfunc",
pyfunc_config=generic_ensembler_pyfunc_config
)


@pytest.fixture
def generic_enricher(generic_resource_request, generic_env_var):
return turing.generated.models.Enricher(
Expand Down Expand Up @@ -508,7 +526,6 @@ def generic_router_config():
]
),
ensembler=DockerRouterEnsemblerConfig(
id=1,
image="test.io/just-a-test/turing-ensembler:0.0.0-build.0",
resource_request=ResourceRequest(
min_replica=1,
Expand Down Expand Up @@ -559,7 +576,7 @@ def generic_routers(project, num_routers, generic_router_status, generic_router_

@pytest.fixture
def generic_events():
return turing.generated.models.InlineResponse2002(
return turing.generated.models.RouterEvents(
events=[
turing.generated.models.Event(
created_at=datetime.now(),
Expand Down
Loading