Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add support union type Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed test Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed test Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Update Union to use tagged Unions Signed-off-by: maximsmol <maximsmol@gmail.com> * Update to use string tags (part 1) Signed-off-by: maximsmol <maximsmol@gmail.com> * Working implementation, update tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixes, more tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix incorrect unwrapped literal-union matching, update test Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix duplicate tag handling, add tests for collections containing unions Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix type hint test Signed-off-by: maximsmol <maximsmol@gmail.com> * Add implicit wrapping union type tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Add union ambiguity tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixup tests, make TypeTransformerFailed compatible with all old exception types Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixup models + add tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Implement changed design Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix PR comments Signed-off-by: maximsmol <maximsmol@gmail.com> * Remote entrypoint serialize (#733) Signed-off-by: Emirhan Karagül <emirhan350z@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix lint error in remote.py (#755) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support enum in dataclass (#753) * Add support enum in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Update test Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix subworkflow and launch plan FlyteRemote behavior (#751) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Extras: Shell task (#747) Signed-off-by: maximsmol <maximsmol@gmail.com> * Add support FlyteSchema in dataclass (#722) * schema in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove workflow_execution.py (#758) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Get raw input/output from remote execution (#675) * [wip] for feast demo Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * clean up a bit Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove unneeded import Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add a part of a test Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * typo Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix mypy errors in flytekit/types (#757) Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * Remote client failed to fetch FlytePickle object (#764) * Fetch pickle value from flytekit remote Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove default value Signed-off-by: Kevin Su <pingsutw@apache.org> * Add support FlyteFile in dataclass (#725) * Add support Flyte File and directory in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * add task_resolver arg to @task decorator (#765) Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Copy metadata into map task from underlying (#766) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support for delayed annotations (#760) Signed-off-by: Stefan Nelson-Lindall <stef@stripe.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Complex dataclass unit tests (#773) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * remote_source lost on serialization of @dataclass_json with FlyteFile (#774) * remote_source lost on serialization of @dataclass_json with FlyteFile Signed-off-by: Kevin Su <pingsutw@apache.org> * updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Single-task execution FlyteRemote sync (#778) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Logging updates (#775) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: doc-requirements.txt to reduce vulnerabilities (#779) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-LXML-2316995 Signed-off-by: maximsmol <maximsmol@gmail.com> * Add cache_serialize parameter to tasks (#673) * added cache_serialize parameter for basic tasks Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed typo Signed-off-by: Daniel Rammer <daniel@union.ai> * updated flyteidl version Signed-off-by: Daniel Rammer <daniel@union.ai> * changed flyteidl requirements everywhere Signed-off-by: Daniel Rammer <daniel@union.ai> * remove flyteidl version requires in setup.py so we can use develop Signed-off-by: Daniel Rammer <daniel@union.ai> * removed flyteidl git repos from a variety of requirements packages Signed-off-by: Daniel Rammer <daniel@union.ai> * updated variable discovery_serializable to cache_serializable Signed-off-by: Daniel Rammer <daniel@union.ai> * updated requirements Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed TaskMetadata _cache_serializable variable name Signed-off-by: Daniel Rammer <daniel@union.ai> * propgating cache_serialize parameter through to tasks Signed-off-by: Daniel Rammer <daniel@union.ai> * added cache_serializable to SdkRawContainerTask Signed-off-by: Daniel Rammer <daniel@union.ai> * fixing cache_serializable variable propogation issues Signed-off-by: Daniel Rammer <daniel@union.ai> * added documentation Signed-off-by: Daniel Rammer <daniel@union.ai> * added unit tests for cache_serialize metadata Signed-off-by: Daniel Rammer <daniel@union.ai> * linter added spaces in unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * When using the `task` and `workflow` decorator, correctly wrap the fu… (#780) * When using the `task` and `workflow` decorator, correctly wrap the function This enables tooling such as docstring search tools to unwrap the object and show the correct docstring. Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Remove blackshark copyright header Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Fix broken great expectations test Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add test for stacked decorators Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add option to flyte-cli for specifying root certificate (#783) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add validation check to cacert switch (#787) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove pytz constraint (#786) * Remove pytz constraint from setup.py Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put pytz back Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * make requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Lint: remove f-string misuse (#788) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Pyarrow greater than 4.0.0 (#790) * pyarrow>=4.0.0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support python 3.10 (#791) * [wip] Support python 3.10 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add Dockerfile.py310 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Stringify python version Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Skip flytekit-modin plugin tests on 3.10 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add 3.9 and 3.10 to list of supported version in plugins Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Comment why flytekit-modin is not running on 3.10 and disable fail-fast in plugin tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * add `with_overrides` to map task (#794) * add with_overrides Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * remove Resources Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * bump docsearch version (#805) Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * update docs for new navbar theme (#806) Signed-off-by: maximsmol <maximsmol@gmail.com> * fix requirment.txt github issue (#810) Signed-off-by: Yuvraj <code@evalsocket.dev> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add sphinx panels (#815) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Schema overhaul (#785) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Parent workflow serialization fails when calling a launch plan with fixed inputs (#814) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix sagemaker plugin (#817) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove legacy API (#807) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Add Bigquery plugin (#789) * Add bigquery plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * Update dependency Signed-off-by: Kevin Su <pingsutw@apache.org> * update get_custom Signed-off-by: Kevin Su <pingsutw@apache.org> * Add structured dataset Signed-off-by: Kevin Su <pingsutw@apache.org> * Add structured dataset Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Add BQ in GA Signed-off-by: Kevin Su <pingsutw@apache.org> * alphabetical order Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed flytekit-papermill ImportError (#818) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add support for string-format-like sytax for shell task (#792) * POC: Add support for f-string like sytax for shell task This commit is a proof of concept adding f-string like syntax for shell_tasks. This supports using nested types for script inputs, such as data classes. This change was motivated by the desire to combine shell_tasks that have multiple inputs with map_tasks which only support tasks with a single input. This commit is only a starting point, since it makes some changes to the shell_task API (adds a template_style field), and modifies some of the default behavior for ease of implementation (e.g. throwing an error when there are unused input arguments). Signed-off-by: Zach Palchick <palchicz@zymergen.com> * Drop support for old/regex style for doing string interpolation Signed-off-by: Zach Palchick <palchicz@zymergen.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Expose configured RawOutputPrefix during execution (#813) * Expose configured RawOutputPrefix during execution Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove sdk_runnable.py and spark_task.py Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add SecretsManager back to old import location (#820) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add some tests (#819) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed flaky spark test (#821) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities (#823) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities (#825) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements-spark2.txt to reduce vulnerabilities (#826) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements.txt to reduce vulnerabilities (#824) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * Intratask checkpointing (#771) * wip - intratask checkpointing Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * sync checkpointer with tests Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Checkpinter in entrypoint Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * checkpoint in progress Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * wip Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Intratask checkpointer Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Checkpoint updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Intra-task checkpointing Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Test and entrypoint updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint fixed Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * test fixes Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fmt Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated entrypoint Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * update Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * print Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * SyncCheckpointer working Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * update Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed import problems Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed imports Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed lints and errors Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint fix Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * addressed comments Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support reading subset column (#822) * Support StructuredDatasetDecoder read subset column Signed-off-by: Kevin Su <pingsutw@apache.org> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed typo Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * [pr into #822] (#827) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * [pr into #822] Final update to structured dataset column subsetting (#828) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Fix spark regression (#830) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Update argument setting for in fast registered, dynamically generated, pod tasks (#835) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * `ctx` Context can be used within shell tasks - to access context vars and secrets (#832) * Adding context to a substitutable parameter in shell task Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Support for secrets in context Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * addressed comments Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Expose Checkpoint as a top-level interface in flytekit (#839) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` (#829) * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add test for execution closure Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add tests to Flyte remote Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Split execution test into with output and with error Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Gate new Structured Dataset feature & remove old config objects (#831) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Fixing out of order for conditional outputs (#843) Signed-off-by: maximsmol <maximsmol@gmail.com> * Set default values to map task template (#841) * Set sane defaults in map task templates Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove unused method Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put ArrayJob.from_dict back Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Define parallelism=0 as unbounded Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove special case to handle 0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Regenerate requirements files - dependencies dropping support for python 3.7 (#838) * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put restrictions on numpy and pandas versions Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Use --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * use pip==22.0.3 everywhere Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Relax click Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerated plugins requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * TypeAnnotation (#759) * feat: support for annotated simple + list Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: addition of annotation att to Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: core obj Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: proto model Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: testing suite Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: more stable typing introspection Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: strip legacy Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: explicitly allow only one annotation Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: direct type transformer tests Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: there and back test Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_extensions for get_origin Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: more semantic list generic unwrap Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: tmp requirements file with custom idl Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: nits Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: semantic error for unsupported complex literals Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: but Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: more tests ;) Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: imports Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: complex annotations Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: temp requirements files for unit tests Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint bug Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: tmp setup.py Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: use typing_extensions Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_extensions for annotated Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_ext Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: plugin tmp requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: doc requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: whitespace Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump flytekit Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: numpy version Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: pandas version Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: test import Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: flake8 lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: merge Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: papermill req Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: req Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * Remove singleton from structured dataset transformer engine (#848) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * Assign input and output to FlyteWorkflowExecution (#842) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add reference entities to FlyteTask and FlyteLaunchPlan (#850) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix fast registration error (#851) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add support for local execute in pod task (#852) Signed-off-by: ggydush-fn <greg.gydush@freenome.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add anonymous retry (#854) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: formatting, linting, typing_extensions * fix: do not use SDK types Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: update test comment Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: also check literal type castability when tags match Signed-off-by: maximsmol <maximsmol@gmail.com> * Point flyteidl to maxim's fork in CI and requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * [Core feature] Add Raw AWS Batch Task (#782) * Init plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * address comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed typo Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated AWS config * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Added comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Update config Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * use pyflyte execute Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Added comment Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add structured dataset encoder/decoder in fsspec plugin (#849) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Delete unnecessary auth configuration (#858) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed format alias in Flytekit docs (#844) * Fixed format alias Fixed docs for file format alias that weren't rendered properly. A warning popped up stating 'ignore' is deprecated, and to use 'ignore_paths' instead. Signed-off-by: SmritiSatyanV <smriti@union.ai> * Bump idl (#862) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Updated authoring.rst (#863) * Updated authoring.rst Added directive Rephrased sentence * Fixed build error Signed-off-by: SmritiSatyanV <smriti@union.ai> * test-build-1 Signed-off-by: SmritiSatyanV <smriti@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Updated authoring.rst (#866) Added the directive correctly Signed-off-by: SmritiSatyanV <smriti@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Change docs for HTMLPage type to say HTMLPage instead of PNGImage (#868) Signed-off-by: maximsmol <maximsmol@gmail.com> * Revisit StructuredDatasetDecoder interface (#865) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove legacy mentions in contributing guide (#870) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add GCS protocol in the structured dataset (#869) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Make fetched entities callable within workflows (#867) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Caching of offloaded objects (#762) * Remove flyteidl from install_requires Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Expose hash in Literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set hash in TypeEngine Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Modify cache key calculation to take hash into account Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Opt-in PandasDataFrameTransformer Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add unit tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Iterate using a flyteidl branch Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Move _hash_overridable to StructureDatasetTransformerEngine Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Move HashMethod to flytekit.core.hash Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix `unit_test` make target Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Split `unit_test` make target in two lines Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add assert to structured dataset compatibility test Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove TODO Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate plugins requirements files pointing to the right version of flyteidl. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set hash as a property of the literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Install plugins requirements in CI. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add hash.setter Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Install flyteidl directly Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Regenerate plugins requirements files pointing to the right version of flyteidl." This reverts commit c2dbb54. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * wip - Add support for univariate lists Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add support for lists of annotated objects Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revamp generation of cache key (to cover case of literals collections and maps) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Leave TODO for warning Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Add support for lists of annotated objects" This reverts commit 4b5f608. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "wip - Add support for univariate lists" This reverts commit adaa448. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove docstring Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add flyteidl>=0.23.0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove mentions to branch flyteidl@add-hash-to-literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Bump flyteidl in plugins requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate plugins requirements again Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Restore papermill/requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Point flytekitplugins-spark to the offloaded-objects-caching branch in papermill tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set flyteidl>=0.23.0 in papermill dev-requirements Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * FlyteRemote fetch of conditional nodes (#772) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Removed root logger (#871) * Removed root logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated log level Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * lint fixed Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix flytekit_compatibility/test_schema_types.py test Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * make lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * fix: annotated type conversion error Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: _are_types_castable based on tests Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: test failing if using random order Signed-off-by: maximsmol <maximsmol@gmail.com> * Merge branch 'master' into maximsmol/union_type Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: merge issue Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: schema transformer error Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: test Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: merge issue Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: union + annotated behavior Signed-off-by: maximsmol <maximsmol@gmail.com> * Regenerate requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Bump requirements in plugins Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Handle nested Annotated Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Leave TODO re: strucutured dataset type castability Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove mention to flyteidl@union_type in doc-requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Linting Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Use tempfile.mkdtemp to create a temporary directory for local data persistence. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Use tempfile.mkdtemp to create a temporary directory for local data persistence." This reverts commit 861b157. Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Force temporary file to not be deleted in test Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Regenerate papermill dev-requirements Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Remove duplicate code Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put a lower bound on the pip version installed in CI Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Emirhan Karagül <emirhan350z@gmail.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com> Co-authored-by: Lisa <30621230+aeioulisa@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Stef Nelson-Lindall <bethebunny@gmail.com> Co-authored-by: Snyk bot <github+bot@snyk.io> Co-authored-by: Dan Rammer <hamersaw@protonmail.com> Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Co-authored-by: Samhita Alla <aallasamhita@gmail.com> Co-authored-by: Yuvraj <code@evalsocket.dev> Co-authored-by: Zach Palchick <palchicz@users.noreply.github.com> Co-authored-by: Snyk bot <snyk-bot@snyk.io> Co-authored-by: Kenny Workman <31255434+kennyworkman@users.noreply.github.com> Co-authored-by: ggydush-fn <69013027+ggydush-fn@users.noreply.github.com> Co-authored-by: SmritiSatyanV <94349093+SmritiSatyanV@users.noreply.github.com> Co-authored-by: Matthew Griffin <1matthewgriffin@gmail.com> Co-authored-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>
- Loading branch information
25 people
authored
Mar 21, 2022
1 parent
077c32a
commit d1702fc