Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Serialization bug due to version incompatibility between azure-core and msrest #763

Merged
merged 15 commits into from
Oct 19, 2022
Merged

Serialization bug due to version incompatibility between azure-core and msrest #763

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
89c9bfd
Adding DevSkim linter to Github actions
jainr Sep 9, 2022
1c8c13d
Merge pull request #1 from jainr/rijai/devskim
jainr Sep 9, 2022
a9c2343
Fix in ARM template to pull latest tagged release image from dockerhub
jainr Sep 14, 2022
cd056a9
Removing dev skim file from this branch
jainr Sep 14, 2022
cf52bd8
Merge branch 'main' into rijai/dockerimageversion
jainr Sep 14, 2022
d7a41da
Fixing linkedin org reference
jainr Sep 14, 2022
ef949a1
Fixing merge conflicts
jainr Sep 14, 2022
dc9ecbd
Removing the docker index url from dockerhub image name as it seems t…
jainr Sep 20, 2022
5e763b5
Merge branch 'main' of https://github.com/feathr-ai/feathr into rijai…
jainr Sep 20, 2022
cb32fb4
Adding to the right file, had a dockerhub workflow file with differen…
jainr Sep 20, 2022
ceefd9f
Adding debug statements to test udf issue on Synapse
jainr Sep 26, 2022
12931bc
Adding more print statements
jainr Sep 28, 2022
efecadb
Merge branch 'main' of https://github.com/feathr-ai/feathr into rijai…
jainr Oct 18, 2022
43d1dfd
Pinning msrest version to work with pinned version of azure-core
jainr Oct 18, 2022
9720232
Removing debug code from previous branch
jainr Oct 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions feathr_project/feathr/udf/_preprocessing_pyudf_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ def build_anchor_preprocessing_metadata(anchor_list: List[FeatureAnchor], local_
continue
preprocessing_func = anchor.source.preprocessing
if preprocessing_func:
print("Preprocessing function defined")
jainr marked this conversation as resolved.
Show resolved Hide resolved
print(anchor.source.preprocessing.name)
_PreprocessingPyudfManager.persist_pyspark_udf_to_file(preprocessing_func, local_workspace_dir)
feature_names = [feature.name for feature in anchor.features]
features_with_preprocessing = features_with_preprocessing + feature_names
Expand Down Expand Up @@ -109,9 +111,13 @@ def persist_pyspark_udf_to_file(user_func, local_workspace_dir):
however if we retrieve features from registry, the current implementation is to use plain strings to store the function body. In that case, the user_fuc will be string.
"""
if isinstance(user_func, str):
print("In If loop -> user function name is : " + user_func)
udf_source_code = [user_func]
else:
print("In else loop -> user function name is : " + user_func)
udf_source_code = inspect.getsourcelines(user_func)[0]
print("UDF SOURCE -----> " + udf_source_code)

lines = []
# Some basic imports will be provided
lines = lines + PROVIDED_IMPORTS
Expand Down
4 changes: 4 additions & 0 deletions feathr_project/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@
# https://github.com/Azure/azure-sdk-for-python/pull/22891
# using a version lower than that to workaround this issue.
"azure-core<=1.22.1",
# azure-core 1.22.1 is dependent on msrest==0.6.21, if an environment(AML) has a different version of azure-core (say 1.24.0),
# it brings a different version of msrest(0.7.0) which is incompatible with azure-core==1.22.1. Hence we need to pin it.
# See this for more details: https://github.com/Azure/azure-sdk-for-python/issues/24765
"msrest<=0.6.21",
"typing_extensions>=4.2.0"
],
tests_require=[ # TODO: This has been depricated
Expand Down