Build against TF2.13 #2835

seanpmorgan · 2023-05-31T02:37:57Z

Description

Waiting on:

TF2.13 release
Update to TFA README regarding EOL
TF-2.13 macos release

seanpmorgan · 2023-05-31T02:57:11Z

@qlzh727 getting a lot of Please ensure wrapped layer is a valid Keras layer -- is there a new spec we have to comply with?

qlzh727 · 2023-05-31T15:56:52Z

I think there is a newly added check in keras-team/keras@7eb8ef2#diff-5b101fb3499cadc6810aaa4ec68b961018d6d0c86ed9e9295e5ffb097e516be2 for the layer instance type checking in wrapper.

seanpmorgan · 2023-05-31T16:07:58Z

Thanks... this is causing breakage throughout the repo ... is there documentation you can share on what what changed here? Can we use a legacy serialization instead?

I'm also seeing seq2seq break for many tests... mind taking a look when time allows?

qlzh727 · 2023-05-31T16:42:30Z

From the test error log, it seems that the wrapper was feeding a tensor, rather than a layer instance, is that expected behavior?

Also @nkovela1 who is the original author for the change on keras side.

seanpmorgan · 2023-06-01T04:27:56Z

From the test error log, it seems that the wrapper was feeding a tensor, rather than a layer instance, is that expected behavior?

Also @nkovela1 who is the original author for the change on keras side.

Ack, the test was to verify an error is thrown on a Tensor input and it now raises ValueError instead of AssertionError. We can test for either or using pytest without much issue.

There is another error though that is only happening on TF2.13:

ValueError: Unknown layer: 'SpectralNormalization'. Please ensure you are using a `keras.utils.custom_object_scope` and that this object is included in the scope. See https://www.tens
orflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.

We register our layers using @tf.keras.utils.register_keras_serializable(package="Addons") . Is there a new check that is forcing a keras.utils.custom_object_scope ? Seems like a regression?

qlzh727 · 2023-06-01T16:13:20Z

Member

There is a fork of SpectralNormalization layer sent to keras with keras-team/keras#17648, and I think you are now getting a naming conflict.

seanpmorgan · 2023-06-26T19:47:44Z

Member

There is a fork of SpectralNormalization layer sent to keras with keras-team/keras#17648, and I think you are now getting a naming conflict.

@qlzh727 Appears that this isn't related to Addons as I can make minimal reproducible examples:
tensorflow/tensorflow#61085

Can you please help get this looked at before final is cut

qlzh727 · 2023-06-26T20:47:58Z

Member

There is a fork of SpectralNormalization layer sent to keras with keras-team/keras#17648, and I think you are now getting a naming conflict.

@qlzh727 Appears that this isn't related to Addons as I can make minimal reproducible examples: tensorflow/tensorflow#61085

Can you please help get this looked at before final is cut

Ack. CC'ed related ppl on the issue.

This reverts commit a6ff967.

bhack · 2023-07-03T17:54:28Z

@seanpmorgan You can cherry-pick a6ff967 when 2.13 final is out.

bhack · 2023-07-10T20:31:59Z

@qlzh727 The test is the same as always but it is failing now. Here the log..
https://source.cloud.google.com/results/invocations/037ec635-4ae1-4658-84bc-d6ec0af40bba/log

qlzh727 · 2023-07-10T20:43:14Z

@qlzh727 The test is the same as always but it is failing now. Here the log.. https://source.cloud.google.com/results/invocations/037ec635-4ae1-4658-84bc-d6ec0af40bba/log

This is quite weird, the same test is passing for 2.12 and failing for 2.13? Do we see same error if we build with tf-nightly?

seanpmorgan · 2023-07-10T21:12:12Z

@qlzh727 The test is the same as always but it is failing now. Here the log.. https://source.cloud.google.com/results/invocations/037ec635-4ae1-4658-84bc-d6ec0af40bba/log

This is quite weird, the same test is passing for 2.12 and failing for 2.13? Do we see same error if we build with tf-nightly?

Yes, the tests pass on TF2.12:
https://source.cloud.google.com/results/invocations/836b1250-2695-44d0-ad6d-b7cda5f65722/targets/tensorflow_addons%2Fubuntu%2Fgpu%2Fpy3%2Fpresubmit/log

Based on this commit:
606071c

This reverts commit 606071c.

seanpmorgan · 2023-07-10T21:35:27Z

Confirmed that it is TF2.13 and not the container:

https://source.cloud.google.com/results/invocations/67d3a1b3-2448-411d-9b83-2f90ab204982/targets/tensorflow_addons%2Fubuntu%2Fgpu%2Fpy3%2Fpresubmit/log

seanpmorgan · 2023-07-10T21:44:29Z

@qlzh727 The test is the same as always but it is failing now. Here the log.. https://source.cloud.google.com/results/invocations/037ec635-4ae1-4658-84bc-d6ec0af40bba/log

This is quite weird, the same test is passing for 2.12 and failing for 2.13? Do we see same error if we build with tf-nightly?

See tests above, confirmed that this is failing for TF2.13 and not TF2.12. It's also failing for tf-nightly:
https://source.cloud.google.com/results/invocations/48c8ca19-45bd-433d-a259-b8cd64ed89bc/targets

This reverts commit 275ef5e.

qlzh727 · 2023-07-10T21:49:40Z

@qlzh727 The test is the same as always but it is failing now. Here the log.. https://source.cloud.google.com/results/invocations/037ec635-4ae1-4658-84bc-d6ec0af40bba/log

This is quite weird, the same test is passing for 2.12 and failing for 2.13? Do we see same error if we build with tf-nightly?

See tests above, confirmed that this is failing for TF2.13 and not TF2.12. It's also failing for tf-nightly: https://source.cloud.google.com/results/invocations/48c8ca19-45bd-433d-a259-b8cd64ed89bc/targets

Thanks for the confirmation, since we should have same test internally, let me see what's the status there.

seanpmorgan · 2023-07-10T21:54:55Z

@seanpmorgan Glad to hear it works with the old serialization flags. The majority of TF OSS packages have been debugged for compatibility with the new format along with backwards compatibility so that these flags are not necessary. However, due to TFAddons EOL, I have given TFAddons lower priority due to lack of bandwidth.

Just want to quickly address this comment. The upcoming EOL of TFA should not lower the priority of TFA testing. Through the years we probably caught 20-30 bugs in releases because of the number of tests we run. Checking TFA tests should help the Google team prevent breaking changes and net new bugs, not simply ensure that our builds pass so we can publish.

bhack · 2023-07-10T21:55:34Z

@qlzh727 The test is the same as always but it is failing now. Here the log.. https://source.cloud.google.com/results/invocations/037ec635-4ae1-4658-84bc-d6ec0af40bba/log

This is quite weird, the same test is passing for 2.12 and failing for 2.13? Do we see same error if we build with tf-nightly?

See tests above, confirmed that this is failing for TF2.13 and not TF2.12. It's also failing for tf-nightly: https://source.cloud.google.com/results/invocations/48c8ca19-45bd-433d-a259-b8cd64ed89bc/targets

@seanpmorgan Is it different on nightly?

RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf . Check the section C-API incompatibility at the Troubleshooting ImportError section at https://numpy.org/devdocs/user/troubleshooting-importerror.html#c-api-incompatibility for indications on how to solve this problem .

qlzh727 · 2023-07-10T21:57:31Z

I just saw the similar error from keras nightly build, which is caused by a numpy version change from tf-nightly.

bhack · 2023-07-10T22:17:15Z

It seems that with 2.13 we have an error tracing:

@tf.function
def apply_gradients():
   opt.apply_gradients([(grads, var)])
    
device.run(apply_gradients)

bhack · 2023-07-10T22:18:04Z

    def autograph_handler(*args, **kwargs):
      """Calls a converted version of original_func."""
      try:
        return api.converted_call(
            original_func,
            args,
            kwargs,
            options=converter.ConversionOptions(
                recursive=True,
                optional_features=autograph_options,
                user_requested=True,
            ))
      except Exception as e:  # pylint:disable=broad-except
        if hasattr(e, "ag_error_metadata"):
>         raise e.ag_error_metadata.to_exception(e)
E         tensorflow.python.autograph.impl.api.StagingError: in user code:
E         
E             File "/usr/local/lib/python3.9/dist-packages/keras/src/optimizers/utils.py", line 175, in _all_reduce_sum_fn  **
E                 return distribution.extended.batch_reduce_to(
E         
E             IndexError: tuple index out of range

/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/polymorphic_function/autograph_util.py:52: StagingError

Build against TF2.13

4ce8745

boring-cyborg bot added the github label May 31, 2023

Build against TF2.13

b875853

Partial fix spectral_normalization_test.py

04520fa

boring-cyborg bot added the layers label Jun 1, 2023

seanpmorgan added 2 commits June 1, 2023 15:38

Partial fix wrappers_test.py

118ec1a

Update RC2

54f1cc2

bhack added 2 commits July 3, 2023 19:49

Update 2.13 final

a6ff967

Revert "Update 2.13 final"

b5aa9dc

This reverts commit a6ff967.

bhack and others added 2 commits July 5, 2023 11:10

Update 2.13 final

bf5a9c5

Updates for serialization

6dd4f55

seanpmorgan requested a review from qlzh727 as a code owner July 6, 2023 19:18

boring-cyborg bot added activations losses labels Jul 6, 2023

Sean Morgan added 2 commits July 6, 2023 13:42

Updates for serialization

6f6de7b

flake8

25b083d

boring-cyborg bot added the optimizers label Jul 6, 2023

Sean Morgan added 4 commits July 6, 2023 14:00

Clean up

5832041

Clean up

888e267

Clean up

1c84509

Clean up

de4c241

Run GPU on 2.12

606071c

Sean Morgan added 2 commits July 10, 2023 14:12

Revert "Run GPU on 2.12"

6469afd

This reverts commit 606071c.

Debug container

f30f807

test nightly

275ef5e

Sean Morgan added 3 commits July 10, 2023 14:44

Revert "test nightly"

ff5e67a

This reverts commit 275ef5e.

Skip failing dist test on tf2.13

147d756

Use tf2.13 container

8c6e39c

seanpmorgan changed the title ~~[WIP] Build against TF2.13~~ Build against TF2.13 Jul 10, 2023

seanpmorgan changed the title ~~Build against TF2.13~~ [WIP] Build against TF2.13 Jul 10, 2023

Sean Morgan and others added 5 commits July 10, 2023 15:34

Debug

f75b7e6

Clean pytest-dist

ac21bee

Skip mirror tests for >2.13

1c553fd

Remove sort

b641597

format

c06b07b

seanpmorgan changed the title ~~[WIP] Build against TF2.13~~ Build against TF2.13 Jul 11, 2023

seanpmorgan requested a review from bhack July 11, 2023 15:07

bhack approved these changes Jul 11, 2023

View reviewed changes

seanpmorgan merged commit aa52afd into tensorflow:master Jul 11, 2023
56 checks passed

seanpmorgan deleted the build-tf213 branch July 11, 2023 15:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build against TF2.13 #2835

Build against TF2.13 #2835

seanpmorgan commented May 31, 2023 •

edited

Loading

seanpmorgan commented May 31, 2023

qlzh727 commented May 31, 2023

seanpmorgan commented May 31, 2023 •

edited

Loading

qlzh727 commented May 31, 2023

seanpmorgan commented Jun 1, 2023

qlzh727 commented Jun 1, 2023

seanpmorgan commented Jun 26, 2023

qlzh727 commented Jun 26, 2023

bhack commented Jul 3, 2023 •

edited

Loading

bhack commented Jul 10, 2023

qlzh727 commented Jul 10, 2023 •

edited

Loading

seanpmorgan commented Jul 10, 2023

seanpmorgan commented Jul 10, 2023 •

edited

Loading

seanpmorgan commented Jul 10, 2023

qlzh727 commented Jul 10, 2023

seanpmorgan commented Jul 10, 2023

bhack commented Jul 10, 2023

qlzh727 commented Jul 10, 2023

bhack commented Jul 10, 2023

bhack commented Jul 10, 2023

Build against TF2.13 #2835

Build against TF2.13 #2835

Conversation

seanpmorgan commented May 31, 2023 • edited Loading

Description

seanpmorgan commented May 31, 2023

qlzh727 commented May 31, 2023

seanpmorgan commented May 31, 2023 • edited Loading

qlzh727 commented May 31, 2023

seanpmorgan commented Jun 1, 2023

qlzh727 commented Jun 1, 2023

seanpmorgan commented Jun 26, 2023

qlzh727 commented Jun 26, 2023

bhack commented Jul 3, 2023 • edited Loading

bhack commented Jul 10, 2023

qlzh727 commented Jul 10, 2023 • edited Loading

seanpmorgan commented Jul 10, 2023

seanpmorgan commented Jul 10, 2023 • edited Loading

seanpmorgan commented Jul 10, 2023

qlzh727 commented Jul 10, 2023

seanpmorgan commented Jul 10, 2023

bhack commented Jul 10, 2023

qlzh727 commented Jul 10, 2023

bhack commented Jul 10, 2023

bhack commented Jul 10, 2023

seanpmorgan commented May 31, 2023 •

edited

Loading

seanpmorgan commented May 31, 2023 •

edited

Loading

bhack commented Jul 3, 2023 •

edited

Loading

qlzh727 commented Jul 10, 2023 •

edited

Loading

seanpmorgan commented Jul 10, 2023 •

edited

Loading