Keras file format updates #1401

rundxdi · 2024-04-25T16:51:53Z

Fixes

Addresses file format changes associated with updates to tensorflow and keras mentioned in issue #1369

Summary/Motivation:

Tensorflow and keras version updates changed how they save/load files. This PR updates the keras_surrogate.py file to accommodate those changes.

Changes proposed in this PR:

Update keras_surrogate.py to save new .keras file format
Update test_keras_surrogate.py with new versions of saved models and accounting for new save/load methods

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the license terms described in the LICENSE.txt file at the top level of this directory.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

andrewlee94 · 2024-04-25T17:55:25Z

idaes/core/surrogate/keras_surrogate.py

@@ -263,7 +263,7 @@ def save_to_folder(self, keras_folder_name):
              The name of the folder to contain the Keras model and additional
              IDAES metadata
        """
-        self._keras_model.save(keras_folder_name)
+        self._keras_model.save(os.path.join(keras_folder_name, "idaes_keras_model.keras"))


Should the file name be hard-coded here, or should we make this an input from the user (either optional or required).

I support letting users pass a string for the file name, so something like

Suggested change

self._keras_model.save(os.path.join(keras_folder_name, "idaes_keras_model.keras"))

self._keras_model.save(os.path.join(keras_folder_name, keras_file_name + ".keras"))

where keras_file_name is an input to the save method. I don't know if we still need to specify the folder name, or if a single string for the entire path is sufficient.

The folder name is used to save/load some model weights and some idaes specific information separate from the filename. But I think the file name should be rename-able as you say -- making that change now

bpaul4 · 2024-04-25T18:04:45Z

@rundxdi thank you for opening this. Once this is ready to test, it would be good to undo the temporary fix implemented in #1373 to ensure the latest version of TensorFlow is used in the tests.

ksbeattie · 2024-05-02T18:35:37Z

@rundxdi any progress on this?

rundxdi · 2024-05-04T04:50:26Z

@rundxdi any progress on this?

Nothing this week -- still trying to figure out plotting tests.

rundxdi · 2024-05-09T01:00:19Z

I have two remaining issues:

Python 3.8 doesn't like test_keras_surrogate.py. Tests all produce failure statuses looking like:

Exception encountered: Unrecognized keyword arguments: ['batch_shape']

These tests all pass on Python 3.9+.

The test_sofc_surrogates.py tests rely on a saved Keras surrogate that cannot be loaded with Keras v3+ (and thus cannot be loaded with Tensorflow 2.16.1). I've found no obvious way to generate an equivalent surrogate in Keras v3 equivalent surrogate; their suggested workaround doesn't work.

lbianchi-lbl · 2024-05-09T19:04:55Z

1. Python 3.8 doesn't like test_keras_surrogate.py.  Tests all produce failure statuses looking like:
Exception encountered: Unrecognized keyword arguments: ['batch_shape']

These tests all pass on Python 3.9+.

My guess is that this is due to NumPy (and consequently I imagine most of the downstream projects depending on it) having dropped support for Python 3.8 a while back. So, when installing on Python 3.8, an older version of the package (the latest compatible) gets installed, which I assume doesn't support the same arguments as the current version.

We're considering removing support for (i.e. stopping testing with) Python 3.8. In the meantime, you should be able to use pytest.mark.skipif() to skip the failing tests if they're being run: https://docs.pytest.org/en/latest/how-to/skipping.html#id1

2. The test_sofc_surrogates.py tests rely on a saved Keras surrogate that cannot be loaded with Keras v3+ (and thus cannot be loaded with Tensorflow 2.16.1).  I've found no obvious way to generate an equivalent surrogate in Keras v3 equivalent surrogate; their suggested workaround doesn't work.

With the disclaimer that I don't have any significant hands-on experience using Keras, I'm at least aware of there being limited or no support for cross-version compatibility (i.e. a model serialized with Keras version X won't work when you try to deserialize it using version Y). I'm not sure how much the new (current) serialization format changes things. If this is the case, I guess the solution would be to regenerate the file used in the tests using the current version. I'm not sure if this addresses the original issue, though.

andrewlee94 · 2024-05-17T14:19:29Z

As mentioned in the dev call, regarding the test failure in the power generation models you should:

Mark the failing tests with pytest.mark.xfail
In the failing code, add a deprecation warning indicating the tests for this code have started failing and the code will be removed no earlier than August if it is not fixed.
Open an issue to fix this and assign it to the code owner(s), and put this on the August release board as a High priority so we track it.

ksbeattie · 2024-05-23T18:41:47Z

@rundxdi we expect to cut the May release next week, so if this PR is to be included, it needs to be merged very soon.

ksbeattie · 2024-06-06T18:48:59Z

@rundxdi, this missed the May release, now on the Aug release board.

bpaul4

Thank you @rundxdi for investigating and implementing the fix. The changes to the Keras/OMLT calls and file writing look good.

I have a couple questions. First, the models previously import from a folder containing saved_model.pb, keras_metadata.pb, and idaes_info.json files and a variables/ folder containing variables.index and variables.data-00000-of-00001 files. Are these still necessary, or can we remove them in favor of the .keras files? I've seen in my research into Keras 3 that it now doesn't support SavedModel files and includes a read-only TFSM format instead.

Second, do we want to look into the Python 3.8 failures? Since it isn't occurring in the newer Python environments, it may just be a compatibility issue that's best skipping in that one environment.

bpaul4 · 2024-07-09T14:05:56Z

@rundxdi when you have a moment, could you please provide a status update? What is required to finalize this?

rundxdi · 2024-07-17T01:28:49Z

@rundxdi when you have a moment, could you please provide a status update? What is required to finalize this?

Hi @bpaul4 -- we can safely remove any of the non .keras files. The Python 3.8 tests should be skipped at this point.

It should be ready to finalize/review for finalization.

codecov-commenter · 2024-07-17T02:05:36Z

Codecov Report

Attention: Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 76.36%. Comparing base (c2825ca) to head (0c2775e).
Report is 1 commits behind head on main.

Files	Patch %	Lines
idaes/core/surrogate/keras_surrogate.py	77.77%	2 Missing ⚠️
...generation/properties/sofc/sofc_keras_surrogate.py	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1401      +/-   ##
==========================================
- Coverage   76.38%   76.36%   -0.03%     
==========================================
  Files         394      394              
  Lines       65121    65126       +5     
  Branches    14427    14426       -1     
==========================================
- Hits        49745    49732      -13     
- Misses      12813    12833      +20     
+ Partials     2563     2561       -2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bpaul4

I think this looks good now, thanks @rundxdi!

lbianchi-lbl · 2024-07-25T18:29:31Z

Before merging this, we should remove Python 3.8 from the CI and see if the tests pass without needing to mark them as xfailing.

…file_format_updates

bpaul4 · 2024-08-16T14:26:44Z

@andrewlee94 it looks like all of your prior review comments have been addressed.

I removed the xfail markers from the keras surrogate tests and everything looks good.

andrewlee94

@bpaul4 I was waiting for the Python 3.8 issue to be merged to make sure the tests passed. I have just one minor request.

andrewlee94 · 2024-08-16T14:34:05Z

idaes/models_extra/power_generation/properties/tests/test_sofc_surrogates.py

@@ -53,6 +53,7 @@ def build_rom():


 @pytest.mark.unit
+@pytest.mark.xfail


Could you add an inline comment to explain why this is xfailed. I know this is mentioned in the deprecation warning in the code, but having comments here will help reinforce that this issue needs to be fixed.

I'll add the comments now. @AlexNoring is aware of the SOFC surrogate issue and will continue the discussion here: #1431

bpaul4 · 2024-08-16T15:15:51Z

@andrewlee94 how would you suggest handling the failing example integration tests? The failures are resolved as part of IDAES/examples#128.

andrewlee94 · 2024-08-16T15:33:50Z

@bpaul4 I would say we need to get that examples PR merged first.

lbianchi-lbl

Mostly minor comments, questions, and suggestions.

lbianchi-lbl · 2024-08-16T14:58:00Z

idaes/models_extra/power_generation/properties/tests/test_sofc_surrogates.py

@@ -53,6 +53,7 @@ def build_rom():


 @pytest.mark.unit
+@pytest.mark.xfail  # test xfailed due to out-of-date, incompatible Keras surrogates


Suggestion: use the reason kwarg instead of a comment allows the message to be displayed when running the tests, increasing its visibility:

Suggested change

@pytest.mark.xfail # test xfailed due to out-of-date, incompatible Keras surrogates

@pytest.mark.xfail(reason="test xfailed due to out-of-date, incompatible Keras surrogates")

lbianchi-lbl · 2024-08-16T14:58:31Z

idaes/models_extra/power_generation/properties/tests/test_sofc_surrogates.py

@@ -78,6 +79,7 @@ def test_basic_build(build_rom):

 @pytest.mark.skipif(solver is None, reason="Solver not available")
 @pytest.mark.component
+@pytest.mark.xfail  # test xfailed due to out-of-date, incompatible Keras surrogates


Suggestion: use the reason kwarg instead of a comment allows the message to be displayed when running the tests, increasing its visibility:

Suggested change

@pytest.mark.xfail # test xfailed due to out-of-date, incompatible Keras surrogates

@pytest.mark.xfail(reason="test xfailed due to out-of-date, incompatible Keras surrogates")

lbianchi-lbl · 2024-08-16T14:58:47Z

idaes/models_extra/power_generation/properties/tests/test_sofc_surrogates.py

@@ -97,9 +99,20 @@ def test_initialize(build_rom):

 @pytest.mark.skipif(solver is None, reason="Solver not available")
 @pytest.mark.component
+@pytest.mark.xfail  # test xfailed due to out-of-date, incompatible Keras surrogates


Suggestion: use the reason kwarg instead of a comment allows the message to be displayed when running the tests, increasing its visibility:

Suggested change

@pytest.mark.xfail # test xfailed due to out-of-date, incompatible Keras surrogates

@pytest.mark.xfail(reason="test xfailed due to out-of-date, incompatible Keras surrogates")

lbianchi-lbl · 2024-08-16T15:02:02Z

idaes/models_extra/power_generation/properties/sofc/sofc_keras_surrogate.py

+msg = "Tests for sofc_keras_surrogate.py have started failing.  The code will be removed no early than August if it is not fixed."
+deprecation_warning(msg=msg, logger=_log, version="2.5.0", remove_in="3.0.0")


Suggested change

msg = "Tests for sofc_keras_surrogate.py have started failing. The code will be removed no early than August if it is not fixed."

deprecation_warning(msg=msg, logger=_log, version="2.5.0", remove_in="3.0.0")

msg = "Tests for sofc_keras_surrogate.py have started failing following Tensorflow/Keras version updates (see IDAES/idaes-pse#1401). The code will be removed no earlier than August 2024 if it is not fixed."

deprecation_warning(msg=msg, logger=_log, version="2.5.0", remove_in="3.0.0")

lbianchi-lbl · 2024-08-16T15:31:24Z

setup.py

@@ -93,7 +93,7 @@ class ExtraDependencies:
    ]
    omlt = [
        "omlt==1.1",  # fix the version for now as package evolves
-        'tensorflow < 2.16.1 ; python_version < "3.12"',
+        "tensorflow",


Question: given the changes in this PR, would it make sense to require a minimum version of 2.16.1 here?

I believe that would be appropriate. although removing the tag will automatically search for the latest version. Pinning a minimum version to support Keras 3 would be a good safety net.

lbianchi-lbl · 2024-08-16T17:30:21Z

idaes/core/surrogate/keras_surrogate.py

+        self._keras_model.save(
+            os.path.join(keras_folder_name, keras_model_name + ".keras")
+        )


Suggestion: since Keras seems to support pathlib.Path objects now, this could be rewritten to be slightly more concise/readable (requires replacing import os.path with from pathlib import Path at the top of the file):

Suggested change

self._keras_model.save(

os.path.join(keras_folder_name, keras_model_name + ".keras")

)

self._keras_model.save(

Path(keras_folder_name, keras_model_name).with_suffix(".keras")

)

lbianchi-lbl · 2024-08-16T17:30:54Z

idaes/core/surrogate/keras_surrogate.py

+
+        keras_model = keras.models.load_model(
+            os.path.join(keras_folder_name, keras_model_name + ".keras")
+        )
+


See previous comments for using pathlib.Path instead of os.path.

lbianchi-lbl · 2024-08-16T17:31:19Z

idaes/core/surrogate/keras_surrogate.py

-    nn.save_weights(os.path.join(path, "{}.h5".format(name)))
+    nn.save(os.path.join(path, "{}.keras".format(name)))
+    nn.save_weights(os.path.join(path, "{}.weights.h5".format(name)))


 def load_keras_json_hd5(path, name):
-    with open(os.path.join(path, "{}.json".format(name)), "r") as json_file:
-        json_model = json_file.read()
-        nn = keras.models.model_from_json(json_model)
-    nn.load_weights(os.path.join(path, "{}.h5".format(name)))
+    nn = keras.models.load_model(os.path.join(path, "{}.keras".format(name)))
+    nn.load_weights(os.path.join(path, "{}.weights.h5".format(name)))


See comments above for using pathlib.Path instead of os.path.

* update save, load syntax * regenerate outdated notebooks * generate new model files * remove old keras files * Test using IDAES/idaes-pse#1401 * Update version constraint for TensorFlow * Address Python 3.8 failures due to Tensorflow 2.16.1 not being available * Remove exclusion for Python 3.12 for Tensorflow * Remove Python 3.8 support * Restore installing idaes-pse from main branch --------- Co-authored-by: Ludovico Bianchi <lbianchi@lbl.gov>

lbianchi-lbl · 2024-08-16T19:01:43Z

There might be another test that needs to be skipped/xfailed as it's currently causing a failure in the "user install" integration checks: https://github.com/IDAES/idaes-pse/actions/runs/10424410674/job/28873157657?pr=1401#step:6:538

andrewlee94 · 2024-08-16T19:07:31Z

@lbianchi-lbl I think we should look into that one a bit more first, and fix it if we can. Being that is in the core code, we should at least know why it fails.

lbianchi-lbl · 2024-08-16T19:14:42Z

@lbianchi-lbl I think we should look into that one a bit more first, and fix it if we can. Being that is in the core code, we should at least know why it fails.

The fact that it's only failing in the user-mode check, plus the exception raised, seem to point towards the file not being included in the non-developer installation:

rundxdi added 2 commits April 18, 2024 14:27

omlt import fix

473a118

updated main surrogate code; baseline tests fixed with re-run NNs

890dd47

rundxdi requested review from andrewlee94 and bpaul4 as code owners April 25, 2024 16:51

rundxdi mentioned this pull request Apr 25, 2024

Resolve Keras File Format Issues #1369

Closed

andrewlee94 reviewed Apr 25, 2024

View reviewed changes

ksbeattie assigned rundxdi Apr 25, 2024

ksbeattie added the Priority:High High Priority Issue or PR label Apr 25, 2024

rundxdi added 2 commits May 8, 2024 11:05

black and filename option

ca0a0d7

remove version restriction on tensorflow

a0f947e

rundxdi requested review from lbianchi-lbl and ksbeattie as code owners May 8, 2024 17:31

rundxdi added 2 commits May 8, 2024 12:05

removing unused vars

76bf172

testing model format fixes

bd446de

rundxdi requested a review from AlexNoring as a code owner May 8, 2024 20:28

rundxdi added 4 commits May 8, 2024 15:22

updating additional tests

bde687f

rerun black

a53970c

setup.py fixes

fa96615

black on setup.py

9868c04

ksbeattie linked an issue May 23, 2024 that may be closed by this pull request

Resolve Keras File Format Issues #1369

Closed

rundxdi added 2 commits June 13, 2024 10:42

expected test failures

7ebd238

rerun black

17016a3

bpaul4 reviewed Jun 17, 2024

View reviewed changes

Merge branch 'main' into keras_file_format_updates

7832c07

bpaul4 added 2 commits July 17, 2024 08:58

remove old keras files

3f40a24

remove more old keras files

03d5438

bpaul4 approved these changes Jul 17, 2024

View reviewed changes

bpaul4 mentioned this pull request Jul 17, 2024

Keras file format updates IDAES/examples#128

Merged

lbianchi-lbl added a commit to bpaul4/examples that referenced this pull request Jul 18, 2024

Test using IDAES/idaes-pse#1401

37a7952

lbianchi-lbl mentioned this pull request Aug 1, 2024

Drop support for Python 3.8 #1462

Closed

bpaul4 added 2 commits August 16, 2024 06:46

Merge branch 'main' of https://github.com/IDAES/idaes-pse into keras_…

98adbe6

…file_format_updates

try removing xfails from keras surrogate tests

7e4e87e

bpaul4 requested a review from andrewlee94 August 16, 2024 14:25

andrewlee94 reviewed Aug 16, 2024

View reviewed changes

add comments explaining xfailing sofc surrogate tests

eb39315

andrewlee94 approved these changes Aug 16, 2024

View reviewed changes

Merge branch 'main' into keras_file_format_updates

efbe7c7

lbianchi-lbl reviewed Aug 16, 2024

View reviewed changes

Try removing workaround for TensorFlow failures for Python 3.11

1113057

Add .keras suffix to package_data

0c2775e

lbianchi-lbl merged commit 9dce004 into IDAES:main Aug 19, 2024
37 checks passed

	self._keras_model.save(os.path.join(keras_folder_name, "idaes_keras_model.keras"))
	self._keras_model.save(os.path.join(keras_folder_name, keras_file_name + ".keras"))

		@@ -53,6 +53,7 @@ def build_rom():


		@pytest.mark.unit
		@pytest.mark.xfail

		@@ -53,6 +53,7 @@ def build_rom():


		@pytest.mark.unit
		@pytest.mark.xfail # test xfailed due to out-of-date, incompatible Keras surrogates

	@pytest.mark.xfail # test xfailed due to out-of-date, incompatible Keras surrogates
	@pytest.mark.xfail(reason="test xfailed due to out-of-date, incompatible Keras surrogates")

		msg = "Tests for sofc_keras_surrogate.py have started failing. The code will be removed no early than August if it is not fixed."
		deprecation_warning(msg=msg, logger=_log, version="2.5.0", remove_in="3.0.0")

Keras file format updates #1401

Keras file format updates #1401

Conversation

rundxdi commented Apr 25, 2024

Fixes

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpaul4 commented Apr 25, 2024

ksbeattie commented May 2, 2024

rundxdi commented May 4, 2024

rundxdi commented May 9, 2024

lbianchi-lbl commented May 9, 2024

andrewlee94 commented May 17, 2024

ksbeattie commented May 23, 2024

ksbeattie commented Jun 6, 2024 • edited Loading

bpaul4 left a comment

Choose a reason for hiding this comment

bpaul4 commented Jul 9, 2024

rundxdi commented Jul 17, 2024

codecov-commenter commented Jul 17, 2024 • edited Loading

Codecov Report

bpaul4 left a comment

Choose a reason for hiding this comment

lbianchi-lbl commented Jul 25, 2024

bpaul4 commented Aug 16, 2024

andrewlee94 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpaul4 commented Aug 16, 2024

andrewlee94 commented Aug 16, 2024

lbianchi-lbl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpaul4 Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbianchi-lbl commented Aug 16, 2024 • edited Loading

andrewlee94 commented Aug 16, 2024

lbianchi-lbl commented Aug 16, 2024

ksbeattie commented Jun 6, 2024 •

edited

Loading

codecov-commenter commented Jul 17, 2024 •

edited

Loading

bpaul4 Aug 16, 2024 •

edited

Loading

lbianchi-lbl commented Aug 16, 2024 •

edited

Loading