Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Breaking change in Keras 2.4: AttributeError: 'ModelCheckpoint' object has no attribute '_implements_train_batch_hooks' #35

Closed
levaphenyl opened this issue Sep 8, 2020 · 0 comments · Fixed by #38
Closed

Breaking change in Keras 2.4: AttributeError: 'ModelCheckpoint' object has no attribute '_implements_train_batch_hooks' #35

levaphenyl opened this issue Sep 8, 2020 · 0 comments · Fixed by #38

Comments

@levaphenyl
Copy link
Collaborator

levaphenyl commented Sep 8, 2020 •
edited
Loading

Since keras-2.4 and tensorflow-2.3, all import keras statements must be replaced by import tensorflow.keras as stated on SO. This is due to an internal change in the library that break earlier installations. See also here.

Two solution here:

  1. We specify keras>=2.2.4,<2.4 and tensorflow>=2.0,<2.3 in the setup.py and conda requirements. However, this only increases the technical debt.
  2. We change all keras import and update the aforementioned requirements.

I can handle the PR and go for solution 2., but I would like your opinion first @PMeira :).

How to reproduce

Libraries:

python-3.8.5
tensorflow-gpu-2.3.0
keras-2.4.2
nilmtk-0.4.2
nilmtk-contrib at origin/master

Code:

import nilmtk.api
import nilmtk.disaggregate
import nilmtk_contrib


test_GPU = {
    "power": { "mains": [ "active" ], "appliance": [ "active" ] },
    "sample_rate": 10,
    "appliances": [ "kettle", ],
    "artificial_aggregate": False,
    "chunk_size": 2**16,
    "DROP_ALL_NANS": True,
    "methods": {
        "Seq2Point": nilmtk_contrib.disaggregate.Seq2Point({
            "batch_size": 1024,
            "chunk_wise_training": True,
        }),
    },
    "train": {
        "datasets": {
            "UK-DALE": {
                "path": "datasets/UK-DALE/ukdale2017.h5",
                "buildings": {
                    1: { "start_time": "2013-03-18", "end_time": "2014-12-01" },
                    2: { "start_time": "2013-02-17", "end_time": "2013-08-05" },
                },
            },
        },
    },
    "test": {
        "datasets": {
            "UK-DALE": {
                "path": "datasets/UK-DALE/ukdale2017.h5",
                "buildings": {
                    5: { "start_time": "2014-06-29", "end_time": "2014-11-13" },
                },
            },
        },
        "metrics": [ "f1_score", "mae", ],
    },
}


if __name__ == "__main__":
    res_test_GPU = nilmtk.api.API(test_GPU)

Traceback:

2020-09-08 10:14:23.070885: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Using TensorFlow backend.
Started training for  Seq2Point
Chunk wise training for  Seq2Point
Loading data for  UK-DALE  dataset
Loading building ...  1
ElecMeter(instance=54, building=1, dataset='UK-DALE', site_meter, appliances=[Appliance(type='immersion heater', instance=1), Appliance(type='water pump', instance=1), Appliance(type='security alarm', instance=1), Appliance(type='fan', instance=2), Appliance(type='drill', instance=1), Appliance(type='laptop computer', instance=2)])
Starting enumeration..........
Dropping missing values
{'kettle': {'mean': 15.483873, 'std': 182.19482}}
...............Seq2Point partial_fit running...............
First model training for  kettle
2020-09-08 10:15:06.392509: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-09-08 10:15:08.067095: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.067391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.56GHz coreCount: 16 deviceMemorySize: 3.82GiB deviceMemoryBandwidth: 119.24GiB/s
2020-09-08 10:15:08.067461: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-09-08 10:15:08.068738: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-09-08 10:15:08.069875: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-09-08 10:15:08.070148: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-09-08 10:15:08.071229: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-09-08 10:15:08.071778: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-09-08 10:15:08.073978: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-09-08 10:15:08.074065: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.074317: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.074506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-09-08 10:15:08.074685: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-09-08 10:15:08.092661: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2601325000 Hz
2020-09-08 10:15:08.092880: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d467d64150 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-08 10:15:08.092893: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-08 10:15:08.162948: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.163251: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d467df0170 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-08 10:15:08.163265: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1650, Compute Capability 7.5
2020-09-08 10:15:08.163449: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.163697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.56GHz coreCount: 16 deviceMemorySize: 3.82GiB deviceMemoryBandwidth: 119.24GiB/s
2020-09-08 10:15:08.163733: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-09-08 10:15:08.163774: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-09-08 10:15:08.163790: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-09-08 10:15:08.163821: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-09-08 10:15:08.163835: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-09-08 10:15:08.163870: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-09-08 10:15:08.163886: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-09-08 10:15:08.164028: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.164291: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.164518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-09-08 10:15:08.164545: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-09-08 10:15:08.529420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-08 10:15:08.529470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-09-08 10:15:08.529476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-09-08 10:15:08.529708: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.530058: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-08 10:15:08.530333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5)
Traceback (most recent call last):
  File "2020-08-rectangular_regression/benchmark.py", line 111, in <module>
    res_test_GPU = nilmtk.api.API(test_GPU)
  File ".venv/lib/python3.8/site-packages/nilmtk-0.4.0.dev1+git.236b169-py3.8.egg/nilmtk/api.py", line 45, in __init__
    self.experiment()
  File ".venv/lib/python3.8/site-packages/nilmtk-0.4.0.dev1+git.236b169-py3.8.egg/nilmtk/api.py", line 80, in experiment
    self.train_chunk_wise(clf,d) 
  File ".venv/lib/python3.8/site-packages/nilmtk-0.4.0.dev1+git.236b169-py3.8.egg/nilmtk/api.py", line 152, in train_chunk_wise
    clf.partial_fit(self.train_mains,self.train_submeters)
  File ".venv/lib/python3.8/site-packages/nilmtk_contrib-0.1.2.dev1+git.de38dab-py3.8.egg/nilmtk_contrib/disaggregate/seq2point.py", line 88, in partial_fit
  File ".venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File ".venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1067, in fit
    callbacks = callbacks_module.CallbackList(
  File ".venv/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 234, in __init__
    self._should_call_train_batch_hooks = any(
  File ".venv/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 235, in <genexpr>
    cb._implements_train_batch_hooks() for cb in self.callbacks)
AttributeError: 'ModelCheckpoint' object has no attribute '_implements_train_batch_hooks'
Closing remaining open files:datasets/UK-DALE/ukdale2017.h5...done
levaphenyl added a commit to levaphenyl/nilmtk-contrib that referenced this issue Sep 14, 2020
@levaphenyl
chore: change all Keras imports by tensorflow.keras imports to be com…
f42637a 
…patible with Keras-2.4 and tensorflow-2.3.

This does not break previous requirements or installations as Keras is part of TF since tensorflow-2.0.
Fixes: nilmtk#35
@levaphenyl levaphenyl mentioned this issue Sep 14, 2020
Merged
levaphenyl added a commit that referenced this issue Dec 2, 2020
@levaphenyl
chore: change all Keras imports by tensorflow.keras imports to be com…
161bf0d 
…patible with Keras-2.4 and tensorflow-2.3.

This does not break previous requirements or installations as Keras is part of TF since tensorflow-2.0.
Fixes: #35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Chore: Breaking change in Keras and TF API. levaphenyl/nilmtk-contrib
1 participant
@levaphenyl