Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ROCm 2.7 Unable to find a suitable algorithm for doing forward convolution #869

Closed
av-miller opened this issue Aug 16, 2019 · 15 comments
Closed

ROCm 2.7 Unable to find a suitable algorithm for doing forward convolution #869

av-miller opened this issue Aug 16, 2019 · 15 comments
Assignees
@zjing14

Comments

@av-miller
Copy link

av-miller commented Aug 16, 2019 •
edited
Loading

I get the following error when running TensorFlow 1.14 benchmark on RX 580 4GB, ROCm 2.7.22.

MIOpen Error: /root/driver/MLOpen/src/hip/hip_build_utils.cpp:43: HIP kernel are not supported on gfx803 architecture
2019-08-16 20:20:31.307468: F tensorflow/stream_executor/rocm/rocm_dnn.cc:2707] Check failed: status == miopenStatusSuccess (7 vs. 0)Unable to find a suitable algorithm for doing forward convolution

Below is the command I ran to produce this error.
python3 tf_cnn_benchmarks.py --num_gpus=1 --batch_size=32 --model=resnet50 --variable_update=parameter_server
Below is the complete output of this command including the error.

root@andrew-MS-7B86:/root/benchmarks/scripts/tf_cnn_benchmarks# python3 tf_cnn_benchmarks.py --num_gpus=1 --batch_size=32 --model=resnet50 --variable_update=parameter_server
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING: Logging before flag parsing goes to stderr.
W0816 20:19:40.693932 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0816 20:19:40.708898 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:127: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0816 20:19:40.728541 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:695: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0816 20:19:40.728735 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3403: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-08-16 20:19:40.756264: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3199620000 Hz
2019-08-16 20:19:40.757494: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b7010 executing computations on platform Host. Devices:
2019-08-16 20:19:40.757566: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-08-16 20:19:40.760255: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libhip_hcc.so
2019-08-16 20:19:40.788528: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5621a00 executing computations on platform ROCM. Devices:
2019-08-16 20:19:40.788570: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Device 67df, AMDGPU ISA version: gfx803
2019-08-16 20:19:40.789126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1651] Found device 0 with properties: 
name: Device 67df
AMDGPU ISA: gfx803
memoryClockRate (GHz) 1.366
pciBusID 0000:26:00.0
2019-08-16 20:19:40.803606: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocblas.so
2019-08-16 20:19:40.807690: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libMIOpen.so
2019-08-16 20:19:40.808702: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocfft.so
2019-08-16 20:19:40.808992: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocrand.so
2019-08-16 20:19:40.809071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-08-16 20:19:40.809097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-16 20:19:40.809110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2019-08-16 20:19:40.809121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2019-08-16 20:19:40.809257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3540 MB memory) -> physical GPU (device: 0, name: Device 67df, pci bus id: 0000:26:00.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        BenchmarkMode.TRAIN
SingleSess:  False
Batch size:  32 global
             32.0 per device
Num batches: 100
Num epochs:  0.00
Devices:     ['/gpu:0']
Data format: NCHW
Optimizer:   sgd
Variables:   parameter_server
==========
W0816 20:19:40.811651 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2649: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0816 20:19:40.812116 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2667: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0816 20:19:40.819564 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:218: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0816 20:19:40.820210 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:225: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0816 20:19:40.829594 140471384160000 deprecation.py:323] From /root/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:126: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0816 20:19:40.831872 140471384160000 deprecation.py:506] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0816 20:19:41.061147 140471384160000 deprecation.py:323] From /root/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:250: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0816 20:19:42.811655 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:318: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0816 20:19:42.813104 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:291: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0816 20:19:42.820784 140471384160000 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0816 20:19:43.475466 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:304: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0816 20:19:43.527988 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1175: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0816 20:19:43.629981 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2009: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0816 20:19:43.632818 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2045: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0816 20:19:43.633010 140471384160000 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2102: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0816 20:19:43.789562 140471384160000 deprecation.py:323] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2157: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-08-16 20:19:44.134614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1651] Found device 0 with properties: 
name: Device 67df
AMDGPU ISA: gfx803
memoryClockRate (GHz) 1.366
pciBusID 0000:26:00.0
2019-08-16 20:19:44.134688: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocblas.so
2019-08-16 20:19:44.134700: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libMIOpen.so
2019-08-16 20:19:44.134710: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocfft.so
2019-08-16 20:19:44.134719: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocrand.so
2019-08-16 20:19:44.134765: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-08-16 20:19:44.134789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-16 20:19:44.134802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2019-08-16 20:19:44.134821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2019-08-16 20:19:44.134896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3540 MB memory) -> physical GPU (device: 0, name: Device 67df, pci bus id: 0000:26:00.0)
2019-08-16 20:19:44.537018: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0816 20:20:11.162929 140471384160000 session_manager.py:500] Running local_init_op.
I0816 20:20:11.209316 140471384160000 session_manager.py:502] Done running local_init_op.
Running warm up
2019-08-16 20:20:12.432005: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocblas.so
2019-08-16 20:20:12.446202: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libMIOpen.so
MIOpen Error: /root/driver/MLOpen/src/hip/hip_build_utils.cpp:43: HIP kernel are not supported on gfx803 architecture
2019-08-16 20:20:31.307468: F tensorflow/stream_executor/rocm/rocm_dnn.cc:2707] Check failed: status == miopenStatusSuccess (7 vs. 0)Unable to find a suitable algorithm for doing forward convolution
Fatal Python error: Aborted

Thread 0x00007fc20af43700 (most recent call first):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1429 in _call_tf_sessionrun
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1341 in _run_fn
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1356 in _do_call
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1350 in _do_run
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1173 in _run
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 950 in run
  File "/root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 818 in benchmark_one_step
  File "/root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 2316 in benchmark_with_session
  File "/root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 2184 in _benchmark_graph
  File "/root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 1975 in _benchmark_train
  File "/root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 1783 in run
  File "tf_cnn_benchmarks.py", line 56 in main
  File "/usr/local/lib/python3.5/dist-packages/absl/app.py", line 251 in _run_main
  File "/usr/local/lib/python3.5/dist-packages/absl/app.py", line 300 in run
  File "tf_cnn_benchmarks.py", line 60 in <module>
Aborted (core dumped)
@sunway513
Copy link
Contributor

@vnikiti , could you provide more information:

  1. Are you using docker container or bare metal?
  2. Can you provide the logs for the following command?
    apt --installed list | grep -E 'hip|hcc|roc|miopen|amdgpu|hsa'

@sunway513 sunway513 self-assigned this Aug 19, 2019
@virus-junior
Copy link

I have very similar problem. I am in KVM ubuntu 18.04 with pcie passtrough.

Epoch 1/5
2019-08-20 03:13:30.563070: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-08-20 03:13:30.568996: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3593000000 Hz
2019-08-20 03:13:30.569229: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3475170 executing computations on platform Host. Devices:
2019-08-20 03:13:30.569254: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-08-20 03:13:30.571064: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libhip_hcc.so
2019-08-20 03:13:30.597150: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x8229a70 executing computations on platform ROCM. Devices:
2019-08-20 03:13:30.597178: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Ellesmere [Radeon RX 470/480/570/570X/580/580X], AMDGPU ISA version: gfx803
2019-08-20 03:13:30.597595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1651] Found device 0 with properties: 
name: Ellesmere [Radeon RX 470/480/570/570X/580/580X]
AMDGPU ISA: gfx803
memoryClockRate (GHz) 1.244
pciBusID 0000:00:0b.0
2019-08-20 03:13:30.614961: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocblas.so
2019-08-20 03:13:30.621002: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libMIOpen.so
2019-08-20 03:13:30.622850: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocfft.so
2019-08-20 03:13:30.623316: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocrand.so
2019-08-20 03:13:30.623437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-08-20 03:13:30.623459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-20 03:13:30.623465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2019-08-20 03:13:30.623468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2019-08-20 03:13:30.623586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7539 MB memory) -> physical GPU (device: 0, name: Ellesmere [Radeon RX 470/480/570/570X/580/580X], pci bus id: 0000:00:0b.0)
2019-08-20 03:13:51.261565: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library librocblas.so
2019-08-20 03:13:51.275263: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libMIOpen.so
MIOpen Error: /root/driver/MLOpen/src/hip/hip_build_utils.cpp:43: HIP kernel are not supported on gfx803 architecture
2019-08-20 03:13:51.675558: F tensorflow/stream_executor/rocm/rocm_dnn.cc:2707] Check failed: status == miopenStatusSuccess (7 vs. 0)Unable to find a suitable algorithm for doing forward convolution
Aborted (core dumped)

apt --installed list | grep -E 'hip|hcc|roc|miopen|amdgpu|hsa'

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

amd64-microcode/bionic-updates,now 3.20180524.1~ubuntu0.18.04.2 amd64 [installed,automatic]
hcc/Ubuntu 16.04,now 2.7.19315 amd64 [installed,automatic]
hip_base/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_doc/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_hcc/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_samples/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hipblas/Ubuntu 16.04,now 0.12.6.202-rocm-rel-2.7-22-fd28d87 amd64 [installed,automatic]
hipsparse/Ubuntu 16.04,now 1.0.9.168-rocm-rel-2.7-22-5fea400 amd64 [installed,automatic]
hsa-amd-aqlprofile/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
hsa-ext-rocr-dev/Ubuntu 16.04,now 1.1.9-99-g835b876 amd64 [installed]
hsa-rocr-dev/Ubuntu 16.04,now 1.1.9-99-g835b876 amd64 [installed]
hsakmt-roct/Ubuntu 16.04,now 1.0.9-194-gbcfdf35 amd64 [installed,automatic]
hsakmt-roct-dev/Ubuntu 16.04,now 1.0.9-194-gbcfdf35 amd64 [installed]
intel-microcode/bionic-updates,bionic-security,now 3.20190618.0ubuntu0.18.04.1 amd64 [installed,automatic]
libcroco3/bionic,now 0.6.12-2 amd64 [installed,automatic]
libdrm-amdgpu1/bionic-updates,now 2.4.97-1ubuntu1~18.04.1 amd64 [installed,automatic]
libprocps6/bionic-updates,bionic-security,now 2:3.3.12-3ubuntu1.1 amd64 [installed,automatic]
libwebrtc-audio-processing1/bionic,now 0.3-1 amd64 [installed,automatic]
miopen-hip/Ubuntu 16.04,now 2.0.1.7405-rocm-rel-2.7-22-4e39a83 amd64 [installed]
miopengemm/Ubuntu 16.04,now 1.1.6.645-rocm-rel-2.7-22-6275a87 amd64 [installed]
procps/bionic-updates,bionic-security,now 2:3.3.12-3ubuntu1.1 amd64 [installed,automatic]
python3-ptyprocess/bionic,bionic,now 0.5.2-1 all [installed,automatic]
rccl/Ubuntu 16.04,now 2.6.0.182-rocm-rel-2.7-22-1fee6f9 amd64 [installed]
rocalution/Ubuntu 16.04,now 1.4.6.419-rocm-rel-2.7-22-5355a65 amd64 [installed,automatic]
rocblas/Ubuntu 16.04,now 2.4.0.1471-rocm-rel-2.7-22-1ac2271 amd64 [installed,automatic]
rocfft/Ubuntu 16.04,now 0.9.5.697-rocm-rel-2.7-22-ed7760e amd64 [installed,automatic]
rocm-clang-ocl/Ubuntu 16.04,now 0.4.0.35-rocm-rel-2.7-22-7ce124f amd64 [installed,automatic]
rocm-cmake/Ubuntu 16.04,now 0.3.0.93-rocm-rel-2.7-22-b727cef amd64 [installed,automatic]
rocm-dev/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-device-libs/Ubuntu 16.04,now 0.0.1 amd64 [installed]
rocm-libs/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-opencl/Ubuntu 16.04,now 1.2.0-2019080925 amd64 [installed]
rocm-opencl-dev/Ubuntu 16.04,now 1.2.0-2019080925 amd64 [installed]
rocm-profiler/Ubuntu 16.04,now 5.6.7262 amd64 [installed]
rocm-smi/Ubuntu 16.04,now 1.0.0-167-ge14d23e amd64 [installed,automatic]
rocm-utils/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocminfo/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocprim/Ubuntu 16.04,now 2.6.0.922-rocm-rel-2.7-22-3e55501 amd64 [installed,automatic]
rocprofiler-dev/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocr_debug_agent/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocrand/Ubuntu 16.04,now 2.7.0.641-rocm-rel-2.7-22-dd953aa amd64 [installed,automatic]
rocsparse/Ubuntu 16.04,now 1.1.10.573-rocm-rel-2.7-22-dc7baef amd64 [installed,automatic]
whiptail/bionic,now 0.52.20-1ubuntu1 amd64 [installed,automatic]
xserver-xorg-video-amdgpu-hwe-18.04/bionic-updates,now 19.0.1-1~18.04.1 amd64 [installed,automatic]

@sunway513
Copy link
Contributor

@zjing14 @daniellowell , could you help review the logs?
The ROCm configuration seems valid for TF1.14.1 whl package:
https://pypi.org/project/tensorflow-rocm/1.14.1/

@daniellowell
Copy link

daniellowell commented Aug 20, 2019 •
edited
Loading 

MIOpen Error: /root/driver/MLOpen/src/hip/hip_build_utils.cpp:43: HIP kernel are not supported on gfx803 architecture

Here is the problem the HIP compilation is failing ungracefully on gx803.

@zjing14 and @asroy please look into this issue.

As a workaround please try setting this blacklist variable:
export MIOPEN_DEBUG_CONV_IMPLICIT_GEMM=0

@sunway513 sunway513 assigned zjing14 and unassigned sunway513 Aug 20, 2019
@virus-junior
Copy link

Setting up blacklist variable solved the problem for me.
Thank you

@av-miller
Copy link
Author

@sunway513 I tried running benchmark inside docker and natively. I get the same error.


WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

hcc/Ubuntu 16.04,now 2.7.19315 amd64 [installed,automatic]
hip_base/Ubuntu 16.04,now 1.5.19284 amd64 [installed]
hip_doc/Ubuntu 16.04,now 1.5.19284 amd64 [installed]
hip_hcc/Ubuntu 16.04,now 1.5.19284 amd64 [installed]
hip_samples/Ubuntu 16.04,now 1.5.19284 amd64 [installed]
hipblas/Ubuntu 16.04,now 0.12.6.202-rocm-rel-2.7-22-fd28d87 amd64 [installed]
hipcub/Ubuntu 16.04,now 2.6.0.71-rocm-rel-2.7-22-5f73bda amd64 [installed]
hipsparse/Ubuntu 16.04,now 1.0.9.168-rocm-rel-2.7-22-5fea400 amd64 [installed]
hsa-amd-aqlprofile/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
hsa-ext-rocr-dev/Ubuntu 16.04,now 1.1.9-99-g835b876 amd64 [installed]
hsa-rocr-dev/Ubuntu 16.04,now 1.1.9-99-g835b876 amd64 [installed]
hsakmt-roct/Ubuntu 16.04,now 1.0.9-194-gbcfdf35 amd64 [installed]
hsakmt-roct-dev/Ubuntu 16.04,now 1.0.9-194-gbcfdf35 amd64 [installed]
libcroco3/bionic,now 0.6.12-2 amd64 [installed]
libdrm-amdgpu1/bionic-updates,now 2.4.97-1ubuntu1~18.04.1 amd64 [installed]
libpostproc54/bionic-updates,bionic-security,now 7:3.4.6-0ubuntu0.18.04.1 amd64 [installed,automatic]
libprocps6/bionic-updates,bionic-security,now 2:3.3.12-3ubuntu1.1 amd64 [installed]
libwebrtc-audio-processing1/bionic,now 0.3-1 amd64 [installed]
miopen-hip/Ubuntu 16.04,now 2.0.1.7405-rocm-rel-2.7-22-4e39a83 amd64 [installed]
miopengemm/Ubuntu 16.04,now 1.1.6.645-rocm-rel-2.7-22-6275a87 amd64 [installed]
procps/bionic-updates,bionic-security,now 2:3.3.12-3ubuntu1.1 amd64 [installed]
python3-ptyprocess/bionic,bionic,now 0.5.2-1 all [installed]
rccl/Ubuntu 16.04,now 2.6.0.182-rocm-rel-2.7-22-1fee6f9 amd64 [installed]
rocalution/Ubuntu 16.04,now 1.4.6.419-rocm-rel-2.7-22-5355a65 amd64 [installed]
rocblas/Ubuntu 16.04,now 2.4.0.1471-rocm-rel-2.7-22-1ac2271 amd64 [installed]
rocfft/Ubuntu 16.04,now 0.9.5.697-rocm-rel-2.7-22-ed7760e amd64 [installed]
rock-dkms/Ubuntu 16.04,now 2.7-22 all [installed,automatic]
rocm-clang-ocl/Ubuntu 16.04,now 0.4.0.35-rocm-rel-2.7-22-7ce124f amd64 [installed]
rocm-cmake/Ubuntu 16.04,now 0.3.0.93-rocm-rel-2.7-22-b727cef amd64 [installed]
rocm-dev/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-device-libs/Ubuntu 16.04,now 0.0.1 amd64 [installed]
rocm-dkms/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-libs/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-opencl/Ubuntu 16.04,now 1.2.0-2019080925 amd64 [installed]
rocm-opencl-dev/Ubuntu 16.04,now 1.2.0-2019080925 amd64 [installed]
rocm-profiler/Ubuntu 16.04,now 5.6.7262 amd64 [installed]
rocm-smi/Ubuntu 16.04,now 1.0.0-167-ge14d23e amd64 [installed,automatic]
rocm-utils/Ubuntu 16.04,now 2.7.22 amd64 [installed,automatic]
rocm_bandwidth_test/Ubuntu 16.04,now 1.0.0 amd64 [installed]
rocminfo/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocprim/Ubuntu 16.04,now 2.6.0.922-rocm-rel-2.7-22-3e55501 amd64 [installed,automatic]
rocprofiler-dev/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocr_debug_agent/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocrand/Ubuntu 16.04,now 2.7.0.641-rocm-rel-2.7-22-dd953aa amd64 [installed]
rocsparse/Ubuntu 16.04,now 1.1.10.573-rocm-rel-2.7-22-dc7baef amd64 [installed]
rocthrust/Ubuntu 16.04,now 2.6.0.398-rocm-rel-2.7-22-ff6992a amd64 [installed]
whiptail/bionic,now 0.52.20-1ubuntu1 amd64 [installed]
xserver-xorg-video-amdgpu-hwe-18.04/bionic-updates,now 19.0.1-1~18.04.1 amd64 [installed]

Blacklist variable did not work for me. I also tried other models and my GPU crashed and system rebooted. I am returning this GPU and going to switch to NVIDIA. I really like AMD. I love open source but the state of ROCm is very incomplete yet. Architecture-wise, we cant use ROCm on other OSs. Implementation-wise, there are prohibitively too many bugs.

@daniellowell
Copy link

This issue should be resolved in ROCm 2.8

@ianzur
Copy link

ianzur commented Aug 22, 2019

I am have run into the same issue after installing rocm and tensorflow on debian 10.

This work around did not work for me either.
export MIOPEN_DEBUG_CONV_IMPLICIT_GEMM=0

apt --installed list | grep -E 'hip|hcc|roc|miopen|amdgpu|hsa' 
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

amd64-microcode/stable,now 3.20181128.1 amd64 [installed,automatic]
hcc/Ubuntu 16.04,now 2.7.19315 amd64 [installed,automatic]
hip_base/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_doc/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_hcc/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_samples/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hipblas/Ubuntu 16.04,now 0.12.6.202-rocm-rel-2.7-22-fd28d87 amd64 [installed,automatic]
hipsparse/Ubuntu 16.04,now 1.0.9.168-rocm-rel-2.7-22-5fea400 amd64 [installed,automatic]
hsa-amd-aqlprofile/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
hsa-ext-rocr-dev/Ubuntu 16.04,now 1.1.9-99-g835b876 amd64 [installed,automatic]
hsa-rocr-dev/Ubuntu 16.04,now 1.1.9-99-g835b876 amd64 [installed,automatic]
hsakmt-roct-dev/Ubuntu 16.04,now 1.0.9-194-gbcfdf35 amd64 [installed,automatic]
hsakmt-roct/Ubuntu 16.04,now 1.0.9-194-gbcfdf35 amd64 [installed,automatic]
intel-microcode/stable,now 3.20190618.1 amd64 [installed,automatic]
libcroco3/stable,now 0.6.12-3 amd64 [installed,automatic]
libdrm-amdgpu1/stable,now 2.4.97-1 amd64 [installed,automatic]
libpostproc55/stable,now 7:4.1.4-1~deb10u1 amd64 [installed,automatic]
libprocps7/stable,now 2:3.3.15-2 amd64 [installed]
libutf8proc2/stable,now 2.3.0-1 amd64 [installed,automatic]
libwebrtc-audio-processing1/stable,now 0.3-1 amd64 [installed,automatic]
miopen-hip/Ubuntu 16.04,now 2.0.1.7405-rocm-rel-2.7-22-4e39a83 amd64 [installed]
miopengemm/Ubuntu 16.04,now 1.1.6.645-rocm-rel-2.7-22-6275a87 amd64 [installed,automatic]
procps/stable,now 2:3.3.15-2 amd64 [installed]
rccl/Ubuntu 16.04,now 2.6.0.182-rocm-rel-2.7-22-1fee6f9 amd64 [installed]
rocalution/Ubuntu 16.04,now 1.4.6.419-rocm-rel-2.7-22-5355a65 amd64 [installed,automatic]
rocblas/Ubuntu 16.04,now 2.4.0.1471-rocm-rel-2.7-22-1ac2271 amd64 [installed,automatic]
rocfft/Ubuntu 16.04,now 0.9.5.697-rocm-rel-2.7-22-ed7760e amd64 [installed,automatic]
rocm-clang-ocl/Ubuntu 16.04,now 0.4.0.35-rocm-rel-2.7-22-7ce124f amd64 [installed,automatic]
rocm-cmake/Ubuntu 16.04,now 0.3.0.93-rocm-rel-2.7-22-b727cef amd64 [installed,automatic]
rocm-dev/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-device-libs/Ubuntu 16.04,now 0.0.1 amd64 [installed,automatic]
rocm-libs/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-opencl-dev/Ubuntu 16.04,now 1.2.0-2019080925 amd64 [installed,automatic]
rocm-opencl/Ubuntu 16.04,now 1.2.0-2019080925 amd64 [installed,automatic]
rocm-smi/Ubuntu 16.04,now 1.0.0-167-ge14d23e amd64 [installed,automatic]
rocm-utils/Ubuntu 16.04,now 2.7.22 amd64 [installed,automatic]
rocminfo/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocprim/Ubuntu 16.04,now 2.6.0.922-rocm-rel-2.7-22-3e55501 amd64 [installed,automatic]
rocprofiler-dev/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocr_debug_agent/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocrand/Ubuntu 16.04,now 2.7.0.641-rocm-rel-2.7-22-dd953aa amd64 [installed,automatic]
rocsparse/Ubuntu 16.04,now 1.1.10.573-rocm-rel-2.7-22-dc7baef amd64 [installed,automatic]
whiptail/stable,now 0.52.20-8 amd64 [installed]
xserver-xorg-video-amdgpu/stable,now 18.1.99+git20190207-1 amd64 [installed,automatic]
screenfetch 
 OS: Debian 10 buster                                                                                                                                                                                             
 Kernel: x86_64 Linux 4.19.0-5-amd64
 Uptime: 2d 11h 0m
 Packages: 1760
 Shell: bash 5.0.3
 Resolution: 3840x1201
 DE: LXQt
 WM: Xfwm4
 WM Theme: Breeze
 GTK Theme: HighContrast [GTK2],  HighContrast [GTK3]
 Icon Theme: Papirus
 Font: Sans 11
 CPU: AMD Ryzen 7 1700X Eight-Core @ 16x 2.814GHz [42.0°C]
 GPU: AMD/ATI Ellesmere [Radeon RX 470/480]
 RAM: 4105MiB / 16034MiB

@anvuongb
Copy link

anvuongb commented Aug 29, 2019 •
edited
Loading

Having the same issue on my RX 580, export MIOPEN_DEBUG_CONV_IMPLICIT_GEMM=0 did not work and I don't know what to do

hcc/Ubuntu 16.04,now 2.7.19315 amd64 [installed,automatic]
hip_base/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_doc/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_hcc/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hip_samples/Ubuntu 16.04,now 1.5.19284 amd64 [installed,automatic]
hipblas/Ubuntu 16.04,now 0.12.6.202-rocm-rel-2.7-22-fd28d87 amd64 [installed,automatic]
hipsparse/Ubuntu 16.04,now 1.0.9.168-rocm-rel-2.7-22-5fea400 amd64 [installed,automatic]
hsa-amd-aqlprofile/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
hsa-ext-rocr-dev/Ubuntu 16.04,now 1.1.9-99-g835b876 amd64 [installed]
hsa-rocr-dev/Ubuntu 16.04,now 1.1.9-99-g835b876 amd64 [installed]
hsakmt-roct/Ubuntu 16.04,now 1.0.9-194-gbcfdf35 amd64 [installed,automatic]
hsakmt-roct-dev/Ubuntu 16.04,now 1.0.9-194-gbcfdf35 amd64 [installed]
intel-microcode/bionic-updates,bionic-security,now 3.20190618.0ubuntu0.18.04.1 amd64 [installed,automatic]
libcroco3/bionic,now 0.6.12-2 amd64 [installed]
libdrm-amdgpu1/bionic-updates,now 2.4.97-1ubuntu1~18.04.1 amd64 [installed]
libpostproc54/bionic-updates,bionic-security,now 7:3.4.6-0ubuntu0.18.04.1 amd64 [installed,automatic]
libprocps6/bionic-updates,now 2:3.3.12-3ubuntu1.2 amd64 [installed]
libwebrtc-audio-processing1/bionic,now 0.3-1 amd64 [installed]
miopen-hip/Ubuntu 16.04,now 2.0.1.7405-rocm-rel-2.7-22-4e39a83 amd64 [installed]
miopengemm/Ubuntu 16.04,now 1.1.6.645-rocm-rel-2.7-22-6275a87 amd64 [installed]
procps/bionic-updates,now 2:3.3.12-3ubuntu1.2 amd64 [installed]
python3-ptyprocess/bionic,bionic,now 0.5.2-1 all [installed]
rccl/Ubuntu 16.04,now 2.6.0.182-rocm-rel-2.7-22-1fee6f9 amd64 [installed]
rocalution/Ubuntu 16.04,now 1.4.6.419-rocm-rel-2.7-22-5355a65 amd64 [installed,automatic]
rocblas/Ubuntu 16.04,now 2.4.0.1471-rocm-rel-2.7-22-1ac2271 amd64 [installed,automatic]
rocfft/Ubuntu 16.04,now 0.9.5.697-rocm-rel-2.7-22-ed7760e amd64 [installed,automatic]
rock-dkms/Ubuntu 16.04,now 2.7-22 all [installed,automatic]
rocm-clang-ocl/Ubuntu 16.04,now 0.4.0.35-rocm-rel-2.7-22-7ce124f amd64 [installed,automatic]
rocm-cmake/Ubuntu 16.04,now 0.3.0.93-rocm-rel-2.7-22-b727cef amd64 [installed,automatic]
rocm-dev/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-device-libs/Ubuntu 16.04,now 0.0.1 amd64 [installed]
rocm-dkms/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-libs/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocm-opencl/Ubuntu 16.04,now 1.2.0-2019080925 amd64 [installed]
rocm-opencl-dev/Ubuntu 16.04,now 1.2.0-2019080925 amd64 [installed]
rocm-profiler/Ubuntu 16.04,now 5.6.7262 amd64 [installed]
rocm-smi/Ubuntu 16.04,now 1.0.0-167-ge14d23e amd64 [installed,automatic]
rocm-utils/Ubuntu 16.04,now 2.7.22 amd64 [installed]
rocminfo/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocprim/Ubuntu 16.04,now 2.6.0.922-rocm-rel-2.7-22-3e55501 amd64 [installed,automatic]
rocprofiler-dev/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocr_debug_agent/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]
rocrand/Ubuntu 16.04,now 2.7.0.641-rocm-rel-2.7-22-dd953aa amd64 [installed,automatic]
rocsparse/Ubuntu 16.04,now 1.1.10.573-rocm-rel-2.7-22-dc7baef amd64 [installed,automatic]
whiptail/bionic,now 0.52.20-1ubuntu1 amd64 [installed]
xserver-xorg-video-amdgpu-hwe-18.04/bionic-updates,now 19.0.1-1~18.04.1 amd64 [installed]```

@daniellowell
Copy link

@anubhavj87 @ianzur @vnikiti

Hello all, these issues may be unrelated in terms of the specific algorithm even though they are resulting in the same behavior. Please collect the logs using the environment variable: MIOPEN_LOG_LEVEL=6 and run your application.

Attach the resulting logs to your new post using the button on the GUI labeled:
"Attached files by dragging & dropping, selecting or pasting them."

@MatPoliquin
Copy link

MatPoliquin commented Sep 6, 2019 •
edited
Loading

I had the same problem with my RX580
using:
export MIOPEN_DEBUG_CONV_IMPLICIT_GEMM=0

Some notes:

  • Worked for OpenAI baselines.
  • The alex_net benchmark (tensorflow models project) still has the same problem for backward pass. Forward pass was always fine to begin with
  • cifar10 benchmark did not have this problem at all to begin with

MIOpen doc pages mentions that they are dropping support for gfx803 (RX580...) after 2.0
The current version is now 2.0.1
https://rocmsoftwareplatform.github.io/MIOpen/doc/html/releasenotes.html#

So the workaround works but I still get messages like:
<instantiation>:29:5: error: instruction not supported on this GPU v_add_u32 v[waddrbuf+xf_slot], 1, v[waddrbuf+xf_slot] ^ <instantiation>:308:3: note: while in macro instantiation .rept in_tile_width ^ <stdin>:1251:2: note: while in macro instantiation .rept pipe_depth ^ MIOpen(HIP): Warning [AmdgcnAssemble] -x assembler -target amdgcn--amdhsa -mno-code-object-v3 -Wa,-defsym,ROCM_METADATA_VERSION=4 -mcpu=gfx803 - -o /tmp/miopen-tmp-d216-0560-6d40-526a/amdgcn-asm-out-XXXXXX MIOpen(HIP): Warning [FindConvBwdWeightsAlgorithm] Find Winograd WrW failed:/root/driver/MLOpen/src/ocl/gcn_asm_utils.cpp:228: Assembly error(1)

@daniellowell
Copy link

This issue should be fixed in 2.1.
We will continue to make sure gfx8 cards functioning correctly with our software. Please feel free to submit any new issues you may come across.

@MatPoliquin
Copy link

@daniellowell Thank you!
Do you have an ETA for 2.1?

@dagamayank
Copy link

MIOpen 2.1 has been released with ROCm 2.8. Please re-open the issue if the problem still persists.

@MatPoliquin
Copy link

After Installing ROCm 3.0 I have again the same issue as before, please see my previous reply in this thread for details

@Laggger164 Laggger164 mentioned this issue Jul 22, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zjing14 zjing14

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@sunway513 @zjing14 @av-miller @MatPoliquin @dagamayank @daniellowell @virus-junior @ianzur @anvuongb