Code

Enable --fast_layer_norm for ROCm#94

Code

Open

hubertlu-tw wants to merge 23 commits intomasterROCm/apex:masterfrom fastlayernormROCm/apex:fastlayernormCopy head branch name to clipboard

+942-321

Commits

Commits on Sep 1, 2022

• add build for --fast_layer_norm

  jeffdaily

  committed
  c4a4ee5

  Copy full SHA for c4a4ee5

Commits on Sep 6, 2022

• use HIP bfloat16 header

  jeffdaily

  committed
  c1e3a72

  Copy full SHA for c1e3a72

• missing <functional> header

  jeffdaily

  committed
  09d7be8

  Copy full SHA for 09d7be8

• warp size considerations, TODOs

  jeffdaily

  committed
  fc13459

  Copy full SHA for fc13459

• work around some compiler errors

  jeffdaily

  committed
  1228e3c

  Copy full SHA for 1228e3c

Commits on Sep 7, 2022

• it finally compiles

  jeffdaily

  committed
  f2399d6

  Copy full SHA for f2399d6

Commits on Sep 9, 2022

• Manually hipify cudaLaunchCooperativeKernel in --fast_layer_norm extension

  

  hubertlu-tw

  committed
  b30d3ba

  Copy full SHA for b30d3ba

• Update setup.py for --fast_layer_norm extension

  

  hubertlu-tw

  committed
  103e128

  Copy full SHA for 103e128

• Merge remote-tracking branch 'origin/master' into fastlayernorm

  

  hubertlu-tw

  committed
  f2908b7

  Copy full SHA for f2908b7

Commits on Feb 20, 2023

• Merge remote-tracking branch 'origin/master' into fastlayernorm

  aspanday

  committed
  d0e7073

  Copy full SHA for d0e7073

Commits on Mar 14, 2023

• CTAS_PER_ROW > 1 requires reduction that seems to be making the accuracy tests to fail in fwd and bwd pass.

  Show description for 637f2e1
  aspanday

  committed
  637f2e1

  Copy full SHA for 637f2e1

• retaining the previous CTAS_PER_ROW values for fwd pass.

  aspanday

  committed
  03002d0

  Copy full SHA for 03002d0

• Merge branch 'fastlayernorm' of https://github.com/ROCmSoftwarePlatform/apex into fastlayernorm

  aspanday

  committed
  a4e4d78

  Copy full SHA for a4e4d78

Commits on May 3, 2023

• removing checks for is_cuda. This interferes with the unittest. May need to revert this change later if needed.

  aspanday

  committed
  bdd5cdb

  Copy full SHA for bdd5cdb

• adding appropriate assert messages. This reduces #warnings as well as helps with decoding impact of kernel parameters.

  aspanday

  committed
  b78abf9

  Copy full SHA for b78abf9

• adding appropriate assert messages. This reduces #warnings as well as helps with decoding impact of kernel parameters.

  aspanday

  committed
  f47022d

  Copy full SHA for f47022d

• adding appropriate assert messages. This reduces #warnings as well as helps with decoding impact of kernel parameters.

  aspanday

  committed
  12cb79f

  Copy full SHA for 12cb79f

• adding appropriate assert messages. This reduces #warnings as well as helps with decoding impact of kernel parameters.

  aspanday

  committed
  f973057

  Copy full SHA for f973057

• adding hipFuncSetAttrbute when IS_ROCM is True

  aspanday

  committed
  37feda6

  Copy full SHA for 37feda6

• Updating REGISTER_BWD_KERNEL macros for various hidden_sizes to get reported performances.

  aspanday

  committed
  5777035

  Copy full SHA for 5777035

• Added option to pass normalized_shape for fast_layer_norm same as fused_layer_norm.

  Show description for 6d58ce1
  aspanday

  committed
  6d58ce1

  Copy full SHA for 6d58ce1

• Updated fast_layer_norm test to call fused_layer_norm test when hidden_sizes > 12K since fast_layer_norm patch calls fused_layer_norm for bwd when hideen_Sizes > 12K

  aspanday

  committed
  ae9de3e

  Copy full SHA for ae9de3e

• Merge branch 'master' into fastlayernorm

  hubertlu-tw

  authored
  fb79a52

  Copy full SHA for fb79a52