Bert Example #440

HCY-11 · 2022-11-09T22:10:17Z

Example using convert_to_analog to run BERT Transformer on SQuAD task.

Pull from IBM master

maljoras

@HCY-11 Shaping up greatly! Seems only that the evaluation loop is still missing.

examples/24_bert_on_squad.py

Signed-off-by: Henry Ye <yehenry11@gmail.com>

* Update travis test stages * Update travis cibuildwheel stages * Update version specifiers * Fix numpy typing issues * Update CHANGELOG * Fix typo from my way future self Signed-off-by: Henry Ye <yehenry11@gmail.com>

* Added multiple RNN cell/layer functionality * Finished multiple rnn functionality * Modified rnn example and finished rnn testing * init * examples cuda fixes * skip large conversion * change log * pycodestyle Co-authored-by: Malte Rasch <malte.rasch@ibm.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

* Add security policy * Add test policy to contributor guidelines Signed-off-by: Henry Ye <yehenry11@gmail.com>

* Initial commit * Updated SelfDefineDevice * Fixed some files * Reformat files * Fixed visualization bug * Polishing: added fitting to device data * Cuda version * Change log * Test fix Co-authored-by: Malte Rasch <malte.rasch@ibm.com> Co-authored-by: Kaoutar El Maghraoui <kaoutar.elmaghraoui@gmail.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

Signed-off-by: Henry Ye <yehenry11@gmail.com>

* Added a fix in .travis.yml file to the last build error that is caused by the error: "The repository 'https://public.dhe.ibm.com/software/server/POWER/Linux/toolchain/at/ubuntu bionic Release' does not have a Release file." The fix adds "apt-get -qq install -y ca-certificates" before any other apt-get commands. Apparently some root certificate changed, so new CA certs are required to pull the release information. The error message is a bit misleading here. the Release file is there, apt-get just can't retrieve it due to cert validation failures. * Update .travis.yml * Update .travis.yml * * Upgraded sphinx from 3.1.2 to 4.0.2 * Added documentation about the new RNN layers * Added api documentation for the new RNN layers * Improved the api python doc for RNN types to explicitely state the various types supported * Added instructions on how to run the examples in the examples readme file * Additional changes to the api reference: added linear_mapped Added readme to the tests folder * Additional changes to the documentation. Added a hw-aware training section. * Additonal fixes and restructuring of the documentation * Minor changes to address Malte's review. Co-authored-by: Kaoutar El Maghraoui <kaoutar@macbook-pro.mynetworksettings.com> Co-authored-by: Malte J. Rasch <17587387+maljoras@users.noreply.github.com> Co-authored-by: Kaoutar El Maghraoui <kaoutar@kaoutars-mbp.mynetworksettings.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

*Total -- 5,055.66kb -> 4,171.28kb (17.49%) /docs/img/momentum_sgd.png -- 22.45kb -> 11.36kb (49.41%) /notebooks/processing-unit-and-conventional-memory.png -- 11.62kb -> 6.34kb (45.44%) /notebooks/processing-unit-and-computional-memory.png -- 8.70kb -> 4.75kb (45.41%) /notebooks/pcm-array.png -- 18.13kb -> 11.16kb (38.44%) /docs/img/analog_dnn_training.png -- 567.95kb -> 388.39kb (31.62%) /docs/img/reram_measurements.png -- 467.18kb -> 320.84kb (31.32%) /docs/img/pcm_drift_model.png -- 285.72kb -> 213.15kb (25.4%) /notebooks/LeNet5_animation.png -- 871.90kb -> 683.96kb (21.56%) /notebooks/pcm_rpu_unit.png -- 85.72kb -> 68.11kb (20.54%) /docs/img/analog_non_idealities.png -- 48.28kb -> 38.84kb (19.56%) /docs/img/analog_mac_time.png -- 61.86kb -> 50.58kb (18.22%) /docs/img/toolkit_quantization.png -- 76.87kb -> 63.94kb (16.82%) /docs/img/hwa_added_noise.png -- 205.13kb -> 173.25kb (15.54%) /docs/img/parallel_update.png -- 73.71kb -> 62.54kb (15.16%) /docs/img/analog_ai_hw.png -- 147.70kb -> 129.50kb (12.33%) /notebooks/analog_Dnn.png -- 1,327.37kb -> 1,212.19kb (8.68%) /docs/img/capacitorCell.png -- 83.86kb -> 77.43kb (7.66%) /notebooks/LeNet.png -- 32.18kb -> 29.79kb (7.42%) /docs/img/mixedprecision_sgd.png -- 111.55kb -> 103.53kb (7.19%) /docs/img/ReRAM-ES.png -- 39.21kb -> 36.51kb (6.9%) /docs/img/ReRAM-SB.png -- 39.15kb -> 36.56kb (6.62%) /docs/img/tikitaka.png -- 166.09kb -> 155.85kb (6.17%) /docs/img/EcRam.png -- 28.25kb -> 26.71kb (5.44%) /docs/img/pulse_trains.png -- 275.10kb -> 266.01kb (3.3%) Signed-off-by: ImgBotApp <ImgBotHelp@gmail.com> Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

Co-authored-by: Malte Rasch <malte.rasch@ibm.com> Co-authored-by: Diego M. Rodríguez <diego.plan9@gmail.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

Co-authored-by: Kaoutar El Maghraoui <kaoutar@kaoutars-mbp.mynetworksettings.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

Fixes a small spelling mistake. Signed-off-by: Henry Ye <yehenry11@gmail.com>

* Updated the changelog * Changed the version of aihwkit Co-authored-by: Kaoutar El Maghraoui <kaoutar@kaoutars-mbp.mynetworksettings.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

…g and in-analog training to the non-idealities of Analog Crossbars (IBM#380) Signed-off-by: Henry Ye <yehenry11@gmail.com>

Signed-off-by: Henry Ye <yehenry11@gmail.com>

…hod (IBM#389) Signed-off-by: Henry Ye <yehenry11@gmail.com>

Signed-off-by: Henry Ye <yehenry11@gmail.com>

…M#393) Signed-off-by: Henry Ye <yehenry11@gmail.com>

Signed-off-by: Henry Ye <yehenry11@gmail.com>

HCY-11

The build is failing at the moment due to dependency issues (transformers, evaluate, datasets aren't in requirements.txt). How should these dependencies be dealt with, since it doesn't seem appropriate to have these in the requirements.txt for just one example?

kaoutar55 · 2022-12-07T07:51:22Z

Hi @HCY-11 can you add the missing libraries to requirements file to fix the dependencies issues in the build:
Examples/24_bert_on_squad.py:26:0: E0401: Unable to import 'transformers.integrations' (import-error)
755examples/24_bert_on_squad.py:28:0: E0401: Unable to import 'transformers' (import-error)
756examples/24_bert_on_squad.py:38:0: E0401: Unable to import 'evaluate' (import-error)
757examples/24_bert_on_squad.py:39:0: E0401: Unable to import 'datasets' (import-error)
758examples/24_bert_on_squad.py:79:4: E0401: Unable to import 'wandb' (import-error)

Signed-off-by: Henry Ye <yehenry11@gmail.com>

maljoras

Many thanks great work!
Please remove the extra dependency and Aimos cluster information though. These do not belong into the public repo.

maljoras · 2022-12-07T19:17:42Z

examples/24_bert_on_squad.py

+                    type=float)
+PARSER.add_argument("-r", "--run_name",
+                    help="Tensorboard run name",
+                    defualt=datetime.now().strftime("%Y%m%d-%H%M%S"),


should this be "default" ?

maljoras · 2022-12-07T19:21:32Z

examples/24_bert_on_squad.py

+DOC_STRIDE = 128
+
+
+def create_ideal_rpu_config(g_max=160, tile_size=256, w_noise=0.0, out_noise=0.0):


the naming is confusing. Adding noise would make it not "ideal" anymore

examples/24_bert_on_squad.py

maljoras · 2022-12-07T19:24:10Z

examples/24_bert_on_squad.py

+
+def create_optimizer(model):
+    """Create the analog-aware optimizer"""
+    optimizer = AnalogSGD(model.parameters(), lr=2e-4)


[OPTIONAL] adding the LR as an input parameter might be helpful for HWA training

examples/24_bert_on_squad.py

examples/README.md

maljoras · 2022-12-07T19:34:18Z

examples/README.md

+
+source ~/.bashrc
+conda activate <env-name>
+srun --mem=32G python ~/barn/aihwkit/examples/24_bert_on_squad.py --noise 0.1 &


please remove all the aimos explanation from this public example

examples/README.md

maljoras · 2022-12-07T19:39:15Z

examples/README.md

+metrics evaluated using the model at various times after training completed.
+
+Commandline arguments can be used to control certain options
+```python


make an real use case example here rather than pasting the parser

maljoras · 2022-12-07T19:40:26Z

requirements.txt

@@ -9,3 +9,7 @@ scipy
 requests>=2.25,<3
 numpy>=1.19
 protobuf>=3.15.0,<4
+transformers


please remove all these additional from requirements.txt! We cannot make this a global dependency. Use the extra dependency in setup.py for that

Signed-off-by: Henry Ye <yehenry11@gmail.com>

maljoras

@HCY-11 many thanks looks almost ready!

only these changes:

restore the old travis, we do not test the examples in travis so no need to install [bert]
it seems that 'tensorboard' is needed as well, please add it in setup.py

setup.py

maljoras · 2022-12-09T15:11:21Z

.travis.yml

@@ -39,6 +39,7 @@ job_compile_common: &job_compile_common
    - $PYTHON_EXECUTABLE -m pip install -r requirements.txt
    # Install the package in editable mode.
    - VERBOSE=1 $PYTHON_EXECUTABLE -m pip install -v -e ".[visualization]"
+    - VERBOSE=1 $PYTHON_EXECUTABLE -m pip install -v ".[bert]"


can you remove this from travis again? we do not test the examples in travis. Thanks

maljoras · 2022-12-09T15:11:49Z

.travis.yml

@@ -100,6 +101,7 @@ jobs:
      - $PYTHON_EXECUTABLE -m pip install -r requirements.txt
      # Install the package in editable mode.
      - VERBOSE=1 $PYTHON_EXECUTABLE -m pip install -v -e ".[visualization]"
+      - VERBOSE=1 $PYTHON_EXECUTABLE -m pip install -v ".[bert]"


remove this line again. Just leave travis as was before.

examples/24_bert_on_squad.py

Merge pull request #1 from IBM/master

cc84091

Pull from IBM master

maljoras requested changes Nov 10, 2022

View reviewed changes

examples/24_bert_on_squad.py Outdated Show resolved Hide resolved

maljoras and others added 28 commits December 6, 2022 17:17

Bump copyright year (IBM#359)

6f9eee5

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Follow-up on out scaling alpha (IBM#360)

4da4387

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Add reverse up / down device option (IBM#361)

87ec8b1

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Drop Python 3.6, introduce Python 3.10 (IBM#368)

c04c929

* Update travis test stages * Update travis cibuildwheel stages * Update version specifiers * Fix numpy typing issues * Update CHANGELOG * Fix typo from my way future self Signed-off-by: Henry Ye <yehenry11@gmail.com>

Contributor documentation updates (IBM#370)

7cf5d40

* Add security policy * Add test policy to contributor guidelines Signed-off-by: Henry Ye <yehenry11@gmail.com>

GNU ABI autodetect and MKL libirc legacy dependence removed (IBM#373)

b716ba6

Signed-off-by: Henry Ye <yehenry11@gmail.com>

fixes (IBM#375)

3a93183

Co-authored-by: Malte Rasch <malte.rasch@ibm.com> Co-authored-by: Diego M. Rodríguez <diego.plan9@gmail.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

Updated the changelog (IBM#378)

69e5e4b

Co-authored-by: Kaoutar El Maghraoui <kaoutar@kaoutars-mbp.mynetworksettings.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

Update analog_ai_hw.rst (IBM#381)

be5894a

Fixes a small spelling mistake. Signed-off-by: Henry Ye <yehenry11@gmail.com>

Change aihwkit version (IBM#382)

9b862a2

* Updated the changelog * Changed the version of aihwkit Co-authored-by: Kaoutar El Maghraoui <kaoutar@kaoutars-mbp.mynetworksettings.com> Signed-off-by: Henry Ye <yehenry11@gmail.com>

Exploring the sensitivity of analog inference, analog-aware retrainin…

f40a6fd

…g and in-analog training to the non-idealities of Analog Crossbars (IBM#380) Signed-off-by: Henry Ye <yehenry11@gmail.com>

Fixes loading CUDA conv model from checkpoint when pre-used (IBM#387)

b90ea87

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Remapping weights functionality (IBM#388)

6620e9d

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Fixing programming noise visualization and program_analog_noise met…

9dee826

…hod (IBM#389) Signed-off-by: Henry Ye <yehenry11@gmail.com>

Concatenating updates in optimizer for performance reason (IBM#390)

b8e777a

Signed-off-by: Henry Ye <yehenry11@gmail.com>

LSTM cell with combined weights (IBM#391)

ba0c1ac

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Fix analog_summary when model is on cuda (IBM#392)

74f5dce

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Analog matrix-vector-product examples using analog tiles directly (IB…

3044578

…M#393) Signed-off-by: Henry Ye <yehenry11@gmail.com>

Began work on BERT on Squad example

1faffe8

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Complete draft example script + comments

638de94

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Remove low_cpu_mem_usage and overlapping tokens

8c51119

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Add analog optimizer and next steps

d7c6af3

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Added AnalogSGD optimizer and custom RPUConfig

5d651a1

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Validation preprocessing and loading bert on squad

aaae5ac

Signed-off-by: Henry Ye <yehenry11@gmail.com>

HCY-11 force-pushed the bert_example_hy branch 3 times, most recently from 3b1cee8 to 419aef4 Compare December 7, 2022 02:00

HCY-11 added 6 commits December 6, 2022 21:02

Comply with pycodestyle and add documentation

419aef4

Signed-off-by: Henry Ye <yehenry11@gmail.com>

More runtime options for run name and hwa training

4b9b477

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Comply with pydocstyle

a1566ca

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Merge branch 'master' of https://github.com/IBM/aihwkit

89f457b

Merge branch 'master' into bert_example_hy

99bc5be

Fix pylint errors

2fe3a03

Signed-off-by: Henry Ye <yehenry11@gmail.com>

HCY-11 commented Dec 7, 2022

View reviewed changes

HCY-11 added 2 commits December 7, 2022 12:34

Added dependencies tor requirements

b5b632c

Signed-off-by: Henry Ye <yehenry11@gmail.com>

More cmd-line arguments added to docs

38d93ed

Signed-off-by: Henry Ye <yehenry11@gmail.com>

kaoutar55 previously approved these changes Dec 7, 2022

View reviewed changes

kaoutar55 requested a review from maljoras December 7, 2022 18:29

maljoras requested changes Dec 7, 2022

View reviewed changes

Inference once for digital, modify config options

7708761

Signed-off-by: Henry Ye <yehenry11@gmail.com>

HCY-11 dismissed kaoutar55’s stale review via b9ffb45 December 7, 2022 22:04

HCY-11 added 5 commits December 7, 2022 17:05

Move dependencies, remove AiMOS info from README

b9ffb45

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Fix typos and add learning rate arg

3bcb963

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Rename optional deps, add install to .travis.yml

f8ffcfb

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Added checkpoint loading

8559e22

Signed-off-by: Henry Ye <yehenry11@gmail.com>

Specify exception type

209fc3a

Signed-off-by: Henry Ye <yehenry11@gmail.com>

maljoras requested changes Dec 9, 2022

View reviewed changes

remove bert from travis

915a491

maljoras previously approved these changes Dec 9, 2022

View reviewed changes

import error

81cf97d

maljoras dismissed their stale review via 81cf97d December 9, 2022 15:46

maljoras approved these changes Dec 9, 2022

View reviewed changes

maljoras merged commit e522c10 into IBM:master Dec 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bert Example #440

Bert Example #440

HCY-11 commented Nov 9, 2022

maljoras left a comment

HCY-11 left a comment •

edited

Loading

kaoutar55 commented Dec 7, 2022

maljoras left a comment

maljoras Dec 7, 2022

maljoras Dec 7, 2022

maljoras Dec 7, 2022

maljoras Dec 7, 2022

maljoras Dec 7, 2022

maljoras Dec 7, 2022

maljoras left a comment

maljoras Dec 9, 2022

maljoras Dec 9, 2022

		DOC_STRIDE = 128


		def create_ideal_rpu_config(g_max=160, tile_size=256, w_noise=0.0, out_noise=0.0):

Bert Example #440

Bert Example #440

Conversation

HCY-11 commented Nov 9, 2022

maljoras left a comment

Choose a reason for hiding this comment

HCY-11 left a comment • edited Loading

Choose a reason for hiding this comment

kaoutar55 commented Dec 7, 2022

maljoras left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maljoras left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HCY-11 left a comment •

edited

Loading