Merging vision into main #4800

dirkgr · 2020-11-17T21:34:58Z

No description provided.

* Some initial work; lots left to do * Initial test mostly passing, though things are still a bit of a mess * tests are passing with small fixtures * remove prints * Test more stuff * PathLike * Make vilbert pass tests * PR comments * call float before log * add CI Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>

* saving state * Code is running, though it is returning zero gradients (but not None) * initial test passing, still working on albert * albert works, but bert-base-uncased still gives zero gradients * Loading of weights should now work * black, flake, mypy * remove drop and mask functionality from reader * make comment better * fix tests * flake Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>

@dirkgr

* first implementation * update docstrings * fixes * fix sharding logic * clean up DatasetReader * fix samplers * fixes * fixes * patch models for now * more fixes * fix linting error * fix model test case * some fixes * fix linting err * updates * rename dataloader -> data_loader * fixes * more JoinableQueue * set daemon=True * fixes * fix * fixes * fix * update shuffle logic * load instances right away when not lazy * add tqdm when num_workers <= 0 * apply_token_indexers * fix bug causing high mem usage * address some of @dirkgr's comments * fix lazy * use sensible default for max_batches_in_mem * ensure workers terminated on err * fix * start adding some tests * more tests * add some more tests * address most of Matt's comments * update PyTorchDataLoader test * get rid of lazy option * fix linting * update docs, change max_batches_per_epoch to max_instances_per_epcoh * update CHANGELOG * fix drop_last validation * fix py2md test fixture * handle drop_last * update docs * implement sharding for most readers * fix worker init fn * limit tqdm output * fixes

* ensure vision CI runs on each commit * fix * try fix CHANGELOG check

* reformat for new version of black (#4605) * reformat for new version of black * pin black * reformat for black * fix

…o' (#4608) * rename 'node_rank' to 'global_rank' * Clarify doc comments * fix line length

# Conflicts: # allennlp/commands/train.py # tests/data/dataset_readers/dataset_reader_test.py # tests/data/samplers/bucket_batch_sampler_test.py

* fix len calculation for new data loader * add test Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>

* make existing readers work with multi-process loading * add 'overrides' decorator * call apply_token_indexers in predictor * clean up * fix tests

* Initial design of the multi-task model * PR comments, more implementation * changelog and docs fix * More tests, and fixes for those tests * mypy and make test less flaky * Update allennlp/models/multitask.py * Update allennlp/models/multitask.py Co-authored-by: Dirk Groeneveld <groeneveld@gmail.com> * Update allennlp/models/multitask.py Co-authored-by: James Barry <james.barry26@mail.dcu.ie> * respect active heads in get_metrics * Clean up changelog * black (apparently github UI doesn't add newlines?) Co-authored-by: Dirk Groeneveld <dirkg@allenai.org> Co-authored-by: Dirk Groeneveld <groeneveld@gmail.com> Co-authored-by: James Barry <james.barry26@mail.dcu.ie>

* Passes a batch of detectron images to the model in the correct format * Loads a model and runs inference on it * Some initial work; lots left to do * Initial test mostly passing, though things are still a bit of a mess * tests are passing with small fixtures * remove prints * More configurable reader * add image_root and feature extraction to detectron model * Use general detectron cfg functions * Adds TensorField * Fix detectron dependency * Adds a detectron processor that we can use in dataset readers * Test more stuff * PathLike * Make vilbert pass tests * PR comments * call float before log * add CI * PathLike * Adds another NLVR2 reader * add region feature and grid feature configuration json and attrtibute to cfg file * change detectron_utils based on https://github.com/vedanuj/grid-feats-vqa/blob/master/extract_feature.py * add bottom up and top down roi head into detectron2 based on allennlp/models/detectron.py * Fix padding in TensorField * Fix field construction * Adds ability to read an arbitrary file * More type annotations * Remove old reader, add test for new one * Use the right kind of field * Run Jiasen's configs as tests * We don't need this field * Removes detectron reader * Remove detectron reader and field * Unify ArrayField and TensorField * Making sure that no merge will go cleanly from now on * Clean up the new output from the detectron processor a bit * Fix Detectron2 version as v0.2 * saving state * Code is running, though it is returning zero gradients (but not None) * initial test passing, still working on albert * albert works, but bert-base-uncased still gives zero gradients * Note * Formatting * Adds Registrable base classes for image operations * Adds a real example of a image2image module * Run the new code (without implementation) in the nlvr2 reader * Solve some issue involving circular imports * add new modules for vilbert * add parameters for detectron image loader. * push current code on implementing proposal generator. * push current progress on proposal generator * Update FasterRCNNProposalGenerator & Merge Detectron2 config * Loading of weights should now work * black, flake, mypy * Run detectron pipeline pieces one at a time This is unfinished and will not run this way. * Fix the data format for the backbone * Handle image sizes separately * remove drop and mask functionality from reader * make comment better * remove proposal_embedder, and finish proposal generator * working on grid embedder * added simple test for resnet backbone, which passes * Got proposal generator test passing * Change default number of detections per image: 100 => 36 * Fix detectron config hierarchy: test_detectron_per_image * Make number of detections configurable & Add test * rename ProposalGenerator to RegionDetector * try to fix makefile * another attempt at makefile * quotes in the pip command... * added a simple test for the dataset reader, made it pass * add feature caching to the dataset reader * another try with the makefile * a better temporary fix for installing detectron * writing files before committing is good... * fix tests * fix (at least part of) the vilbert tests * ok, this makefile change should actually work * add torchvision, try to remove eager import of detectron code * flake * cleanup * more cleanup * mypy, flake * add back code I shouldn't have removed * black * test and flake fixes * fix region_detector for multiple images and add feature and coords padding * fix imports * restore null grid embedder * add back (todo) null region detector * Bring back import changes, to fix circular imports caused by NLVR2 reader * region detector test passing * model test finally passing * update torchvision version * add vqav2 dataset * add gpu support for detectron feature extraction * add lmdbCache to cache feature into lmdb database * fix typo * update vqa jsonnet * fix url adding by cat * Fixes type annotation * Fixes borked error message * New feature cache * Formatting * Fix the tensor cache * Be explicit about our dependencies * Use the new tensor cache * Adds a test using the tensor cache * Run NLVR dataprep on GPU * Tqdm when finding images * Fixes padding in array field * Adjust max_length when truncating in PretrainedTransformerTokenizer * Fewer print statements * remove VQA from this branch and copy default vilbert parameters. * Sanjay's vision features cache script (#4633) * Use LMDB cache in NLVR2 dataset reader; fix a few typos * Standalone script for caching image features * Removing reference to LMDB cache in NLVR2 dataset reader * Adding back asterisk in nlvr2 dataset reader * Fixing one variable name mistake * Decreasing batch size and making a few cuda-related changes * Loading images in batches to avoid GPU OOM error * Pedantic changes for consistency * Run the pre-processing with the models and not the data loading * Filter out paths of images already cached * Add image extensions other than png * Fixes import error * Makes the vision features script work alongside other scripts or training runs Co-authored-by: sanjays <sanjays@ip-10-0-0-157.us-west-2.compute.internal> Co-authored-by: sanjays <sanjays@ip-10-1-10-157.us-west-2.compute.internal> Co-authored-by: Sanjay Subramanian <sanjays@allennlp-server1.corp.ai2> Co-authored-by: Sanjay Subramanian <sanjays_ssubramanian@hotmail.com> * Adds missing imports * Makes TensorCache into a real MutableMapping * Formatting * Changelog * Fix typecheck * Makes the NLVR2 reader work with Pete's new code * Fix type annotation * Formatting * Backwards compatibility * Fix tests * Fix broken config * Update grid embedder test * Fix vilbert_from_huggingface configuration * Don't run the vilbert_from_huggingface test anymore * Remove unused test fixtures * Fix the region detector test * Fix vilbert-from-huggingface and bring it back * Fuck the linter * Run the region detector test on GPU * Run more stuff on GPU The CPU test runner doesn't have enough memory. * Depend on newer version of Detectron * Reinstall Detectron before running tests * Just force CUDA to be on, instead of reinstalling Detecton2 * Detectron needs CUDA_HOME to be set during install At least this thing fails quickly. * Try a different way of wrangling the detectron installer * Bring back amp * Trying to make tests faster, and passing * use two regions, to make tests pass * black * Documentation for TensorCache * Documentation for the NLVR2 dataset reader * Rename ArrayField to TensorField Co-authored-by: Matt Gardner <mattg@allenai.org> Co-authored-by: jiasenlu <jiasenlu@gatech.edu> Co-authored-by: Jaemin Cho <heythisischo@gmail.com> Co-authored-by: jiasenlu <echosenm@gmail.com> Co-authored-by: sanjays <sanjays@ip-10-0-0-157.us-west-2.compute.internal> Co-authored-by: sanjays <sanjays@ip-10-1-10-157.us-west-2.compute.internal> Co-authored-by: Sanjay Subramanian <sanjays@allennlp-server1.corp.ai2> Co-authored-by: Sanjay Subramanian <sanjays_ssubramanian@hotmail.com>

* transformer toolkit: BertEmbeddings * transformer toolkit: BertSelfAttention * transformer toolkit: BertSelfOutput * transformer toolkit: BertAttention * transformer toolkit: BertIntermediate * transformer toolkit: BertOutput * transformer toolkit: BertLayer * transformer toolkit: BertBiAttention * transformer toolkit: BertEmbeddings * transformer toolkit: BertSelfAttention * transformer toolkit: BertSelfOutput * transformer toolkit: BertAttention * transformer toolkit: BertIntermediate * transformer toolkit: BertOutput * transformer toolkit: BertLayer * transformer toolkit: BertBiAttention * Attention scoring functions * merging output and self output * utility to replicate layers, further cleanup * adding sinusoidal positional encoding * adding activation layer * adding base class for generic loading of pretrained weights * further generalizing, adding tests * updates * adding bimodal encoder, kwargs in from_pretrained_module * vilbert using transformer toolkit * fixing test function * changing to torch.allclose * fixing attention score api * bug fix in bimodal output * changing to older attention modules * _construct_default_mapping returns mapping * adding kwargs to _get_input_arguments, adding examples * using cached_transformers * making transformer_encoder more general * added get_relevant_module, loading by name * fixing constructor name * undoing failure after merge * misc minor changes Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>

…eads` for both modalities (#4728) * separate num_attention_heads for both modalities, default arguments * adding tests for toolkit examples * debug statements for failing test * removing debug statements, reordering * Let's be more tolerant * removing commented code Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>

* separating TransformerPooler as a new module * adding size check

* improve worker error handling * rename test file

* adding cross_attention, renaming block -> stack * stack can be initialized with layer too Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>

* resolve _read type * fix sharded reader * fix data loader arg

* Make the VQA reader work for the other datasets * Also find pngs * Really support pngs * Remove debug code * More logging * Unexpected formatting * Respect the device * This is how your replace things in named tuples. * Remove unused import * This is how you override a method properly. * This is how you set parameters in detectron. * Also set the device for the region detector * Training configs for all three datasets contained in VQA * Bigger batches * Bigger batches for image processing * Fix vilbert-from-huggingface config * Make the config switch modes for constructing vocab * More vocab, more docs, better way of deriving vocab * Modernize the from_huggingface config * More updates to the from_huggingface config * Better hyperparameters stolen from another project * Fix for inverted parameter * Formatting * Throw a meaningful error message when we don't have images * Add a warning that includes instructions for how to fix things * Remove unused script * Merge issue * Adds named splits to the SNLI-VE reader * Make the multitask data loader discoverable * Formatting * More flexible inputs to the dataset readers * Prototype config for the multitask training job * json_lines_from_file() already calls cached_path() * Visual entailment should track accuracy * Switching to torch * Fixing VE image paths * Formatting * Experimentally use threaded_generator to read instances from readers simultaneously * Vilbert backbone * Fixed paths * Formatting * Adds heads * Revert "Experimentally use threaded_generator to read instances from readers simultaneously" This reverts commit a633e67. * Multitask trains now! * Remove useless parameter from GQA reader * Updated multitask config * Schedulers produce batches, not instances * Track multiple metrics * Make mypy happy * Formatting * Keep better track of which heads have been called * Fix the merge * We have more than strings for input * Remove unused imports * -1 is CPU * Go back to tracking instances per epoch so that the samplers can work * Better error message * A useful sampler to have * We haven't indexed until we've indexed * Makes tests pass * Formatting * Fine-tuning the metric tracker * Update model configs for my changes * Fixing model configs for Akshita's changes * Implement VisionTextModel in terms of VilbertBackbone * Formatting * Fix stale comment * Use the server paths by default, not Dirk's desktop * Fix tests * Formatting again * Removed data loader parameters that don't exist anymore * Clarified comment Co-authored-by: Evan Pete Walsh <epwalsh10@gmail.com>

* Moves vision models to allennlp-models * Also move test fixtures * Don't return so many instances if we're cutting them out later anyways * We actually need this image * Formatting * Fixing more paths

…on branch

* Better Callbacks * Reformatting * Fixes * Tests for updated TrainerCallback * Formatting and Type-Checking fixes

* Makes the metric tracker more consistent * Turns out we need best_epoch_metrics after all. * Backwards compatibility * Formatting

# Conflicts: # mkdocs-skeleton.yml

dirkgr · 2021-01-26T01:17:31Z

The tests will now fail because we also need this on the -models side.

epwalsh · 2021-01-26T02:11:45Z

CHANGELOG.md

@@ -6,10 +6,53 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).


-## Unreleased
+## Unreleased (2.x branch)


Suggested change

## Unreleased (2.x branch)

## Unreleased

epwalsh · 2021-01-26T02:12:21Z

CHANGELOG.md

+  much better scores.
+
+
+## Unreleased (1.x branch)


This section should probably be merged with above.

epwalsh

LSTM (looks scary to me), but let's do it

chenqiuyuan · 2021-08-20T06:55:39Z

Hello, I notice that the parameter "cache_directory" is removed from the dataset_reader.py in this change. How can I use a similar configuration as "datasetreader.cache_directory": "my_cache_path"?

dirkgr · 2021-08-20T16:37:38Z

You can set the cache path by setting the environment variable ALLENNLP_CACHE_ROOT.

dirkgr · 2021-08-20T16:38:19Z

This was necessary because the cache is used in many different places, not just in dataset readers anymore.

chenqiuyuan · 2021-08-22T23:14:52Z

You can set the cache path by setting the environment variable ALLENNLP_CACHE_ROOT.

Thank you very much : )!

matt-gardner and others added 30 commits July 16, 2020 01:15

Merge remote-tracking branch 'origin/master' into vision

f87df83

Merge branch 'master' into vision

6cc508d

Merge remote-tracking branch 'origin/master' into vision

3137961

Merge branch 'master' into vision

71d7cb4

Merge branch 'master' into vision

3833f7a

Merge remote-tracking branch 'origin/master' into vision

6f82005

Merge branch 'master' into vision

95e8253

ensure vision CI runs on each commit (#4582)

44c8791

* ensure vision CI runs on each commit * fix * try fix CHANGELOG check

ensure models check runs on right branch

1b08fd6

Changelog

cde06e6

Formatting updates for new version of black (#4607)

3d11419

* reformat for new version of black (#4605) * reformat for new version of black * pin black * reformat for black * fix

rename 'node_rank' to 'global_rank' in dataset reader 'DistributedInf…

de9165e

…o' (#4608) * rename 'node_rank' to 'global_rank' * Clarify doc comments * fix line length

remove duplicate padding calculations in collate fn (#4617)

319794a

Merge branch 'master' into vision

8746361

# Conflicts: # allennlp/commands/train.py # tests/data/dataset_readers/dataset_reader_test.py # tests/data/samplers/bucket_batch_sampler_test.py

fix len calculation for new data loader (#4618)

d7124d4

* fix len calculation for new data loader * add test Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>

make existing readers work with multi-process loading (#4597)

191b641

* make existing readers work with multi-process loading * add 'overrides' decorator * call apply_token_indexers in predictor * clean up * fix tests

Merge remote-tracking branch 'origin/master' into vision

f886fd0

Merge remote-tracking branch 'origin/master' into vision

41872ae

Merge remote-tracking branch 'origin/master' into vision

fa22f73

Merge remote-tracking branch 'origin/master' into vision

e39a5f6

This should have been part of the previously merged PR

2985236

Merge remote-tracking branch 'origin/master' into vision

677a9ce

separating TransformerPooler as a new module (#4730)

cc53afe

* separating TransformerPooler as a new module * adding size check

epwalsh and others added 13 commits January 14, 2021 10:39

improve worker error handling in MultiProcessDataLoader (#4912)

d7c9eab

* improve worker error handling * rename test file

Toolkit decoder (#4914)

5229da8

* adding cross_attention, renaming block -> stack * stack can be initialized with layer too Co-authored-by: Dirk Groeneveld <dirkg@allenai.org>

resolve _read type (#4916)

0f00d4d

* resolve _read type * fix sharded reader * fix data loader arg

fix merge conflicts

412896b

Moves vision models to allennlp-models (#4918)

9a4a424

* Moves vision models to allennlp-models * Also move test fixtures * Don't return so many instances if we're cutting them out later anyways * We actually need this image * Formatting * Fixing more paths

Prepare for release v2.0.0rc1

0a12299

Make releasing work with the renamed master branch, and with the visi…

4e43b94

…on branch

Debugging the release process in the slowest way possible

1f79840

Another attempt at fixing the release process

e413c78

Generic Callbacks (#4917)

93054bc

* Better Callbacks * Reformatting * Fixes * Tests for updated TrainerCallback * Formatting and Type-Checking fixes

Consistent metric tracker (#4928)

f91502c

* Makes the metric tracker more consistent * Turns out we need best_epoch_metrics after all. * Backwards compatibility * Formatting

Merge remote-tracking branch 'origin/main' into vision

aa5dae1

# Conflicts: # mkdocs-skeleton.yml

dirkgr marked this pull request as ready for review January 26, 2021 00:51

dirkgr added 2 commits January 25, 2021 17:03

Remove old script

409b5b9

Changes CI since we won't have a vision branch anymore

f0a9e38

dirkgr requested review from epwalsh and AkshitaB January 26, 2021 01:17

epwalsh reviewed Jan 26, 2021

View reviewed changes

CHANGELOG.md Outdated

much better scores.

## Unreleased (1.x branch)

Copy link

Member

epwalsh Jan 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section should probably be merged with above.

fix up CHANGELOG

b019e77

epwalsh approved these changes Jan 26, 2021

View reviewed changes

AkshitaB approved these changes Jan 26, 2021

View reviewed changes

dirkgr merged commit 67fa291 into main Jan 26, 2021

dirkgr deleted the vision branch August 20, 2021 16:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merging vision into main #4800

Merging vision into main #4800

dirkgr commented Nov 17, 2020 •

edited

Loading

dirkgr commented Jan 26, 2021

epwalsh Jan 26, 2021

epwalsh Jan 26, 2021

epwalsh left a comment

chenqiuyuan commented Aug 20, 2021

dirkgr commented Aug 20, 2021

dirkgr commented Aug 20, 2021

chenqiuyuan commented Aug 22, 2021

Merging vision into main #4800

Merging vision into main #4800

Conversation

dirkgr commented Nov 17, 2020 • edited Loading

dirkgr commented Jan 26, 2021

epwalsh Jan 26, 2021

Choose a reason for hiding this comment

epwalsh Jan 26, 2021

Choose a reason for hiding this comment

epwalsh left a comment

Choose a reason for hiding this comment

chenqiuyuan commented Aug 20, 2021

dirkgr commented Aug 20, 2021

dirkgr commented Aug 20, 2021

chenqiuyuan commented Aug 22, 2021

dirkgr commented Nov 17, 2020 •

edited

Loading