Issue Training Spancat with Transformers Component

[dlinde]

Hi,

I generated a config for spancat data on prodigy. I tried to adapt the config to accommodate a transformers component using the guidance at https://spacy.io/usage/embeddings-transformers#transformers. When I run the debug data command, I receive "ValueError: Cannot get dimension 'nO' for model 'transformer-listener': value unset." Obviously the nO value is null by default, but I wasn't able to find a resource for how to adapt the spancat training pipeline to work with transformers, so I'd appreciate knowing any other modifications to the config that should make the transformers component compatible.

• spaCy version: 3.2.1
• Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9
• Python version: 3.6.13

My config and error messages are reproduced below.

[paths]
train = "./medical_span_classifier_binary/train.spacy"
dev = "./medical_span_classifier_binary/dev.spacy"
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["spancat"]
batch_size = 64
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy.ngram_range_suggester.v1"
min_size = 1
max_size = 13

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.spancat]

[initialize.components.spancat.labels]
@readers = "spacy.read_labels.v1"
path = "medical_span_classifier_binary/labels/spancat.json"
require = false

[initialize.tokenizer]

Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/spacy/cli/debug_data.py", line 73, in debug_data_cli
    silent=False,
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/spacy/cli/debug_data.py", line 107, in debug_data
    nlp.initialize(lambda: train_corpus(nlp))
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/spacy/language.py", line 1305, in initialize
    proc.initialize(get_examples, nlp=self, **p_settings)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/spacy/pipeline/spancat.py", line 417, in initialize
    self.model.initialize(X=(docs, spans), Y=Y)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/layers/chain.py", line 88, in init
    layer.initialize(X=curr_input)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/layers/with_getitem.py", line 44, in init
    model.layers[0].initialize(X=X_i, Y=Y_i)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/layers/chain.py", line 86, in init
    layer.initialize(X=curr_input, Y=Y)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/model.py", line 299, in initialize
    self.init(self, X=X, Y=Y)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/layers/chain.py", line 90, in init
    curr_input = layer.predict(curr_input)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/model.py", line 315, in predict
    return self._func(self, X, is_train=False)[0]
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/spacy_transformers/layers/listener.py", line 61, in forward
    width = model.get_dim("nO")
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/thinc/model.py", line 175, in get_dim
    raise ValueError(err)
ValueError: Cannot get dimension 'nO' for model 'transformer-listener': value unset

[polm]

You have a TransformerListener but no Transformer for it to listen to. There are multiple ways to do this but usually we would put a Transformer in the pipeline (so pipeline = ["transformer", "spancat"], and there's a block for it).

You can see what this looks like by using the quickstart to get a config with a Transformer for another component (say textcat). You may find it easier to set up the config by taking a config like that and replacing the parts for the component you're not using with your spancat bits.

[adrianeboyd]

There's an example here: #10059. I did exactly what Paul is suggesting, just with a taggerin the quickstart. Be aware that this config is customized for the non-default spans_key setting in the demo project.

When spancat is no longer experimental, it will be integrated better into init config and the quickstart. But this has come up several times recently, so we should have another look at how init config works for built-in components that don't have full support in the config template yet.

[dlinde]

Thanks to you both!