`rasa train` breaks on clean `rasa init` 2021-02-04 #7885

koaning · 2021-02-04T16:53:45Z

The rasa train command fails. I noticed this on 2021-02-04 after a fresh install via pip install rasa in a clean virtual environment.

> rasa train
rasa train
Training NLU model...
2021-02-04 17:48:55 INFO     rasa.shared.nlu.training_data.training_data  - Training data stats:
2021-02-04 17:48:55 INFO     rasa.shared.nlu.training_data.training_data  - Number of intent examples: 69 (7 distinct intents)

2021-02-04 17:48:55 INFO     rasa.shared.nlu.training_data.training_data  -   Found intents: 'bot_challenge', 'affirm', 'mood_unhappy', 'mood_great', 'goodbye', 'deny', 'greet'
2021-02-04 17:48:55 INFO     rasa.shared.nlu.training_data.training_data  - Number of response examples: 0 (0 distinct responses)
2021-02-04 17:48:55 INFO     rasa.shared.nlu.training_data.training_data  - Number of entity examples: 0 (0 distinct entities)
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Starting to train component WhitespaceTokenizer
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Finished training component.
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Finished training component.
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Finished training component.
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-02-04 17:48:55 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 80 vocabulary slots consumed out of 1080 slots configured for text attribute.
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Finished training component.
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-02-04 17:48:55 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 697 vocabulary slots consumed out of 1697 slots configured for text attribute.
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Finished training component.
2021-02-04 17:48:55 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
Traceback (most recent call last):
  File "/Users/vincent/Development/rasa-workshop/venv/bin/rasa", line 10, in <module>
    sys.exit(main())
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/cli/train.py", line 58, in <lambda>
    train_parser.set_defaults(func=lambda args: train(args, can_exit=True))
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/cli/train.py", line 102, in train
    finetuning_epoch_fraction=args.epoch_fraction,
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/train.py", line 109, in train
    loop,
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/utils/common.py", line 308, in run_in_loop
    result = loop.run_until_complete(f)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/train.py", line 174, in train_async
    finetuning_epoch_fraction=finetuning_epoch_fraction,
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/train.py", line 353, in _train_async_internal
    finetuning_epoch_fraction=finetuning_epoch_fraction,
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/train.py", line 396, in _do_training
    finetuning_epoch_fraction=finetuning_epoch_fraction,
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/train.py", line 818, in _train_nlu_with_validated_data
    **additional_arguments,
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/nlu/train.py", line 116, in train
    interpreter = trainer.train(training_data, **kwargs)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/nlu/model.py", line 209, in train
    updates = component.train(working_data, self.config, **context)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 818, in train
    self.component_config[BATCH_STRATEGY],
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py", line 220, in fit
    ) = self._get_tf_train_functions(eager, model_data, batch_strategy)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py", line 482, in _get_tf_train_functions
    train_dataset_function, self.train_on_batch, eager, "train"
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py", line 464, in _get_tf_call_model_function
    tf_call_model_function(next(iter(init_dataset)))
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
    *args, **kwds))
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper
    raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:

    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py:293 train_on_batch  *
        prediction_loss = self.batch_loss(batch_in)
    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/nlu/classifiers/diet_classifier.py:1413 batch_loss  *
        sequence_lengths = self._get_sequence_lengths(
    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/rasa/utils/tensorflow/models.py:1112 _get_sequence_lengths  *
        sequence_lengths = tf.ones([batch_dim], dtype=tf.int32)
    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper  **
        return target(*args, **kwargs)
    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:3041 ones
        output = _constant_if_small(one, shape, dtype, name)
    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2732 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:6 prod

    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3031 prod
        keepdims=keepdims, initial=initial, where=where)
    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/numpy/core/fromnumeric.py:87 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:848 __array__
        " a NumPy call, which is not supported".format(self.name))

    NotImplementedError: Cannot convert a symbolic Tensor (strided_slice_6:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

Ran on both;

Rasa Version     : 2.2.9
Rasa SDK Version : 2.2.0
Rasa X Version   : None
Python Version   : 3.7.7
Operating System : Darwin-19.3.0-x86_64-i386-64bit
Python Path      : /Users/vincent/Development/rasa-workshop/venv/bin/python

Rasa Version     : 2.2.4
Rasa SDK Version : 2.2.0
Rasa X Version   : None
Python Version   : 3.7.7
Operating System : Darwin-19.3.0-x86_64-i386-64bit
Python Path      : /Users/vincent/Development/rasa-workshop/venv/bin/python

I fear this might be a tensorflow bug.

The text was updated successfully, but these errors were encountered:

koaning · 2021-02-04T16:55:21Z

TensorFlow made a 2.4.1 release on Jan 21st. Might be related?

It's strange that our tests didn't catch this though. I'm checking TensorFlow versions in the meantime to see if that fixes it. I Will report here.

koaning · 2021-02-04T17:20:03Z

It seems unrelated to TF since we install <2.4. @dakshvar22 doesn't experience this bug while @JustinaPetr and myself do. It might be related to our OS. Still investigating.

b-quachtran · 2021-02-04T22:22:47Z

@koaning A customer ran into this issue and running pip install numpy==1.19.5 ended up resolving it.

koaning · 2021-02-05T08:30:09Z

Interesting. I had numpy==1.20.0 before. While installing I got this warning from pip.

ERROR: tensorflow 2.3.2 has requirement numpy<1.19.0,>=1.16.0, but you'll have numpy 1.19.5 which is incompatible.

Running rasa train now seems to work. Thanks, @b-quachtran! I'll let folks who are preparing the workshop know.

koaning · 2021-02-05T08:36:09Z

I'm wondering two things now.

How did our CI not pick this up? I fear it might be that I'm using pip while our CI is using poetry. @alwx maybe you have a more educated guess than mine.
Both @dakshvar22 and myself couldn't find the origin of this bug. We were using rasa --version to check our installations. I'm wondering if it makes sense to expand the command. Something like rasa --version --verbose that also gives us tensorflow, numpy, sanic versions. I wouldn't want to list everything, but I'd like to prevent the long search I tried to do last night.

dakshvar22 · 2021-02-05T08:39:23Z

We should get @wochinge 's opinion on this. Would pinning numpy to 1.19.5 in `pyproject.toml` help?

wochinge · 2021-02-05T09:21:11Z

Interesting. I had numpy==1.20.0 before. While installing I got this warning from pip.

I think the question is how did you install numpy? If you have installed it after installing Rasa Open Source, then it's on you. If you installed Rasa Open Source using poetry install after numpy, then something is wrong.

poetry / pip dependency resolving can't help in case you manually upgrade a dependency after an installation.

How did our CI not pick this up? I fear it might be that I'm using pip while our CI is using poetry. @alwx maybe you have a more educated guess than mine.

poetry.lock correctly resolved the version and uses 1.18.5 (

rasa/poetry.lock

Line 2252 in c5ee208

version = "1.18.5"

)

koaning · 2021-02-05T15:27:01Z

@wochinge All the bugs that I am reporting occurred after merely running pip install rasa. Manually reinstalling NumPy fixed it.

Do we want our community to switch to use poetry to install Rasa?

wochinge · 2021-02-05T15:28:40Z

Do we want our community to switch to use poetry to install Rasa?

no no no.

Which pip version do you have?

wochinge · 2021-02-05T15:29:11Z

after merely running

Clean environment or was numpy already installed?

wochinge · 2021-02-05T15:33:16Z

pip 20.3.3. in a clean env installs numpy 1.18.5 for me (Python 3.8.6)
pip 21.0.1 in a clean env installs numpy 1.18.5 for me (Python 3.8.6)
pip 20.1.1. in a clean env installs numpy 1.20.0 for me (Python 3.7.9) 🚨
pip 21.0.1 in a clean env installs numpy 1.18.5 for me (Python 3.7.9)

TensorFlow pinned it https://github.com/tensorflow/tensorflow/blob/e84e4b3c841b0e81191f6d564bb61cfb9e1c33db/tensorflow/tools/pip_package/setup.py#L81

This is definitely a pip issue due to the missing dependency resolution in pip <= 20.2

❯ pip install rasa
Collecting rasa
  Using cached rasa-2.2.9-py3-none-any.whl (689 kB)
Collecting SQLAlchemy<1.4.0,>=1.3.3
  Using cached SQLAlchemy-1.3.23-cp37-cp37m-macosx_10_14_x86_64.whl (1.2 MB)
Collecting numpy<2.0,>=1.16
  Using cached numpy-1.20.0-cp37-cp37m-macosx_10_9_x86_64.whl (16.0 MB)
Collecting redis<4.0,>=3.4

...

Using legacy setup.py install for boto3, since package 'wheel' is not installed.
ERROR: sanic 20.12.1 has requirement multidict<6.0,>=5.0, but you'll have multidict 4.7.6 which is incompatible.
ERROR: sanic-jwt 1.6.0 has requirement pyjwt==2.0.0, but you'll have pyjwt 2.0.1 which is incompatible.
ERROR: tensorflow 2.3.2 has requirement gast==0.3.3, but you'll have gast 0.4.0 which is incompatible.
ERROR: tensorflow 2.3.2 has requirement numpy<1.19.0,>=1.16.0, but you'll have numpy 1.20.0 which is incompatible.

wochinge · 2021-02-05T15:55:44Z

We can definitely limit our numpy dependency but this will hurt maintainability as we'd have to have to double check numpy for every tensorflow dependency change. What do you think @joejuzl @alwx ? Btw, I couldn't find a related sentry issue 🎉

koaning · 2021-02-05T16:21:40Z

Clean environment or was numpy already installed?

Yes. Clean env.

Which pip version do you have?

I've been using;

pip 19.2.3 from /Users/vincent/Development/rasa-workshop/venv/lib/python3.7/site-packages/pip (python 3.7)

wochinge · 2021-02-05T17:03:33Z

That makes sense 👍🏻 What do you think in regards to my previous comment @koaning ?

koaning · 2021-02-05T22:25:57Z

This is among one of the reason why I've always pinned numpy with some of my other projects.

It's not great that we have an extra concern, but if doing this prevents errors at the 'pip install rasa'-part of the learning journey I'm all in favor.

koaning added type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. area:rasa-oss 🎡 Anything related to the open source Rasa framework labels Feb 4, 2021

koaning changed the title ~~rasa train breaks on Rasa~~ rasa train breaks on clean rasa init 2021-02-04 Feb 5, 2021

TyDunn added priority:high area:rasa-oss/cli Issues focused on the rasa command-line-interface type:dependencies Pull requests that update a dependency file labels Feb 5, 2021

This was referenced Feb 8, 2021

Error on train using DIET classifier autograph #7876

Closed

fix incompatible numpy version #7904

Merged

wochinge self-assigned this Feb 8, 2021

wochinge closed this as completed Feb 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`rasa train` breaks on clean `rasa init` 2021-02-04 #7885

`rasa train` breaks on clean `rasa init` 2021-02-04 #7885

koaning commented Feb 4, 2021 •

edited

Loading

koaning commented Feb 4, 2021 •

edited

Loading

koaning commented Feb 4, 2021

b-quachtran commented Feb 4, 2021

koaning commented Feb 5, 2021

koaning commented Feb 5, 2021 •

edited

Loading

dakshvar22 commented Feb 5, 2021 via email •

edited by koaning

Loading

wochinge commented Feb 5, 2021

koaning commented Feb 5, 2021

wochinge commented Feb 5, 2021

wochinge commented Feb 5, 2021

wochinge commented Feb 5, 2021 •

edited

Loading

wochinge commented Feb 5, 2021 •

edited

Loading

koaning commented Feb 5, 2021

wochinge commented Feb 5, 2021

koaning commented Feb 5, 2021 •

edited

Loading

rasa train breaks on clean rasa init 2021-02-04 #7885

rasa train breaks on clean rasa init 2021-02-04 #7885

Comments

koaning commented Feb 4, 2021 • edited Loading

koaning commented Feb 4, 2021 • edited Loading

koaning commented Feb 4, 2021

b-quachtran commented Feb 4, 2021

koaning commented Feb 5, 2021

koaning commented Feb 5, 2021 • edited Loading

dakshvar22 commented Feb 5, 2021 via email • edited by koaning Loading

wochinge commented Feb 5, 2021

koaning commented Feb 5, 2021

wochinge commented Feb 5, 2021

wochinge commented Feb 5, 2021

wochinge commented Feb 5, 2021 • edited Loading

wochinge commented Feb 5, 2021 • edited Loading

koaning commented Feb 5, 2021

wochinge commented Feb 5, 2021

koaning commented Feb 5, 2021 • edited Loading

`rasa train` breaks on clean `rasa init` 2021-02-04 #7885

`rasa train` breaks on clean `rasa init` 2021-02-04 #7885

koaning commented Feb 4, 2021 •

edited

Loading

koaning commented Feb 4, 2021 •

edited

Loading

koaning commented Feb 5, 2021 •

edited

Loading

dakshvar22 commented Feb 5, 2021 via email •

edited by koaning

Loading

wochinge commented Feb 5, 2021 •

edited

Loading

wochinge commented Feb 5, 2021 •

edited

Loading

koaning commented Feb 5, 2021 •

edited

Loading