Rasa1.7.1 segmentation fault(core dumped) #5228

gitpranjal · 2020-02-12T09:50:45Z

Rasa version:

Rasa SDK version (if used & relevant):

Rasa X version (if used & relevant):
1.7.1

Python version:
3.6.8

Operating system (windows, osx, ...):
Ubuntu18.04LTS

Issue:
Segmentation fault(core dumped) upon attempting to training the rasa core model

Error (including full traceback):

Segmentation fault (core dumped)

Command or request that led to error:

rasa train -c config_bluekaktus_-_manufacturing.yml --out models

Content of configuration file (config.yml) (if relevant):

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
pipeline:
- name: "SpacyNLP"
  case_sensitive: false
- name: "SpacyTokenizer"
- name: "CountVectorsFeaturizer"
  analyzer: 'word'
  min_ngram: 1
  max_ngram: 3
  lowercase: true
  OOV_token: oov
  return_sequence: true
- name: "SpacyFeaturizer"
  return_sequence: true
- name: "RegexFeaturizer"
  return_sequence: true
- name: "SpacyEntityExtractor"
- name: "CRFEntityExtractor"
- name: "EntitySynonymMapper"
- name: "EmbeddingIntentClassifier"
  loss_type: "margin"
- name: "ResponseSelector"
- name: "retrieval_action_fallback.ResponseThreshold"


# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: "MemoizationPolicy"
    max_history: 81
  - name: "KerasPolicy"
    featurizer:
    - name: MaxHistoryTrackerFeaturizer
      max_history: 81
      state_featurizer:
        - name: BinarySingleStateFeaturizer
  - name: "MappingPolicy"
  - name: "FormPolicy"
  - name: "FallbackPolicy"
    nlu_threshold: 0.6
    core_threshold: 0.3
    fallback_action_name: "action_default_fallback"


importers:
- name: "RasaFileImporter"
- name: "my_training_data_importer.MyImporter"
  project_name: "bluekaktus_-_manufacturing"

Content of domain file (domain.yml) (if relevant):

The text was updated successfully, but these errors were encountered:

sara-tagger · 2020-02-12T13:00:05Z

Thanks for raising this issue, @wochinge will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

erohmensing · 2020-02-12T15:15:51Z

@gitpranjal is this thread helpful at all? #5196

gitpranjal · 2020-02-13T05:15:04Z

Yes that partially helped. Uninstalling gym and reinstalling a previous version solved the sementation fault.
pip uninstall gym
pip install gym==0.15.4

But it started giving Memory error. Hence the training worldn't start.

B/W, my OS is Ubuntu18.04 and rasa version is 1.7.1. Both the segmentation fault and Memory error don't come on windows but linux/ubuntu.

erohmensing · 2020-02-13T09:06:14Z

Got it. Yeah, the segmentation fault seems to only happen on Ubuntu based on the issues i've seen.

The rest seems like just a general memory error, how much memory does your machine have?

benos · 2020-02-16T19:43:14Z

I have the same problem when using the Docker images.

With this image, training works fine:
FROM rasa/rasa:1.7.0-spacy-en

With these, core training fails: (Segmentation fault right at the beginning of training core).
FROM rasa/rasa:1.7.1-spacy-en
FROM rasa/rasa:1.7.2-spacy-en

Tried boosting the RAM of Docker to 6GB with a 4GB swap but to no avail. Also, the failure happens at the exact same point regardless of the amount of RAM.

wochinge · 2020-02-17T10:10:32Z

Thanks @benos That made it really easy to reproduce for me. I'll have a look

wochinge · 2020-02-17T10:12:55Z

Using python -q -X faulthandler I get this stack trace

Training Core model...
Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1316.98it/s, # trackers=1]
Fatal Python error: Segmentation fault

Thread 0x00007fb7ca23d740 (most recent call first):
  File "/build/lib/python3.6/site-packages/rasa/core/utils.py", line 73 in subsample_array
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 482 in _create_start_trackers_for_augmentation
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 374 in generate
  File "/build/lib/python3.6/site-packages/rasa/core/training/__init__.py", line 69 in load_data
  File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 672 in load_data
  File "/build/lib/python3.6/site-packages/rasa/core/train.py", line 65 in train
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 361 in _train_core_with_validated_data
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 223 in _do_training
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 188 in _train_async_internal
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 101 in train_async
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 50 in train
  File "/build/lib/python3.6/site-packages/rasa/cli/train.py", line 76 in train
  File "/build/lib/python3.6/site-packages/rasa/__main__.py", line 76 in main
  File "/build/lib/python3.6/site-packages/rasa/__main__.py", line 87 in <module>
  File "/usr/local/lib/python3.6/runpy.py", line 85 in _run_code
  File "/usr/local/lib/python3.6/runpy.py", line 193 in _run_module_as_main
Segmentation fault

Getting this in 1.7.2-full:

Fatal Python error: Segmentation fault

Thread 0x00007febb5427740 (most recent call first):
  File "/build/lib/python3.6/site-packages/rasa/core/events/__init__.py", line 98 in __init__
  File "/build/lib/python3.6/site-packages/rasa/core/events/__init__.py", line 366 in __init__
  File "/build/lib/python3.6/site-packages/rasa/core/events/__init__.py", line 422 in empty
  File "/build/lib/python3.6/site-packages/rasa/core/trackers.py", line 525 in _reset
Processed Story Blocks:   0%|                                                                                                             | 0/5 [00:00<?, ?it/s]Processed Story Blocks:   0%|                                                                                               | 0/5 [00:00<?, ?it/s, # trackers=5]
  File "/build/lib/python3.6/site-packages/rasa/core/trackers.py", line 437 in update
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 137 in update
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 98 in copy
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 530 in _process_step
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 290 in generate
Processed Story Blocks:   0%|                                                                                               | 0/5 [00:00<?, ?it/s, # trackers=5]it__.py", line 69 in load_data
  File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 672 in load_data
  File "/build/lib/python3.6/site-packages/rasa/core/train.py", line 65 in train
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 361 in _train_core_with_validated_data
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 223 in _do_training
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 188 in _train_async_internal
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 101 in train_async
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 50 in train
  File "/build/lib/python3.6/site-packages/rasa/cli/train.py", line 76 in train
Processed Story Blocks:   0%|                                                                                               | 0/5 [00:00<?, ?it/s, # trackers=5]_main__.py", line 76 in main
  File "/build/lib/python3.6/site-packages/rasa/__main__.py", line 87 in <module>
  File "/usr/local/lib/python3.6/runpy.py", line 85 in _run_code
  File "/usr/local/lib/python3.6/runpy.py", line 193 in _run_module_as_main

Fatal Python error: Segmentation fault

Thread 0x00007f20eda56740 (most recent call first):
  File "/usr/local/lib/python3.6/copy.py", line 88 in copy
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 98 in copy
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 493 in _create_start_trackers_for_augmentation
  File "/build/lib/python3.6/site-packages/rasa/core/training/generator.py", line 374 in generate
  File "/build/lib/python3.6/site-packages/rasa/core/training/__init__.py", line 69 in load_data
  File "/build/lib/python3.6/site-packages/rasa/core/agent.py", line 672 in load_data
  File "/build/lib/python3.6/site-packages/rasa/core/train.py", line 65 in train
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 361 in _train_core_with_validated_data
Processed Story Blocks:   0%|                                                                                                             | 0/5 [00:00<?, ?it/s]Processed Story Blocks:   0%|                                                                                               | 0/5 [00:00<?, ?it/s, # trackers=5]e-packages/rasa/train.py", line 223 in _do_training
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 188 in _train_async_internal
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 101 in train_async
  File "/build/lib/python3.6/site-packages/rasa/train.py", line 50 in train
  File "/build/lib/python3.6/site-packages/rasa/cli/train.py", line 76 in train
  File "/build/lib/python3.6/site-packages/rasa/__main__.py", line 76 in main
  File "/build/lib/python3.6/site-packages/rasa/__main__.py", line 87 in <module>
  File "/usr/local/lib/python3.6/runpy.py", line 85 in _run_code

wochinge · 2020-02-17T10:19:59Z

pip install gym==0.15.4 fixes it @erohmensing (the others don't work). Thanks for the pointer 👍
I also checked if it's related to pyglet but it isn't

wochinge · 2020-02-17T10:55:19Z

I suggest pinning this to <=0.15.4. Research is removing the gym dependency with TF2.0 (cc @dakshvar22 )

wochinge · 2020-02-17T14:11:39Z

i'm fixing the segmentation fault with #5248 .

@gitpranjal How much memory has your machine?

gitpranjal · 2020-02-17T14:12:59Z

I have got 16 GB ram, Ubuntu 18.04 OS

…

On Mon, 17 Feb, 2020, 7:41 PM Tobias Wochinger, ***@***.***> wrote: i'm fixing the segmentation fault with #5248 <#5248> . @gitpranjal <https://github.com/gitpranjal> How much memory has your machine? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5228?email_source=notifications&email_token=AGPDPDXXLHFZLXLKTD3H2QTRDKLJZA5CNFSM4KTWIQR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL6RPCY#issuecomment-587011979>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGPDPDSO7L5UEF5WSM533VDRDKLJZANCNFSM4KTWIQRQ> .

wochinge · 2020-02-17T14:14:22Z

And how much memory is consumed? Can you please monitor the memory consumption during the training with a tool like htop or similar

gitpranjal · 2020-02-18T06:18:28Z

[image: Screenshot from 2020-02-18 11-31-26.png] [image: Screenshot from 2020-02-18 11-31-58.png] [image: Screenshot from 2020-02-18 11-32-00.png] [image: Screenshot from 2020-02-18 11-31-47.png] [image: Screenshot from 2020-02-18 11-29-57.png] In my config.yml file, max history is set to 251. It is getting killed at even 181

…

On Mon, Feb 17, 2020 at 7:44 PM Tobias Wochinger ***@***.***> wrote: And how much memory is consumed? Can you please monitor the memory consumption during the training with a tool like htop or similar — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5228?email_source=notifications&email_token=AGPDPDXUKTCO6Y54LGFG5BDRDKLT7A5CNFSM4KTWIQR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL6RYDA#issuecomment-587013132>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGPDPDWN5MTP3QAQYW4I723RDKLT7ANCNFSM4KTWIQRQ> .

wochinge · 2020-02-19T08:56:05Z

In my config.yml file, max history is set to 251. It is getting killed at

This is a very very very large max_history. Any specific reason to set it that hight? Can you maybe give it a try with something in the range 3-10?

gitpranjal · 2020-02-20T07:33:16Z

Yes, the stories are very long with a lot of slot filling and buttons being clicked giving multiple cases. On reducing the max_history below 180, stories are not followed properly.

…

On Wed, Feb 19, 2020 at 2:26 PM Tobias Wochinger ***@***.***> wrote: In my config.yml file, max history is set to 251. It is getting killed at This is a very very very large max_history. Any specific reason to set it that hight? Can you maybe give it a try with something in the range 3-10? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5228?email_source=notifications&email_token=AGPDPDW3IHUQKQLCCMCNLCDRDTX2NA5CNFSM4KTWIQR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMG45SQ#issuecomment-588107466>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGPDPDVNZOXOEYIJ4DHE2BTRDTX2NANCNFSM4KTWIQRQ> .

wochinge · 2020-02-20T08:36:35Z

I don't know your use case / story structure, but I would argue that this seems like an issue with the stories. I talked to a couple of colleagues and nobody has ever seen a max_history settings even close to it. I'm closing this issue since the segmentation fault is fixed and the max_history problem is a different thing. However, feel free to continue our discussion in here or in the Rasa Forum

gitpranjal added the type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. label Feb 12, 2020

erohmensing mentioned this issue Feb 12, 2020

dependency seems to cause segmentation fault #5231

Closed

wochinge added the area:rasa-oss 🎡 Anything related to the open source Rasa framework label Feb 17, 2020

wochinge mentioned this issue Feb 17, 2020

Pin gym #5248

Merged

4 tasks

wochinge closed this as completed Feb 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rasa1.7.1 segmentation fault(core dumped) #5228

Rasa1.7.1 segmentation fault(core dumped) #5228

gitpranjal commented Feb 12, 2020 •

edited

Loading

sara-tagger commented Feb 12, 2020

erohmensing commented Feb 12, 2020

gitpranjal commented Feb 13, 2020 •

edited

Loading

erohmensing commented Feb 13, 2020 •

edited

Loading

benos commented Feb 16, 2020

wochinge commented Feb 17, 2020

wochinge commented Feb 17, 2020 •

edited

Loading

wochinge commented Feb 17, 2020 •

edited

Loading

wochinge commented Feb 17, 2020

wochinge commented Feb 17, 2020

gitpranjal commented Feb 17, 2020 via email

wochinge commented Feb 17, 2020

gitpranjal commented Feb 18, 2020 via email

wochinge commented Feb 19, 2020

gitpranjal commented Feb 20, 2020 via email

wochinge commented Feb 20, 2020

Rasa1.7.1 segmentation fault(core dumped) #5228

Rasa1.7.1 segmentation fault(core dumped) #5228

Comments

gitpranjal commented Feb 12, 2020 • edited Loading

sara-tagger commented Feb 12, 2020

Please also check out the docs and the forum in case your issue was raised there too 🤗

erohmensing commented Feb 12, 2020

gitpranjal commented Feb 13, 2020 • edited Loading

erohmensing commented Feb 13, 2020 • edited Loading

benos commented Feb 16, 2020

wochinge commented Feb 17, 2020

wochinge commented Feb 17, 2020 • edited Loading

wochinge commented Feb 17, 2020 • edited Loading

wochinge commented Feb 17, 2020

wochinge commented Feb 17, 2020

gitpranjal commented Feb 17, 2020 via email

wochinge commented Feb 17, 2020

gitpranjal commented Feb 18, 2020 via email

wochinge commented Feb 19, 2020

gitpranjal commented Feb 20, 2020 via email

wochinge commented Feb 20, 2020

gitpranjal commented Feb 12, 2020 •

edited

Loading

gitpranjal commented Feb 13, 2020 •

edited

Loading

erohmensing commented Feb 13, 2020 •

edited

Loading

wochinge commented Feb 17, 2020 •

edited

Loading

wochinge commented Feb 17, 2020 •

edited

Loading