Training a rather large model (250 StoryBlocks) with multiple checkpoints halts on Linux

[guillaumeraymond]

Rasa version: 1.8.1

Rasa SDK version (if used & relevant): 1.10.0

Rasa X version (if used & relevant):

Python version: Python 3.7.4

Operating system (windows, osx, ...): OSX (working) Linux (not working)

Issue:
Training a rather big model halts on Linux but work on OS X.
It has been run on various configuration of build executors (4Gb Ram, 8 Gb Ram, 16 gb Ram) and it constantly "stops" (CPU not working) when training the model. On MacOS however, the training makes it through.

Error (including full traceback):

Processed Story Blocks: 100%|██████████| 272/272 [00:16<00:00, 16.33it/s, # trackers=3]  
Processed Story Blocks: 100%|██████████| 272/272 [00:00<00:00, 277985.06it/s]
Processed Story Blocks: 100%|██████████| 272/272 [04:03<00:00,  1.34it/s, # trackers=150]
Processed Story Blocks: 100%|██████████| 272/272 [04:20<00:00,  1.30it/s, # trackers=150]
Processed Story Blocks:  85%|████████▍ | 230/272 [03:26<00:59,  1.42s/it, # trackers=500]

Too long with no output (exceeded 10m0s): context deadline exceeded

Command or request that led to error:

poetry run rasa train -d data/domain.yml --out models -c policies.yml

Content of configuration file (config.yml) (if relevant):

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
# using the default pipeline copied from the docs
pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 25
    random_seed: 0

policies:
  - name: rasa_stack.policies.FormWithFallbackPolicy
  - name: MemoizationPolicy
    featurizer: 
      - name: rasa_stack.featurizer.MaxHistoryTrackerFeaturizerFormFallback
    max_history: 5
  # from https://rasa.com/docs/core/0.14.3/fallbacks/
  - name: FallbackPolicy
    # min confidence needed to accept an NLU prediction, 0.0 to avoid any unintended consequences of unrecognized intents.
    # this means that the fallback policy will only be triggered as a result of other policies 
    # not predicting any actions, but not as a result of NLU failing to recognized the intent
    # unrecognized intents will be handled in the form actions
    nlu_threshold: 0.99
    # min confidence needed to accept an action prediction from Rasa Core
    core_threshold: 0.99
    fallback_action_name: 'action_default_form_fallback'

Content of domain file (domain.yml) (if relevant):

[kevinthenet]

I wonder if it's CPU or Memory, I'm on Linux and have similar issues running with the TED Policy.

[sara-tagger]

Thanks for the issue, @chkoss will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[Ghostvv]

I think most probably it is out-of-memory issue, your checkpoints create so many stories, that it cannot load all them into memory

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please create a new issue if you need more help.