Segment bash help docs

[hmltn-0]

import spacy, re, subprocess

raw = subprocess.run([“help wait”], stdout=subprocess.PIPE)
plaintxt = raw.stdout.decode(“utf-8”)
raw_sen = spacy.load(“en_core_web_sm”)(plaintxt).sents

# The sentences are pretty well segmented but sometimes little command brackets or headers would be better if separated. 
# Now I “suture” back up broken sentences while also stripping them of extra surrounding newlines. 
# I don’t really love this and I wish it were cleaner and simpler, maybe getting rid of the re module.

sen = [re.sub(“[\s]+”, “ “, s.strip()) for s in raw_sen]

Now I have the sentences of a Bash help file.

Apart from general critique of better ways to do this I’d like to hear more about how Spacy segmentation works.

In a different thread we were discussing how training a new model would be the only convenient way for Spacy to distinguish code elements, headers and sentences from each other.

I realized I can’t afford to use the Prodigy software so maybe someone can give me advice on how I should train my own segmentation model for the above application.

I’ll also research this myself now, was just curious to open some discussion about this.

Thanks

[svlandeg]

Hi Peter!

Do I understand your code snippet correctly that you're currently using re to try and fix common mistakes with your sentence segmentation? If that's the case, perhaps you can create yourself a "silver" training dataset automatically, by running this correction onto a larger set of sentences, and using the corrected output as training input? You could review the corrections in an intermediate step and correct them, to get to a more truely "gold" training dataset. The goal here would be that once you train a model, and it predicts sentences the way you expect, you won't need the re module in postprocessing anymore.