unsupported character train as [unk] token

[WongVi]

@baudm Please help with some questions to solve

1. Could you please let me know how can I train characters rather than the character set as an unknown character so that network will be able to predict [unk] token for an unsupported character for demo purpose?

2. when I train your module using the Japanese language after some iteration training is killed automatically. how can I train until some epoch without early killing?

[baudm]

@WongVi

1. Could you please let me know how can I train characters rather than the character set as an unknown character so that network will be able to predict [unk] token for an unsupported character for demo purpose?

See #9.

2. when I train your module using the Japanese language after some iteration training is killed automatically. how can I train until some epoch without early killing?

If your process is being killed, your machine might be running out of memory.

[WongVi]

@baudm I checked the issue but there is no explanation about the unknown token.
I checked your code so I found that you are just ignoring the unknown character instead of making a token. without ignore how can I handled it?

[baudm]

@WongVi

1. Designate an unknown token, e.g. [U] and add it to Tokenizer as specials_first

   parseq/strhub/data/utils.py

   Line 108 in 048f0bb

   specials_first = (self.EOS,)

2. Modify the following and replace blank string with '[U]'

   parseq/strhub/data/utils.py

   Line 41 in 048f0bb

   label = re.sub(self.unsupported, '', label)

3. Modify _tok2ids() to support the conversion of '[U]' to the corresponding token ID

   parseq/strhub/data/utils.py

   Lines 54 to 55 in 048f0bb

   	def _tok2ids(self, tokens: str) -> List[int]:
   	return [self._stoi[s] for s in tokens]

[WongVi]

@baudm Thank you I will check and update soon.

[WongVi]

@baudm I have checked and it works very well.
I have one more question

1. is it possible to save train weight during testing with updated test parameters?
   I mean save pretrained weight by changing some hyperparameters without training.