About consis_model.py : Using reberta-base instead of xlm-roberta-base?

[Fang-git0]

The line: AutoTokenizer.from_pretrained("/data/lilinfang/clv/xlm-roberta-base")
use this can work out well.

however your code is
The line: RobertaTokenizer.from_pretrained("/data/lilinfang/clv/xlm-roberta-base")
got problem as follows
OSError: Can't load tokenizer for '/data/lilinfang/clv/xlm-roberta-base'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/data/lilinfang/clv/xlm-roberta-base' is the correct path to a directory containing all relevant files for a RobertaTokenizer tokenizer.

I guess you maybe use reberta-base instead of xlm-roberta-base?
thank you and looking for replying.

您好我想知道您用的reberta是xlm-reberta-base还是reberta-base，根据加载方式我判断是后者，前者运行会出错。感谢您的回复

[Toyhom]

感谢指出错误，应该是xlm-reberta-base.

…

---Original--- From: ***@***.***> Date: Wed, Nov 1, 2023 15:14 PM To: ***@***.***>; Cc: ***@***.***>; Subject: [Toyhom/CLV] About consis_model.py : Using reberta-base instead ofxlm-roberta-base? (Issue #4) The line: AutoTokenizer.from_pretrained("/data/lilinfang/clv/xlm-roberta-base") use this can work out well. however your code is The line: RobertaTokenizer.from_pretrained("/data/lilinfang/clv/xlm-roberta-base") got problem as follows OSError: Can't load tokenizer for '/data/lilinfang/clv/xlm-roberta-base'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/data/lilinfang/clv/xlm-roberta-base' is the correct path to a directory containing all relevant files for a RobertaTokenizer tokenizer. I guess you maybe use reberta-base instead of xlm-roberta-base? thank you and looking for replying. 您好我想知道您用的reberta是xlm-reberta-base还是reberta-base，根据加载方式我判断是后者，前者运行会出错。感谢您的回复 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Fang-git0]

感谢您的回复，我已经调整。
运行过程中产生了一些新的问题，
Traceback (most recent call last):
File "/data/lilinfang/clv/Consis_Model.py", line 362, in
dm.setup('fit')
File "/data/lilinfang/clv/Consis_Model.py", line 40, in setup
with open(os.getcwd() + self.data_path[0] + "_train.txt",encoding='utf-8') as f:
AttributeError: 'Consis_Data' object has no attribute 'data_path'

代码这里
class Consis_Data(pl.LightningDataModule):
def init(self):
super().init()
self.dataset_train = None
self.dataset_valid = None
self.batch_size = 16
config = Config()
data_language = config.data_language
请问使用的数据集文件还是code中Data文件夹内的相同格式数据集吗？这里valid和train对应test和train的俩个文件路径吗？

[Toyhom]

你好，训练一致性模型时，数据被组织成两个子句的形式，以便进行分类，我会对这一点尽快做出补充。您可以先把一致性评价关闭。

…

---Original--- From: ***@***.***> Date: Wed, Nov 1, 2023 15:55 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [Toyhom/CLV] About consis_model.py : Using reberta-base insteadof xlm-roberta-base? (Issue #4) 感谢您的回复，我已经调整。 运行过程中产生了一些新的问题， Traceback (most recent call last): File "/data/lilinfang/clv/Consis_Model.py", line 362, in dm.setup('fit') File "/data/lilinfang/clv/Consis_Model.py", line 40, in setup with open(os.getcwd() + self.data_path[0] + "_train.txt",encoding='utf-8') as f: AttributeError: 'Consis_Data' object has no attribute 'data_path' 代码这里 class Consis_Data(pl.LightningDataModule): def init(self): super().init() self.dataset_train = None self.dataset_valid = None self.batch_size = 16 config = Config() data_language = config.data_language 请问使用的数据集文件还是code中Data文件夹内的相同格式数据集吗？ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[Fang-git0]

非常感谢，一致性模型这块好像有很多地方无法跑通，从input id到embedding部分都很多报错，可能是我理解不够，还未调整好。如果有更新就非常感谢了。

…

---Original--- From: ***@***.***> Date: Wed, Nov 1, 2023 18:45 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [Toyhom/CLV] About consis_model.py : Using reberta-base insteadof xlm-roberta-base? (Issue #4) 你好，训练一致性模型时，数据被组织成两个子句的形式，以便进行分类，我会对这一点尽快做出补充。您可以先把一致性评价关闭。

---Original--- From: ***@***.***> Date: Wed, Nov 1, 2023 15:55 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [Toyhom/CLV] About consis_model.py : Using reberta-base insteadof xlm-roberta-base? (Issue #4) 感谢您的回复，我已经调整。 运行过程中产生了一些新的问题， Traceback (most recent call last): File "/data/lilinfang/clv/Consis_Model.py", line 362, in dm.setup('fit') File "/data/lilinfang/clv/Consis_Model.py", line 40, in setup with open(os.getcwd() + self.data_path[0] + "_train.txt",encoding='utf-8') as f: AttributeError: 'Consis_Data' object has no attribute 'data_path' 代码这里 class Consis_Data(pl.LightningDataModule): def init(self): super().init() self.dataset_train = None self.dataset_valid = None self.batch_size = 16 config = Config() data_language = config.data_language 请问使用的数据集文件还是code中Data文件夹内的相同格式数据集吗？ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>