Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Insert eval_collector.data_collect when evaluate from checkpoint (calculate ItemCoverage) #1364

Conversation

fufufukakaka
Copy link
Contributor

@fufufukakaka fufufukakaka commented Jul 25, 2022

Hi
When I added ItemCoverage to my metrics config and ran trainer.evaluate from checkpoint, I got the following error.

Traceback (most recent call last):
  File "src/tools/evaluate_only.py", line 156, in <module>
    main()
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "src/tools/evaluate_only.py", line 147, in main
    test_result = trainer.evaluate(test_data, model_file=checkpoint)
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/recbole/trainer/trainer.py", line 481, in evaluate
    result = self.evaluator.evaluate(struct)
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/recbole/evaluator/evaluator.py", line 40, in evaluate
    metric_val = self.metric_class[metric].calculate_metric(dataobject)
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/recbole/evaluator/metrics.py", line 462, in calculate_metric
    item_matrix, num_items = self.used_info(dataobject)
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/recbole/evaluator/metrics.py", line 458, in used_info
    num_items = dataobject.get('data.num_items')
  File "/home/yusuke-fukasawa/.cache/pypoetry/virtualenvs/teller-novel-recbole-nYDNORo--py3.8/lib/python3.8/site-packages/recbole/evaluator/collector.py", line 39, in get
    raise IndexError("Can not load the data without registration !")
IndexError: Can not load the data without registration !

After reviewing the code, I think the code in the documentation would not set data.item_nums to eval_collector, resulting in the above error.
I inserted the code trainer.eval_collector.data_collect(train_data) and ran it again, then it worked.

So, in this PR, I propose to insert this code in we present how to test a model based on the previous saved parameters. section (use_modules.rst).

@Sherry-XLL
Copy link
Member

@fufufukakaka Hi, Yusuke!

Thanks very much for your timely feedback and contributions!
We will fix this problem in the latest update.

Best Wishes.

@Sherry-XLL Sherry-XLL changed the base branch from master to 1.1.x July 27, 2022 15:03
@Sherry-XLL Sherry-XLL merged commit 5ccc0d1 into RUCAIBox:1.1.x Jul 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants