[BUG] "Ignore reviews before" Not Working

[anthonysharpy]

Describe the bug

When I try to use the "Ignore reviews before" feature it always tells me I have 0 reviews (or a very small number of reviews if I go back far enough), even though I have thousands of reviews.

I have a deck called "Core 2000". Its preset is "Core 2000 Deck". It's a subdeck of a deck called "Japanese". (BTW, this also seems to be not working for all my other decks too, this is just an example.)

This deck has 100-200 reviews per day:

When I don't enter a "ignore reviews before" date, it seems to show the correct number of reviews. For example, this is me pressing "Evaluate":

If I set "ignore reviews before" to a recent date - say 1st April 2024 - it will say "Only 0 reviews were found":

I have tried changing the filter settings but nothing seems to make a difference.

To Reproduce

Here's an export of my entire collection here, maybe this will help:

japanese.zip

Let me know if I've done the export wrong, never had to do one before...

Expected behavior

There should be thousands of reviews, not 0.

Screenshots

See above.

Environment

• Anki version: 24.04.01 (Desktop) (This is also broken on AnkiDroid, which is the same version)
• OS: Windows 10 / Android
• Scheduler version: Not sure where to find this
• Optimizer version: Not sure where to find this
• Simulator version: Not sure where to find this

Additional context

I've read that this can be caused by not setting up presets properly, but don't think that applies to me (I hope)? https://forums.ankiweb.net/t/failing-to-optimize-evaluate-fsrs-for-some-presets-error-500-only-0-reviews-were-found/43687

I would imagine that if it's able to show me my reviews in the stats graph, it should also be able to figure out my reviews in the optimisation dashboard.

[anthonysharpy]

Interestingly this was opened today ankitects/anki#3242

(edit: personally I've had this issue since I started using FSRS about 3 months ago)

[L-M-Sherlock]

It works as intended. The "ignore reviews before" filters out cards whose first review log's date is before the setting.

[anthonysharpy]

@L-M-Sherlock The description in the app says "If set, reviews before the provide dated will be ignored when optimizing & evaluating FSRS parameters". If that's the intended behaviour then shouldn't it say something like "If set, cards whose first review is before the provided date will be ignored when optimizing & evaluating FSRS parameters"?

[L-M-Sherlock]

The description seems wrong.

[brishtibheja]

IIRC, the reason this feature was added because some people might have changed their grading behaviour at some time in past, in which case those revlogs will be irrelevant, but might not the later reviews still be useful? I don't know why whole cards are being ignored. (the tooltip also mentions the situation where you imported somebody else's decks with their learning history)

@anthonysharpy If you have revlogs from before this particular date you are using for ignore reviews before, how is it possible your first reviews weren't before this date? It shouldn't show "0 reviews found" in my opinion.

[L-M-Sherlock]

I don't know why whole cards are being ignored.

FSRS optimizer doesn't accept incomplete review history.

[user1823]

I think that the last comment by L-M-Sherlock is slightly confusing. He means to say that FSRS optimizer can't use incomplete review history. The reason is that, in the FSRS algorithm, every review modifies the memory states from the previous one and you can't calculate memory states if you don't have the complete review history.

During reviews, however, we use a hack to estimate the memory states of cards with incomplete revlogs. Otherwise, we won't be able to calculate the intervals. However, this hack is not used when optimizing because optimization is a more high-stakes operation (because optimization affects all the cards not only those with incomplete revlogs).

If that's the intended behaviour then shouldn't it say something like "If set, cards whose first review is before the provided date will be ignored when optimizing & evaluating FSRS parameters"?

It makes sense. You can suggest this on forums.ankiweb.net.

[user1823]

One correction:

When optimizing, the whole card is ignored. But while evaluating the parameters, only the reviews before the provided date are ignored.

[anthonysharpy]

IIRC, the reason this feature was added because some people might have changed their grading behaviour at some time in past, in which case those revlogs will be irrelevant, but might not the later reviews still be useful? I don't know why whole cards are being ignored. (the tooltip also mentions the situation where you imported somebody else's decks with their learning history)

I am definitely not at all an expert in the way Anki works, but this pretty much sums up what I thought this was for too (and what the description and tutorial seem to say).

If you have revlogs from before this particular date you are using for ignore reviews before, how is it possible your first reviews weren't before this date? It shouldn't show "0 reviews found" in my opinion.

@brishtibheja Might be misunderstanding but my first reviews were before this date (at least for this deck). I guess that would explain why setting it to 1st April causes it to say "0 reviews" - because all my first reviews are from much longer ago.

If it's not possible to optimise using cards with incomplete review histories, I wonder if the ignore reviews before feature actually has any purpose? The tutorial and in-app description seem to say that this can be used for excluding some reviews but not others (which is as the tutorial mentions is useful if e.g. your answering patterns change, or you inherit someone else's deck). But as has been mentioned this seems to not actually be true.

Is it just that this option is useful for evaluating but not for optimising? (Sorry I don't actually know what evaluating is haha)

If this can't be used during optimisation, then maybe this option should be ignored when optimising? And a clearer description added.

If that makes sense then I can go suggest to the folks over on the forums.

[user1823]

If it's not possible to optimise using cards with incomplete review histories, I wonder if the ignore reviews before feature actually has any purpose?

When a date is set, Anki uses the review history of cards introduced AFTER the date to optimize the parameters. The assumption is that you have at least some cards you started reviewing after the date.

Sorry I don't actually know what evaluating is

Well, this was unexpected because the Evaluate button is placed just to the right of the Optimize button and also explained in the tutorial (which you seem to have read).

[anthonysharpy]

Well, this was unexpected because the Evaluate button is placed just to the right of the Optimize button and also explained in the tutorial (which you seem to have read).

Oh yeah true, brain fart, I do know what it is.

When optimizing, the whole card is ignored. But while evaluating the parameters, only the reviews before the provided date are ignored.

Checking again, I'm not sure this is the case since if I set the date to something like 1st April, it says 0 reviews found when pressing evaluate. So it seems the behaviour is the same (it ignores whole cards) whether evaluating or optimising.

When a date is set, Anki uses the review history of cards introduced AFTER the date to optimize the parameters. The assumption is that you have at least some cards you started reviewing after the date.

That does seem to be the assumption. But I think it's fair to say that that assumption is at odds with what the instructions say at the moment. Sort of seems to me this feature was implemented to solve problem X, but actually does not/can not solve that problem, since the fact that it can only ignore whole cards and not reviews makes the feature a lot less useful. Especially because I think most of the people who will want to use this feature will want to use it on decks that have lots of review history, and decks with lots of review history are less likely to have new cards (unless the user is still adding to it maybe).

[user1823]

it says 0 reviews found when pressing evaluate

But do you see some values of log loss and RMSE on pressing Evaluate?

If yes, this is another issue (ankitects/anki#3242) where there is a bug in the number of reviews displayed by Anki.

But I think it's fair to say that that assumption is at odds with what the instructions say at the moment.

Suggestions to improve the instructions are welcome. The Anki Forums are a better place for them, though.

since the fact that it can only ignore whole cards and not reviews makes the feature a lot less useful.

I understand what you want to say, but there is no option other than ignoring the whole card.

[L-M-Sherlock]

What's the problem you want to solve via "Ignore reviews before"?

[anthonysharpy]

But do you see some values of log loss and RMSE on pressing Evaluate?

Unfortunately not, it just says "Error: 500: ⁨Only ⁨0⁩ reviews were found.⁩ You must have at least 400 reviews for this operation". (edit: to be clear this is only when I select a recent date, e.g. 1st January 2024)

Suggestions to improve the instructions are welcome. The Anki Forums are a better place for them, though.

Sounds good, I'm about to go on holiday for about a week though, so will probably do it when I come back.

What's the problem you want to solve via "Ignore reviews before"?

Other than the things that have already been mentioned, personally I wanted to see if using only the more up-to-date data (I've had some of these decks for over three years now) might make optimisation more accurate. Not sure how realistic that is, but I was just thinking that the way I answer things might have changed a little bit since then.

[user1823]

Unfortunately not, it just says "Error: 500: ⁨Only ⁨0⁩ reviews were found.⁩ You must have at least 400 reviews for this operation".

I can't reproduce this. Can you try using the latest Anki version (24.06.2)?

In ankitects/anki#3106, the minimum limit for Evaluating was removed. Probably, you don't have 400 reviews since 1st January 2024.

Note that FSRS counts reviews differently. For example, it counts one review when you review the same card on at least two different days.

[anthonysharpy]

@user1823 I get this on 24.06.03 as well

Here's a video - am I doing it right?

2024-07-09.12-12-44.mp4

[Expertium]

When optimizing, the whole card is ignored. But while evaluating the parameters, only the reviews before the provided date are ignored.

Wait, really? God, this is even more of a mess than I remember. @user1823, can you write a description of "Ignore reviews before" that will be used for ankitects/anki#3288 and in the tutorial? And a description of historical retention. And make sure that both of them don't make the user think "oh no, my reviews will be permanently deleted!". Sorry for the trouble, but I don't know how to write it

[brishtibheja]

Tip (if user1823 would get us a description): Brevity and clarity both are important if we are writing for the tooltips. Thus, write as less as you can but as much as you should.

[omer681]

I think the issue (judging by the images in the first post) is that the user expects the date convention to be dd/mm/yyyy while it's actually mm/dd/yyyy.

[brishtibheja]

I don't think that's the issue.

[anthonysharpy]

Assuming all the above is true then I imagine the feature should probably also be renamed from "ignore reviews before" to something like "ignore cards learned before"?

[user1823]

If you forget about Evaluate for a moment, FSRS in Anki has two main components: the optimizer and the scheduler.

As you have correctly interpreted, when a date is set, the optimizer ignores cards learned before the provided date.

However, we can't do this with the scheduler. If the scheduler ignores the whole card, FSRS won't work for that card and SM-2 will instead be used for scheduling. So, as a workaround, the scheduler only ignores the reviews before the provided date and then tries to guess the memory states at the first review done after the provided date.

So, the behaviour of the "ignore reviews before" option is inconsistent between the scheduler and the optimizer. Unfortunately, this means that there is no way to succinctly explain what this option does.

PS:
Evaluation behaves exactly the same as the scheduler. This is because the aim of evaluation is to determine how good the scheduler predicts the memory states, which would be most reliable when it works exactly the same as the scheduler.

[Expertium]

Wait, I thought that for scheduling, no reviews are ignored, and cards just use parameters that were generated by the optimizer while ignoring reviews? So the reviews are ignored for the purpose of optimization, but not for scheduling.
...yeah, I really wish someone other than me wrote that section of the tutorial, I have absolutely no clue how any of this works

[Expertium]

Ok, so there are only three people in the world who understand how "Ignore reviews before" works: @L-M-Sherlock, @user1823, and @Luc-Mcgrady. Unfortunately, LMSherlock's English isn't good enough to write the tutorial and the tooltip for Anki. That leaves user1823 and Luc. Can either of you guys help with this, this and this, please?

[Luc-Mcgrady]

As has been explained in this thread and for reasons that were explained to me at length which I have since somewhat forgotten, this option ignores cards which are introduced after the date set. (I implemented it the way it is described at first but was told it would mess with the first review which is very important for FSRS).

This means if a card does not have a learning step after the given date, all reviews for that card are completely ignored.

This, as I see it, leaves people with 2 options:

• If your review strategy has changed to a very large degree (e.g you have completely changed the cards or possibly if you were answering "hard" instead of "again"), you should "forget" all or some proportion of the cards so that FSRS will be able to make assumptions regarding the first review again. (probably still a bad idea because you will of course still remember the card)
• Add more cards, If more cards are added these will of course have been introduced after the date and therefore not ignored, this will mean however the other cards will be eternally ignored and not

The manual should really read like this

If set, 
- reviews before 
+ cards which have no learning steps after
the provided date will be ignored when optimising & evaluating FSRS parameters. This can be useful if you imported someone else's scheduling data, or have changed the way you use the answer buttons.

I suppose the option should be named "Ignore cards before"

[Expertium]

I suppose the option should be named "Ignore cards before"

I'm afraid that will make users think that cards that were created before that date will be ignored.

[Luc-Mcgrady]

"Ignore cards introduced before"?

[Expertium]

"Ignore cards reviewed before" then, though it's a bit too long

[Luc-Mcgrady]

"Ignore introduced before"??
little less clear

[brishtibheja]

I think this discussion belongs here: https://forums.ankiweb.net/t/ignore-reviews-before-is-misleading/46686

Dae and others in the forum can also chip in.

[L-M-Sherlock]

This issue has been inactive for a while. Let me summarize it:

1. The previous description of "Ignore reviews before" is misleading. Now it has been fixed in ankitects/anki@495b07a
2. The current implementation cannot only ignore reviews instead of cards unless I refactor the training process of FSRS in the backend to support sample weight.

I have no plan to support it, so I close this issue.