Methods for users to exclude observations

[kdraeder]

Summary of the Issue

This discussion stems from #401, which is focused more on possibly incorrect or misleading QC values ending up in obs_seq_final files. Here the focus is on the benefits and drawbacks of methods for enabling users to exclude obs that are in obs_seq.out files which they want to use. The focus is not on handling observations which the user intends to use, but cannot be used for reasons that are not apparent until the assimilation is run. These methods include excluding obs:

1. when creating obs_seq.out files,
2. by processing obs_seq.out to exclude obs (before a job or as part of the cycle)
3. by namelist variable (typically in model_mod.nml),
4. by hardwired fortran code (typically in model_mod.f90)
5. Other methods ?

• It seems likely that we will continue to use a combination of these methods.
• Each exclusion may be best suited for one method, but we may favor one method over others for more consistency.
• The strategy should not impose burdens on other model interfaces, especially low order models.
• Whatever strategy we adopt may only be relevant until we refactor the obs_seq architecture into NetCDF.

Summary of the discussion (@kdraeder view, as of 2022-10-18)

+ means an advantage
- means a disadvantage

1. When creating obs_seq.out files

+ Completely clear which obs the creator might like to be used.
+ Smaller obs_seq.out files.
+ Use obs_sequence_tool (and ... ?) to exclude many classes of obs.
+ Can exclude based on horizontal location (limited area models) and many other characteristics.
- More sets of obs_seq.out files, with varying amounts of redundant content.
- Will require more scripts (or scripts with more, flexible options) to make the larger collection of obs_seq sets. This raises the question of how much we will support and advertize these scripts.
- obs_sequence_tool may not have all of the options a user wants. The rest will need to be done by scripting( ?).

2. By preprocessing obs_seq.out to exclude obs (before a job or as part of the cycle)

+ Starts with a pre-existing, comprehensive, obs_seq.out file, so obs types don't need to be gathered each time an variant set of observations is needed.
+ The preprocessor program can use 3. and 4 (below).
+ Can exclude based on horizontal location (limited area models) and many other characteristics.
+ Avoids partially redundant sets of obs_seq files created in 1.
+ But the variants could be saved for repeated use.
+ Can be done by fortran and scripting, whichever is most efficient.
- If run as part of the assimilation cycle, it adds to the run time. We expect this to be small, but will vary by case. It may cause more queue waiting time for the biggest jobs.
- It may recreate obs_seq files that have been created before (if they weren't saved).
- If an exclusion is done by fortran, changes require recompiling (and testing) the preprocessor.

3. By namelist variable (typically in model_mod.nml),

+ No recompilation is needed to change the exclusion, which is useful during experiment development and debugging.
+ It's convenient for users (if it's already installed).
+ It's recorded in the archived namelist.
- May not be able to pass the reason (QC) for the exclusion to filter or obs_seq.final.
This is already true for many or most kinds of model-specific exclusions.
This may distort the obs space diagnostics (e.g. showing that some obs, which the user intended to exclude,
appear to have been assimilated, but caused no increments).
One exception is obs_kind_nml: assimilate_these_obs and evaluate_these_obs.
Obs types which are not listed in either of those are excluded and given QC=5, which is written to the obs_seq.final.
Some other exclusions generate "failed forward operator" (QC=4), which is also written to obs_seq.final.
- Adding exclusions requires code modification and recompiling (and testing). Removing an exclusion can be done using a special value.

4. By fortran code (typically in model_mod.f90)

+ In some cases the exclusion may generate a meaningful QC value which is passed to obs_seq.out; if it happens at a point where data is in sync, e.g. in model_interpolate. The QC value probably won't identify exactly why the exclusion happened.
+ Documentation may be easier; the exclusion is always done in the same way.
- Changing the exclusion requires recompiling (and testing).
- In some cases the exclusion may not be able to pass a QC value to obs_seq.out; if it's implemented at a point in the code where there has been no sync of the tasks since distributed calculations have been done. E.g. in get_close_obs.

Nancy adds the idea:

i do think it might be worthwhile to make a general facility where any obs that has 0 close items is marked with a different QC. to do it consistently in parallel would require an additional global broadcast and sync point, which impacts performance quite a bit. it also would require additional code in the obs diagnostics, unfortunately. since it comes with a performance cost, it could be a useful debugging technique if we turned it on and off with a namelist item.

[nancycollins]

it's important to distinguish between obs which are not assimilated and obs which are assimilated but have no impact on the model state. in your discussion you conflate the two.

i believe dart currently correctly marks with a QC any obs which are not assimilated. whether it is worth doing anything for obs which are assimilated but have no impact on the state is a discussion topic. however it needs to be consistent for all obs which have no impact on the state, not just a selected subset.

[kdraeder]

@nancycollins, please point to the comments that conflate the 2 and I'll try to fix them.
The specific example we've been discussing is that I still believe that in cam-fv a user
can instruct DART to ignore (in my mind that = "do not assimilate") high observations.
That happens in get_close_obs, which does not pass a QC back to assim_tools.
It sets num_close = 0, which is passed back, but is not used to set a QC value.
So the ob is assimiliated (despite the user's instructions),
but has 0 impact (which mostly satisfies the user's intent).
Yes, there may be other obs that have 0 impact, but the user did not tell DART to ignore those,
so the user has no expectation about QCs.
When looking at the obs space (prior) diagnostics, which is our recommended method,
those high obs appear as "assimilated", and there's no evidence that the increments are 0,
so it looks like the user's instructions were ignored.
This is why I listed it as a potential drawback of namelist exclusions.

[nancycollins]

in cam-fv a user
can instruct DART to ignore (in my mind that = "do not assimilate") high observations.

this right here. if the obs is processed in the SEQUENTIAL_OBS loop, it's being assimilated.

[nancycollins]

i'll add a last comment and then give up here. i think the best way to handle this is to do as wrf and mpas do, which is to preprocess the obs_seq file before running filter. in those preprocess programs they do many things, such as: remove obs outside the model area, increase obs errors for obs near the boundaries to lessen their impact on the state, superob obs which are too dense, and anything else that improves the observations passed into filter. i think removing the high obs before filter sees them is the best way to handle this. giving them to filter to assimilate and then saying "no, just kidding" doesn't make sense. i don't see this as missing functionality in filter given there is an existing solution which follows what other models already do.

[kdraeder]

I agree that a preprocessor is a fine solution to this example.
In my view it's not perfect and not the best in every situation.
If model interfaces should never use namelist variables to exclude observations,
then let's put that in the documentation about interfaces (and in this issue) and fix the ones that do.

It seems to me that an exclusion by a preprocessor namelist would be robust,
and would avoid making filter ignore observations.

My hope was that this broad overview issue will help people be aware of the options
and make the best choice, without needing to discuss all of these issues over again.

[kdraeder]

in cam-fv a user
can instruct DART to ignore (in my mind that = "do not assimilate") high observations.

this right here. if the obs is processed in the SEQUENTIAL_OBS loop, it's being assimilated.

Yes it is, despite the user trying to exclude it using a variable named "no_obs_assim_above_level".
The user can't know, without digging into the code and state space results,
that those obs will be assimilated and generate 0 increments.
I think that this is not a conflation, it's a conflict between what the variable name communicates to the user,
and what actually happens. If we keep this variable, which we probably won't, we could rename it
to no_impact_from_obs_above_level.

[nancycollins]

very very good point about changing the name of that cam namelist item and the internal cam model_mod variable. it used to make the forward operator fail which did prevent assimilation. now that the functionality moved to get_close it prevents impact. your suggested new name is very apt and should be changed in a pull request along with the code change to where high obs are handled. otherwise namelists are irrelevant to anything else in this discussion. there's no reason a model_mod shouldn't use them anywhere they wish. good variable names are always a plus, however! :)

if you think additional documentation is needed i'd suggest this paragraph be added to the end of the docs for both get_close_state() and get_close_obs():

To prevent an individual assimilated observation from impacting the model state, get_close_state can return 0 close items. For consistent results both get_close_state and get_close_obs should return 0 for the same observations. Returning 0 close items will have no affect on the DART Qualtity Control (QC) value. For additional ways to control the assimilation and impact of observations see the documentation for assimilate_these_obs_types and evaluate_these_obs_types in the obs_kind_nml
namelist, and the obs_impact_tool program. Also see wrf_dart_obs_preprocess and mpas_dart_obs_preprocess for examples of preprocessing observation sequence files to modify how observations affect the assimilation.

(i intentionally did not say the QC will be 0. it could be set to a different value if posterior FO fails or if the obs value fails the outlier threshold, etc. also, i have no idea how to insert links to the other pages in the docs, sorry!)

[nancycollins]

i will add this comment here so there is a record of it.

during one of the cam model top discussions, jeff mentioned that if we aren't impacting the state at the model top, then should the interpolate routine really return expected obs values for high locations? if the answer is no then it really complicates forward operator routines which need to query for a column of values. in the past we've always done it by looping over model levels from 1-N. for cam level 1 is at the model top. if it fails there's no way to know the first "good" model level to start at.

for now i think the interpolate routine has to return good expected values for the entire model domain but there is a science question here. then there is a difficult implementation question on how to make column values work if we choose to fail interp at some model levels.

[hkershaw-brown]

Preprocessor to remove high obs for cam-fv added to dart in #480