How to mask out fillvalues?

[stacyqxz]

Hi all,

I found that preprocessed data somehow included fill values, which turned out the output data were not able to be used for plotting.

For example, the fill values are usually 1e+20. I used regrid, seasonal_statistics, and area_statistics preprocessor functions for time series plots. The output was like

masked_array(data=[9.98104678e+17, 1.97682999e+19, 3.76259652e+19,
                   2.11887674e+18, 7.12585560e+17, 2.42904988e+19,
                   3.83060293e+19, 1.14585870e+18, 6.50874462e+17,
                   2.49780601e+19, 4.00753311e+19, 2.40443328e+18,
                   1.01333649e+18, 2.01941613e+19, 3.53207350e+19,
                   1.55684433e+18, 4.17846030e+17, 1.88506152e+19,
                   3.81267268e+19, 2.08546991e+18, 4.85132018e+17,
                   1.98732387e+19, 3.56400771e+19, 1.13438841e+18,
                   8.66187564e+17, 2.64195638e+19, 4.11718433e+19,
                   3.15882984e+18, 2.41743648e+18, 2.74264995e+19,
                   3.73239807e+19, 2.55567917e+18, 7.96358680e+17,
                   2.19147503e+19, 3.31268926e+19, 1.26322753e+18,
                   7.05486334e+17, 2.01779061e+19, 3.42177782e+19,
                   2.57779970e+18, 8.84380633e+17, 1.87215106e+19,
                   3.85598141e+19, 1.96694714e+18, 1.36234503e+18,
                   2.49704046e+19, 3.17884645e+19, 1.40000715e+18,
                   1.45940066e+18, 2.24998855e+19, 3.68371609e+19,
                   2.71718387e+18, 5.30683960e+17, 1.93526903e+19,
                   3.18554174e+19, 2.98984997e+17, 3.62673521e+17,
                   2.04708614e+19, 3.56169112e+19, 3.17689884e+18,
                   9.82726084e+17, 2.33117825e+19, 3.63891554e+19,

Is there a way to mask out fill values in the preprocessor step?

[valeriupredoi]

Normally these result files should contain a mask ie the data in the iris cube being structured in a cube.data member that contains valid values and masked values (where the data loaded a simple array is indeed 1e+20) and a cube.data.mask that is a Numpy mask object. Two questions:

• are you looking at the final preprocessed files?
• and how are you loading these files?

It could be that. depending on how these files are loaded, the mask is ignored, or, there is a bug in the last step of preprocessing that ignores the mask/doesn't write it to disk.

Terse/functional answer to your question would be - if you want to mask an array with a condition, use Numpy's masked_where function, but let's see what the answers are for those questions first, it's possible we'd have to fix some code 🍺

[zklaus]

@stacyqxz, as @valeriupredoi indicates, this seems to stem from an error somewhere along the way. It may be wrong attribute in the input file, it may be a problem in one of our preprocessors. Could you give us a bit more detail about what exactly you are using? Ideally, the recipe whose preprocessor output is problematic.