-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Read colocated files made outside of pyaerocom #1060
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main-dev #1060 +/- ##
============================================
+ Coverage 78.53% 78.62% +0.08%
============================================
Files 120 125 +5
Lines 19939 19975 +36
============================================
+ Hits 15660 15706 +46
+ Misses 4279 4269 -10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
f60eafc
to
98957ba
Compare
98957ba
to
36dc6a5
Compare
Example config of usage. For the observation network, you must state that #!/usr/bin/env python3
import os
# BASE FILTERS
ALTITUDE_FILTER = {"altitude": [0, 1000]}
MODEL_AUX_VARS = {
"vmrco2": dict(vars_required=["mmrco2"], fun="mmr_to_vmr"),
"vmrch4": dict(vars_required=["mmrch4"], fun="mmr_to_vmr"),
}
# Setup for models used in analysis
MODELS = {
"IFS-Reanalysis-GHG": dict(
model_id="ECMWF_I1DG",
obs_vars=["vmrco2", "vmrch4"],
model_read_aux=MODEL_AUX_VARS,
),
}
OBS_GROUNDBASED = {
"ICOS": dict(
obs_id="ICOS",
obs_vars=["vmrco2", "vmrch4"],
obs_vert_type="Surface",
only_json = True,
coldata_dir= "/lustre/storeB/users/lewisb/Python/Evaluations/aeroval/coldata/cams2-82/ICOS/IFS-Reanalysis-GHG"
),
}
# Setup for supported satellite evaluations
OBS_SAT = {}
OBS_CFG = {**OBS_GROUNDBASED, **OBS_SAT}
DEFAULT_RESAMPLE_CONSTRAINTS = dict(monthly=dict(daily=21), daily=dict(hourly=18))
CFG = dict(
model_cfg=MODELS,
obs_cfg=OBS_CFG,
json_basedir=os.path.abspath("../../data"),
coldata_basedir=os.path.abspath("../../coldata"),
io_aux_file=os.path.abspath("../eval_py/gridded_io_aux.py"),
reanalyse_existing=True,
only_json=False,
add_model_maps=True,
only_model_maps=False,
clear_existing_json=False,
raise_exceptions=False,
# Regional filter for analysis
filter_name="ALL-wMOUNTAINS",
# colocation frequency (no statistics in higher resolution can be computed)
ts_type="monthly",
map_zoom="World",
freqs=["daily", "monthly"],
periods=[
"2018-2021",
"2018",
"2019",
"2020",
"2021",
],
main_freq="monthly",
stats_tseries_base_freq="daily",
zeros_to_nan=False,
min_num_obs=DEFAULT_RESAMPLE_CONSTRAINTS,
colocate_time=False,
obs_remove_outliers=True,
model_remove_outliers=False,
harmonise_units=True,
regions_how="htap",
annual_stats_constrained=False,
proj_id="testing",
exp_id="fix857",
exp_name="Read colocated files",
exp_descr=("Read colocated files #857"),
exp_pi="Lewis",
public=True,
# directory where colocated data files are supposed to be stored
weighted_stats=True,
var_order_menu=["vmrco2", "vmrch4"],
)
if __name__ == "__main__":
from pyaerocom.aeroval import EvalSetup, ExperimentProcessor
stp = EvalSetup(**CFG)
ana = ExperimentProcessor(stp)
ana.exp_output.delete_experiment_data(also_coldata=True)
res = ana.run() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You might want to correct the typo I mentioned.
Otherwise ready to merge.
Close #857
The idea here this that while an entire experiment can be set up to only produce json files by setting
only_json=True
, a specific obs-network can not. In this approach, we defineonly_json
andcoldata_dir
as keys in a obs config, and assuming both are provided, proceed with the standardColdataToJsonEngine
processing. Outside of pyaerocom, work will need to be done to take colocated data objects that partners provide, and convert them into a format (i.e., with appropriate metadata) that pyaerocom expects.