[FEATURE] separate or integrated module for projections

[kellijohnson-NOAA]

The topic of how to implement software to do projections has been discussed several times in different spaces, e.g., GitHub Issues, side conversations, Implementation Team Meetings. The main question is should the projection module be integrated into FIMS or an external, stand-alone module?

Options

Internal

• output projections at the same time as parameter estimates

Stand-alone

• Like a linear model in R where you first estimate parameters and then you run predict() on the results to generate forecasts; e.g., sdmTMB uses sdmtmb() to get fits and predict() to get predictions using TMB.

Management considerations

Each management council has their specifications regarding management quantities, decision tables, etc that are mandatory for reporting and although it is not needed for the population dynamics the ability to produce MSY, SBmsy, and Fmsy are needed for the minimum viable comparison.

[nathanvaughan-NOAA]

I would suggest that alternative 2 is the most practical and flexible approach. By separating parameter fitting from prediction/simulation we can develop code that is adaptable to serve both forecasting and future MSE applications. This approach should also simplify future simulation code additions such as region-specific forecasting modules by making them more independent of the underlying model estimation code. This could definitely benefit from further discussion.

[iantaylor-NOAA]

Would alternative 2 (separate estimation of main time series from forecast) limit our ability to extend time-varying processes into the forecast period?
For instance, if we have a variable represented as a random walk, I presume that if the forecasting module had access to the variance parameter and the final value, then the random walk could be extended by the forecast module in a way that would match the alternative 1 equivalent where main time series and forecast were estimated together.

[nathanvaughan-NOAA]

@iantaylor-NOAA That sort of scenario was part of my thinking for further discussion. My rationale for alternative 2 was that if we start with the assumption that forecasting will be external that will motivate a development approach that ensures all relevant model details such as TV parameter values, variances, and covariances are available to the forecasting module. This would avoid issues that could occur with an integrated forecasting approach where intermediate values are calculated internally and not exposed to the user preventing custom forecast modifications. I'm far from an expert on ADMB vs TMB but my initial impression is that this will be less of an issue for a modular FIMS package than a compiled executable like SS3.

[JonBrodziak]

Another rationale for alternative 2 is that the best estimation model, or ensemble model, for the assessment time horizon and based on the BSIA assessment data, may not necessarily be the best predictor for future forecasts of system state. Alternative forecasting model structure(s) may provide better results, e.g., exponential smoothing or AI applications. It is important to keep the FIMS platform extensible and flexible. In the near future, most (all?) fishery systems are going to be subject to some degree of nonstationary forcing, in some cases outside the range of observed oceanographic conditions. For example, global average atmospheric CO2 was 412 ppm in 2020 This is not a stationary environmental driver and is strongly linked to ocean conditions both short- and long-term.

[JonBrodziak]

No, it would not limit our ability to extend time-varying processes into the forecast time frame. But it would increase the amount of information needed to initiate the forecast module, if the forecast module included the exat same modeled time-varying structure, which is not a necessary condition for a candidate forecasting model.

[Andrea-Havron-NOAA]

@iantaylor-NOAA, with respect to logistics, I think it will be straightforward to pull out the necessary values and covariances from a TMB model to forecast externally. I took a quick look at both SAM and WHAM and it looks like R scripts are set up for external projections. I agreen with @nathanvaughan-NOAA and @JonBrodziak wrt the added flexibility of alternative 2.

This also addresses issues brought up during M1 code review if projections are internal:

• need to be careful about which recruitment values are getting included in the likelihood (would not want to include projected recruitments).
• need to carefully define years/ ages for observations vs. predictions

[timjmiller]

@Andrea-Havron-NOAA from what I gather is meant by external vs. internal. WHAM can essentially do both, but typically used more like option 2. where a model is fitted, then projections options are incorporated to form a new tmb model with the projection attributed added and the sdreport is called again to get any new adreported ouput. Note that when you do this with wham, you just extend the model beyond the "data period" so that you can incorporate uncertainty of estimated a parameters into that of the projected attributes. Also, it makes simulation possible for the whole time series. It may also be beneficial for closed loop/MSE simulations where keeping the random seed the same over the loops might be important.

[Rick-Methot-NOAA]

I agree with Tim regarding the merits of integrating the forecast by making it an extension of same time series.

Regarding estimating future recruitments: SS3 treats future recruitments as part of the same process as recruitments during the time series. They are not fundamentally different than recruitments near the end of the time series which also have decreasing data to inform their values. They all have the same inherent true variability (sigmaR). By treating the future recruitments as being estimated with nil data, their variance is just sigmaR and that variance then propagates onto derived quantities in the future.