Skip to content

Latest commit

 

History

History
362 lines (259 loc) · 21.1 KB

README.md

File metadata and controls

362 lines (259 loc) · 21.1 KB
Raw

thesis-sims

MATLAB code and data to reproduce the simulation results in my Laval University masters thesis in electrical and computer engineering:

For the code to reproduce the results of the following conference paper, refer to this repository instead https://github.com/billtubbs/ifac-2022-mmkf.

  • William Tubbs, André Desbiens, Jocelyn Bouchard, An Observer to Detect Infrequently-Occurring Disturbances in Grinding Operations, IFAC-PapersOnLine, Volume 55, Issue 21, 2022, Pages 13-18, ISSN 2405-8963, https://doi.org/10.1016/j.ifacol.2022.09.236.

The code in this repository has been tested with MATLAB versions 2019b, 2020b, and 2021b. It may or may not work with other versions!

1. Generating RODD disturbances (section 3.1 of thesis report)

The files for these simulations are currently in this repository:

2. Observer simulations with linear systems (section 3.2 of thesis report)

The files for these simulations are in the linear-sims sub-directory. Navigate to this directory and then follow the instructions below.

System models

The two systems considered are defined in the following files

Process observers

Observer parameter tuning experiments with SISO linear system (section 3.2.2)

The observer parameters were chosen by running multiple simulations with different combinations of parameter values. Run the following scripts in the sequence shown to generate the latex for the summary tables of observer parameter search results. Note: these simulations can take a long time to run.

For the Kalman filter (KF3):

For the MKF_SF observer (1995 version):

For the MKF_SF observer (1998 version):

For the MKF_SP observer:

Observer evaluation simulations with SISO linear system (section 3.2.2)

Run the following scripts (these take a while):

Observer parameter tuning experiments with 2x2 linear system (section 3.2.3)

For the Kalman filter (KF3):

For the MKF_SF observer (1995 version):

For the MKF_SF observer (1998 version):

For the MKF_SP observer:

Observer evaluation simulations with 2x2 linear system (section 3.2.3)

Run the following scripts (these take a while):

Plots

To make the various plots of the simulation results in section 3.2.2, run the following script, with the line sim_name = "rod_obs_sim1_all_seed"; uncommented.

Also uncomment one of the lines in the section choose observers to include in plots to determine which observer results to include int he plots. For example, to make the box plot in Figure 3.20, use the following:'

obs_sel_labels = {'KF3', 'MKF_SF95', 'MKF_SF1', 'MKF_SP1', 'SKF'};

To produce the plot 'Effect of random variables on the RMSE results=' in Fig. A.2 in the Appendix, run the following in sequence:

3. Observer simulations with grinding process simulator (section 3.3 of thesis report)

The files for these simulations are in the grind-sims sub-directory. Navigate to this directory and then follow the instructions below.

Data from grinding simulation model

The grinding simulation model used in these simulations is not included. Instead, input-output data from simulations with this model is saved in the following directory:

The data subdirectory contains time-series data sets from 15 simulations of a grinding process model (the model itself is not available here). Data sets 1 to 5 contain short simulations of 300 time steps. Simulations 6 to 15 contain longer simulations of 2460 time steps. The files contain data for 7 process variables although only two were used in this work (BASE_ORE_MIX and SAG_OF_P80_M).

# Filename Use
1 sim_OL_rc_est_mix_factor_300_1_ident.csv Process model estimation (Fig. 4 in paper)
2 sim_OL_rc_est_mix_factor_300_2_ident.csv Process model validation (model selection)
3 sim_OL_rc_est_mix_factor_300_3_ident.csv Initial observer test (Fig. 5 in paper)
4 sim_OL_rc_est_mix_factor_300_4_ident.csv Not used
5 sim_OL_rc_est_mix_factor_300_5_ident.csv Observer parameter optimization
6 ... 15 sim_OL_rc_est_mix_factor_2460_6_ident.csv ... sim_OL_rc_est_mix_factor_2460_15_ident.csv Observer evaluation (RMSE results in Table 2 and Fig. 6 in paper)

Process observers

  • grind-sims/rod_obs_P2DcTd4.m - observers with the identified linear system model and optimized parameters used for the grinding simulations in section 3.2.3.
  • grind-sims/rod_obs_P2Dcd1_T.m - these observers have a system model identified from simulation data without measurement nosie. This model provides better predictions of the true simulation outputs than the model identified from noisy measurements. However, results using these observers were not included in the thesis results.

Instructions to reproduce the results

Open the script rod_obs_sim.m and specify the input data sequences to include in the simulations in lines 55-57. For example, specify the first 5 as follows:

i_in_seqs = [1, 2, 3, 4, 5];

To change which observers are included in the simulations, edit line 82:

observers = {KF1, KF2, KF3, MKF_SF95, MKF_SF1, MKF_SP1, SKF};

The observers are defined in separate script files.

Running rod_obs_sim.m with the above settings should produce the following output:

Starting observer simulations with input seq. #1 ...
Observer simulation results saved to file: rod_obs_sim_1_1.csv
MKF simulation results saved to file: rod_obs_sim_1_1_MKF_SF95.csv
MKF simulation results saved to file: rod_obs_sim_1_1_MKF_SF1.csv
MKF simulation results saved to file: rod_obs_sim_1_1_MKF_SP1.csv
                                  KF1       KF2        KF3      MKF_SF95    MKF_SF1    MKF_SP1      SKF
                                _______    ______    _______    ________    _______    _______    _______

    RMSE                         3.5629    2.9746     2.1599     2.0206       2.014     2.1431     1.2235
    RMSE in transitions           3.937    2.7732     2.6771     2.6254      2.5853      2.814     1.5633
    RMSE in steady-state         3.1534    3.1596     1.4876     1.1525      1.2169     1.1546    0.75418
    Variance in steady-state    0.67006    8.7855     1.5117      1.369      1.3116     1.1122    0.36806
    RMSD in steady-state        0.26817    3.0764    0.67384    0.59802     0.59391    0.57502    0.29097

Existing results loaded from file: rod_obs_sim_1_summary.csv
Summary results saved to file: rod_obs_sim_1_summary.csv
Step responses identified: 1
Existing step responses loaded from file: rod_obs_sim_1_resps.csv
Step responses saved to file: rod_obs_sim_1_resps.csv

Starting observer simulations with input seq. #2 ...
Observer simulation results saved to file: rod_obs_sim_1_2.csv
MKF simulation results saved to file: rod_obs_sim_1_2_MKF_SF95.csv
MKF simulation results saved to file: rod_obs_sim_1_2_MKF_SF1.csv
MKF simulation results saved to file: rod_obs_sim_1_2_MKF_SP1.csv
                                  KF1       KF2        KF3      MKF_SF95    MKF_SF1    MKF_SP1      SKF
                                _______    ______    _______    ________    _______    _______    ________

    RMSE                         3.5217    3.0487     2.3228     2.4687      2.4607     2.5192      1.7649
    RMSE in transitions          3.9721    3.3371     2.9114     3.1844      3.1581     3.2259       2.216
    RMSE in steady-state         2.8002    2.6088     1.1098    0.79101     0.87124    0.92697     0.82963
    Variance in steady-state    0.18654     6.595     0.6185    0.15753     0.24418     0.2708    0.089818
    RMSD in steady-state        0.34913    2.5483    0.59618    0.42227     0.46404    0.44366     0.37122

Existing results loaded from file: rod_obs_sim_1_summary.csv
Summary results saved to file: rod_obs_sim_1_summary.csv
Step responses identified: 1
Existing step responses loaded from file: rod_obs_sim_1_resps.csv
Step responses saved to file: rod_obs_sim_1_resps.csv

Starting observer simulations with input seq. #3 ...
Observer simulation results saved to file: rod_obs_sim_1_3.csv
MKF simulation results saved to file: rod_obs_sim_1_3_MKF_SF95.csv
MKF simulation results saved to file: rod_obs_sim_1_3_MKF_SF1.csv
MKF simulation results saved to file: rod_obs_sim_1_3_MKF_SP1.csv
                                  KF1       KF2        KF3      MKF_SF95    MKF_SF1    MKF_SP1      SKF
                                _______    ______    _______    ________    _______    _______    _______

    RMSE                         3.5484    3.0647     2.1525      2.187      2.2579     2.1788     1.5972
    RMSE in transitions          4.2352     3.178     3.1659     3.3339      3.3265     3.3047     2.3832
    RMSE in steady-state         3.0623    2.9946     1.1804    0.98834      1.2294      1.018     0.8188
    Variance in steady-state     1.3538     10.76      1.795    0.89845      1.5499    0.99461    0.26255
    RMSD in steady-state        0.21132    2.8264    0.60988    0.50948     0.66169    0.58495    0.24328

Existing results loaded from file: rod_obs_sim_1_summary.csv
Summary results saved to file: rod_obs_sim_1_summary.csv
Step responses identified: 2
Existing step responses loaded from file: rod_obs_sim_1_resps.csv
Step responses saved to file: rod_obs_sim_1_resps.csv

Starting observer simulations with input seq. #4 ...
Observer simulation results saved to file: rod_obs_sim_1_4.csv
MKF simulation results saved to file: rod_obs_sim_1_4_MKF_SF95.csv
MKF simulation results saved to file: rod_obs_sim_1_4_MKF_SF1.csv
MKF simulation results saved to file: rod_obs_sim_1_4_MKF_SP1.csv
                                  KF1       KF2        KF3      MKF_SF95    MKF_SF1    MKF_SP1      SKF
                                _______    ______    _______    ________    _______    _______    _______

    RMSE                         3.4658    3.3323     2.0001     1.5196      1.7333     1.6168     1.1457
    RMSE in transitions          4.5037    3.3486     2.9724     2.3109      2.6663     2.5423     1.7677
    RMSE in steady-state         2.8932    3.3251     1.3684    0.98738      1.0944    0.95898    0.71758
    Variance in steady-state    0.81292    11.633     1.5673    0.99212      1.1773    0.72147     0.1773
    RMSD in steady-state        0.24586    2.7882    0.61642    0.59055     0.60218    0.51876    0.27521

Existing results loaded from file: rod_obs_sim_1_summary.csv
Summary results saved to file: rod_obs_sim_1_summary.csv
Step responses identified: 2
Existing step responses loaded from file: rod_obs_sim_1_resps.csv
Step responses saved to file: rod_obs_sim_1_resps.csv

Starting observer simulations with input seq. #5 ...
Observer simulation results saved to file: rod_obs_sim_1_5.csv
MKF simulation results saved to file: rod_obs_sim_1_5_MKF_SF95.csv
MKF simulation results saved to file: rod_obs_sim_1_5_MKF_SF1.csv
MKF simulation results saved to file: rod_obs_sim_1_5_MKF_SP1.csv
                                  KF1       KF2        KF3      MKF_SF95    MKF_SF1    MKF_SP1      SKF
                                _______    ______    _______    ________    _______    _______    ________

    RMSE                         3.0909    3.0379     2.0435     2.3128      2.1737     2.414       1.6779
    RMSE in transitions          3.7779    3.2699     2.5635     3.0196      2.8178    2.9945       2.2029
    RMSE in steady-state         1.9567    2.7302     1.1266    0.88113     0.90784    1.4206      0.58628
    Variance in steady-state    0.76163    9.5025     1.3601    0.69641     0.67505    1.1633     0.070481
    RMSD in steady-state        0.35642    2.5906    0.62558    0.55192     0.52433    0.6603      0.36777

Existing results loaded from file: rod_obs_sim_1_summary.csv
Summary results saved to file: rod_obs_sim_1_summary.csv
Step responses identified: 0
Existing step responses loaded from file: rod_obs_sim_1_resps.csv
Step responses saved to file: rod_obs_sim_1_resps.csv
run rod_obs_sim_plots.m to produce plots.
run rod_obs_calc_metrics.m to calculate evaluation metrics.
run rod_obs_step_plots.m to produce step response summary plot.

Simulation results

After running rod_obs_sim.m, the results of the simulations are saved as CSV files to the results subdirectory. For example:

  • rod_obs_sim_1_1_MMKF.csv
  • rod_obs_sim_1_1.csv
  • rod_obs_sim_1_2_MMKF.csv
  • rod_obs_sim_1_2.csv
  • rod_obs_sim_1_3_MMKF.csv
  • rod_obs_sim_1_3.csv
  • rod_obs_sim_1_4_MMKF.csv
  • rod_obs_sim_1_4.csv
  • rod_obs_sim_1_5_MMKF.csv
  • rod_obs_sim_1_5.csv
  • rod_obs_sim_1_resps.csv
  • rod_obs_sim_1_summary.csv

Explanation of output results files:

  • The files 'rod_obs_sim_1_1.csv', 'rod_obs_sim_1_2.csv', ... etc. contain the state estimates, output estimates and output estimation errors of each observer for the duration of each simulation.
  • The files 'rod_obs_sim_1_1_MMKF.csv', 'rod_obs_sim_1_2_MMKF.csv', ... etc. contain additional data on the multi-model observer (MMKF), such as the state estimates and conditional probabilities of each of the observer's Kalman filters.
  • The file 'rod_obs_sim_1_resps.csv' contains the data used to create the plot of observer responses to shocks (Fig. 6 in the paper)
  • The file 'rod_obs_sim_1_summary.csv' will contain a records of all the simulation parameters, model parameters, observer parameters, and overall RMSE metrics for each simualtion. This file is not over-written by 'rod_obs_sim.m'. Every time a new simulation is run, a new row is added to 'rod_obs_sim_1_summary.csv'. Before re-running simulations, remove the previous results from this file or erase it completely, otherwise, some results may be duplicated.

Plots

To produce the plots shown in the report, run simulations for all the datasets (1 to 15) and then run the scripts rod_obs_sim_plots.m and rod_obs_step_plots.m.

After running these scripts, images of the plot figures will be saved in the plots folder in pdf format:

  • rod_obs_sim_1_ioplot.pdf
  • rod_obs_sim_3_est.pdf
  • rod_obs_sim_resp_plot1.pdf
  • rod_obs_sim_resp_plot2.pdf

Evaluation metrics

To calculate the evaluation metrics in Table 3.8 of the thesis report, run the script rod_obs_calc_metrics.m. This should produce the following output:

Simulation results loaded from file: rod_obs_sim_1_summary.csv
Results for the following simulations found:
     6     7     8     9    10    11    12    13    14    15

Observer performance metrics
                                  KF1       KF2        KF3      MKF_SF95    MKF_SF1    MKF_SP1      SKF
                                _______    ______    _______    ________    _______    _______    _______

    RMSE                         3.2491    3.2575     1.8059     1.6982      1.6558     1.6719     1.2441
    RMSE in transitions          4.4357    3.2636     2.6902     2.9474      2.8302     2.9277      2.041
    RMSE in steady-state         2.8276    3.2547     1.4636     1.1186      1.1225     1.0807    0.89732
    Variance in steady-state     1.8572    10.064     1.6117    0.66163     0.69619    0.57594     0.2463
    RMSD in steady-state        0.13665    2.9053    0.61527    0.43946     0.45231    0.43034    0.16967

Latex table code:
\hline
% See script rod_obs_calc_metrics.m
% 16-Jun-2023 17:26:43 results with system P2DcTd4, sigma_M = 5, tau_ss = 1.2
RMSE($\hat{\mathbf{Y}},\mathbf{Y}$) overall & 1.81 & 1.70 & 1.66 & 1.67 & 1.24 \\
RMSE($\hat{\mathbf{Y}},\mathbf{Y}$) transient & 2.69 & 2.95 & 2.83 & 2.93 & 2.04 \\
RMSE($\hat{\mathbf{Y}},\mathbf{Y}$) steady-state & 1.46 & 1.12 & 1.12 & 1.08 & 0.90 \\
Var($\hat{\mathbf{Y}}$) steady-state & 1.61 & 0.66 & 0.70 & 0.58 & 0.25 \\
RMSD($\hat{\mathbf{Y}},\mathbf{Y}$) steady-state &0.62 & 0.44 & 0.45 & 0.43 & 0.17 \\
\hline

Sensitivity analysis

To run the simulations for the sensitivity analysis in section 3.3.4, run the following script:

rod_obs_sim_sens_P2DcTd4

There is also a script to run the sensitivity analysis with observers that have the improved system model, but these results were not included in the thesis as they were not significantly different:

rod_obs_sim_sens_P2Dcd1_T

To produce the heat-map plots in figures 3.33 and 3.34, run the following Python notebook:

Unit tests

A set of test scripts are included in each of the main sub-directories, to verify that the main sub-routines are working correctly. To run the tests run the following command from each sub-directory of the main repository.

runtests