"
]
@@ -1190,7 +6332,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.6"
+ "version": "3.11.9"
}
},
"nbformat": 4,
diff --git a/notebooks/motivating_example.ipynb b/notebooks/motivating_example.ipynb
index 5e7c2af..dfa2d42 100644
--- a/notebooks/motivating_example.ipynb
+++ b/notebooks/motivating_example.ipynb
@@ -10,7 +10,7 @@
}
},
"source": [
- "# Motivating example: Figure 4"
+ "# Motivating Example: Figure 4"
]
},
{
@@ -27,7 +27,7 @@
}
},
"source": [
- "Figure below is the motivating example in this paper: *Eliater: an open source software for causal query estimation from observational measurements of biomolecular networks*. This graph contains one mediator $M_1$ that connects the exposure $X$ to the outcome $Y$."
+ "Figure below is the motivating example in this paper: *Eliater: an open source software for causal query estimation from observational measurements of biomolecular networks*. This graph contains one mediator $M_1$ that connects the treatment $X$ to the outcome $Y$."
]
},
{
@@ -74,17 +74,17 @@
"
"
+ ""
],
"text/plain": [
- " X M1 Z1 Z2 Z3 R1 R2 R3 Y\n",
- "0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0\n",
- "1 1.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0\n",
- "2 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0\n",
- "3 1.0 1.0 1.0 0.0 0.0 1.0 1.0 1.0 1.0\n",
- "4 1.0 0.0 1.0 1.0 0.0 0.0 0.0 1.0 1.0\n",
- ".. ... ... ... ... ... ... ... ... ...\n",
- "995 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
- "996 1.0 1.0 1.0 0.0 0.0 1.0 0.0 1.0 1.0\n",
- "997 1.0 1.0 1.0 0.0 1.0 0.0 0.0 1.0 1.0\n",
- "998 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0\n",
- "999 1.0 1.0 0.0 1.0 1.0 1.0 0.0 0.0 1.0\n",
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/markdown": [
+ "Of the 26 d-separations implied by the ADMG's structure, only 1 (3.85%) rejected the null hypothesis for the cressie_read test at p<0.01.\n",
"\n",
- "[1000 rows x 9 columns]"
+ "Since this is less than 30%, Eliater considers this minor and leaves the ADMG unmodified. Finished in 0.23 seconds.\n"
+ ],
+ "text/plain": [
+ ""
]
},
- "execution_count": 7,
"metadata": {},
- "output_type": "execute_result"
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/markdown": [
+ "| left | right | given | stats | p | dof | p_adj | p_adj_significant |\n",
+ "|:-------|:--------|:--------|--------:|------------:|------:|------------:|:--------------------|\n",
+ "| R1 | R3 | R2;Y | 30.9659 | 3.11086e-06 | 4 | 8.08823e-05 | True |"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
}
],
"source": [
- "data_trans"
+ "eliater.step_1_notebook(graph=graph, data=data, binarize=True)"
]
},
{
"cell_type": "code",
"execution_count": 8,
- "id": "4e3220a2c48cb9b4",
+ "id": "6a1d0da707ca1d2f",
"metadata": {
"ExecuteTime": {
- "end_time": "2024-01-25T14:36:23.142139Z",
- "start_time": "2024-01-25T14:36:20.906764Z"
+ "end_time": "2024-01-25T14:36:23.240315Z",
+ "start_time": "2024-01-25T14:36:23.157745Z"
},
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "\n",
+ "## Step 2: Check Query Identifiability\n",
+ "\n",
+ "The causal query of interest is the average treatment effect of $X$ on $Y$, defined as: \n",
+ "$\\mathbb{E}[Y \\mid do(X=1)] - \\mathbb{E}[Y \\mid do(X=0)]$.\n",
+ "\n",
+ "\n",
+ "Running the ID algorithm defined by [Identification of joint interventional distributions in recursive\n",
+ "semi-Markovian causal models](https://dl.acm.org/doi/10.5555/1597348.1597382) (Shpitser and Pearl, 2006)\n",
+ "and implemented in the $Y_0$ Causal Reasoning Engine gives the following estimand:\n",
+ "\n",
+ "$\\sum\\limits_{M_1, Z_1, Z_2, Z_3} P(M_1 | X, Z_1) P(Y | M_1, X, Z_1, Z_2, Z_3) P(Z_2 | Z_1) P(Z_3 | Z_1, Z_2) \\sum\\limits_{M_1, X, Y, Z_2, Z_3} \\sum\\limits_{R_1, R_2, R_3} P(M_1, R_1, R_2, R_3, X, Y, Z_1, Z_2, Z_3)$\n",
+ "\n",
+ "Because the query is identifiable, we can proceed to Step 3.\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "eliater.step_2_notebook(graph=graph, treatment=treatment, outcome=outcome)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "b52a7618-d06f-42f5-827c-ed5df6ed3264",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "## Step 3/4: Identify Nuisance Variables and Simplify the ADMG"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/markdown": [
+ "The following 3 variables were identified as _nuisance_ variables,\n",
+ "meaning that they appear as descendants of nodes appearing in paths between\n",
+ "the treatment and outcome, but are not themselves ancestors of the outcome variable:\n",
+ "\n",
+ "$R_1$, $R_2$, $R_3$\n",
+ "\n",
+ "These variables are marked as \"latent\", then\n",
+ "the algorithm proposed in [Graphs for margins of Bayesian\n",
+ "networks](https://arxiv.org/abs/1408.1809) (Evans, 2016) and implemented in\n",
+ "the $Y_0$ Causal Reasoning Engine is applied to the ADMG to\n",
+ "simplify the graph. This minimally removes the latent variables and makes\n",
+ "further simplifications if the latent variables are connected by bidirected\n",
+ "edges to other nodes.\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/markdown": [
+ "The simplification did not modify the graph."
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "reduced_graph = eliater.step_3_notebook(graph=graph, treatment=treatment, outcome=outcome)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "ee2d9eec-56d4-49b4-91c6-ba7af815a73d",
+ "metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Of the 26 d-separations implied by the network's structure, only 1(3.85%) rejected the null hypothesis at p<0.01.\n",
- "\n",
- "Since this is less than 30%, Eliater considers this minor and leaves the network unmodified.]\n",
- "\n",
- "Finished in 0.31 seconds.\n",
- "\n"
+ "Graph was not reduced, no need to re-check identifiability\n"
]
- },
- {
- "data": {
- "text/html": [
- "
\n",
- "\n",
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
left
\n",
- "
right
\n",
- "
given
\n",
- "
stats
\n",
- "
p
\n",
- "
dof
\n",
- "
p_adj
\n",
- "
p_adj_significant
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
0
\n",
- "
R1
\n",
- "
R3
\n",
- "
R2|Y
\n",
- "
30.915398
\n",
- "
0.000003
\n",
- "
4
\n",
- "
0.000083
\n",
- "
True
\n",
- "
\n",
- "
\n",
- "
1
\n",
- "
Z1
\n",
- "
Z3
\n",
- "
Z2
\n",
- "
9.646506
\n",
- "
0.008041
\n",
- "
2
\n",
- "
0.201015
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
2
\n",
- "
R3
\n",
- "
X
\n",
- "
R2|Y
\n",
- "
2.430030
\n",
- "
0.657207
\n",
- "
4
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
3
\n",
- "
X
\n",
- "
Z2
\n",
- "
Z1
\n",
- "
0.771982
\n",
- "
0.679777
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
4
\n",
- "
R2
\n",
- "
Z1
\n",
- "
X
\n",
- "
2.994924
\n",
- "
0.223697
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
5
\n",
- "
R3
\n",
- "
Z2
\n",
- "
R2|Y
\n",
- "
2.677977
\n",
- "
0.613074
\n",
- "
4
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
6
\n",
- "
R3
\n",
- "
Z1
\n",
- "
R2|Y
\n",
- "
5.641139
\n",
- "
0.227599
\n",
- "
4
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
7
\n",
- "
M1
\n",
- "
Z2
\n",
- "
Z1
\n",
- "
0.087005
\n",
- "
0.957430
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
8
\n",
- "
M1
\n",
- "
Z1
\n",
- "
X
\n",
- "
5.765985
\n",
- "
0.055967
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
9
\n",
- "
R1
\n",
- "
Z3
\n",
- "
Z2
\n",
- "
0.345576
\n",
- "
0.841316
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
10
\n",
- "
Y
\n",
- "
Z1
\n",
- "
X|Z2
\n",
- "
4.006201
\n",
- "
0.405167
\n",
- "
4
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
11
\n",
- "
R1
\n",
- "
Z2
\n",
- "
Z1
\n",
- "
2.326265
\n",
- "
0.312506
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
12
\n",
- "
X
\n",
- "
Y
\n",
- "
M1|Z2
\n",
- "
9.170047
\n",
- "
0.056987
\n",
- "
4
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
13
\n",
- "
R3
\n",
- "
Z3
\n",
- "
R2|Y
\n",
- "
4.481456
\n",
- "
0.344752
\n",
- "
4
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
14
\n",
- "
R1
\n",
- "
Y
\n",
- "
M1
\n",
- "
2.001399
\n",
- "
0.367622
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
15
\n",
- "
R2
\n",
- "
Y
\n",
- "
R1
\n",
- "
1.083802
\n",
- "
0.581642
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
16
\n",
- "
R2
\n",
- "
X
\n",
- "
R1
\n",
- "
1.204575
\n",
- "
0.547558
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
17
\n",
- "
R1
\n",
- "
Z1
\n",
- "
X
\n",
- "
4.207389
\n",
- "
0.122005
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
18
\n",
- "
R2
\n",
- "
Z3
\n",
- "
Z2
\n",
- "
0.534283
\n",
- "
0.765565
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
19
\n",
- "
Y
\n",
- "
Z2
\n",
- "
Z1|Z3
\n",
- "
1.092994
\n",
- "
0.895382
\n",
- "
4
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
20
\n",
- "
R2
\n",
- "
Z2
\n",
- "
Z1
\n",
- "
0.630084
\n",
- "
0.729758
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
21
\n",
- "
X
\n",
- "
Z3
\n",
- "
Z2
\n",
- "
0.411271
\n",
- "
0.814130
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
22
\n",
- "
M1
\n",
- "
R3
\n",
- "
R2|Y
\n",
- "
9.764665
\n",
- "
0.044584
\n",
- "
4
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
23
\n",
- "
M1
\n",
- "
R2
\n",
- "
R1
\n",
- "
4.138184
\n",
- "
0.126300
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
24
\n",
- "
M1
\n",
- "
Z3
\n",
- "
Z2
\n",
- "
0.532446
\n",
- "
0.766268
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- "
\n",
- "
25
\n",
- "
R1
\n",
- "
X
\n",
- "
M1
\n",
- "
1.280539
\n",
- "
0.527150
\n",
- "
2
\n",
- "
1.000000
\n",
- "
False
\n",
- "
\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " left right given stats p dof p_adj p_adj_significant\n",
- "0 R1 R3 R2|Y 30.915398 0.000003 4 0.000083 True\n",
- "1 Z1 Z3 Z2 9.646506 0.008041 2 0.201015 False\n",
- "2 R3 X R2|Y 2.430030 0.657207 4 1.000000 False\n",
- "3 X Z2 Z1 0.771982 0.679777 2 1.000000 False\n",
- "4 R2 Z1 X 2.994924 0.223697 2 1.000000 False\n",
- "5 R3 Z2 R2|Y 2.677977 0.613074 4 1.000000 False\n",
- "6 R3 Z1 R2|Y 5.641139 0.227599 4 1.000000 False\n",
- "7 M1 Z2 Z1 0.087005 0.957430 2 1.000000 False\n",
- "8 M1 Z1 X 5.765985 0.055967 2 1.000000 False\n",
- "9 R1 Z3 Z2 0.345576 0.841316 2 1.000000 False\n",
- "10 Y Z1 X|Z2 4.006201 0.405167 4 1.000000 False\n",
- "11 R1 Z2 Z1 2.326265 0.312506 2 1.000000 False\n",
- "12 X Y M1|Z2 9.170047 0.056987 4 1.000000 False\n",
- "13 R3 Z3 R2|Y 4.481456 0.344752 4 1.000000 False\n",
- "14 R1 Y M1 2.001399 0.367622 2 1.000000 False\n",
- "15 R2 Y R1 1.083802 0.581642 2 1.000000 False\n",
- "16 R2 X R1 1.204575 0.547558 2 1.000000 False\n",
- "17 R1 Z1 X 4.207389 0.122005 2 1.000000 False\n",
- "18 R2 Z3 Z2 0.534283 0.765565 2 1.000000 False\n",
- "19 Y Z2 Z1|Z3 1.092994 0.895382 4 1.000000 False\n",
- "20 R2 Z2 Z1 0.630084 0.729758 2 1.000000 False\n",
- "21 X Z3 Z2 0.411271 0.814130 2 1.000000 False\n",
- "22 M1 R3 R2|Y 9.764665 0.044584 4 1.000000 False\n",
- "23 M1 R2 R1 4.138184 0.126300 2 1.000000 False\n",
- "24 M1 Z3 Z2 0.532446 0.766268 2 1.000000 False\n",
- "25 R1 X M1 1.280539 0.527150 2 1.000000 False"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "# print_graph_falsifications(graph, data, method=\"chi-square\", verbose=True, significance_level=0.01)\n",
- "print_graph_falsifications(\n",
- " graph, data_trans, method=\"chi-square\", verbose=True, significance_level=0.01\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8b40df88c10664e3",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "All the d-separations implied by the network are validated by the data. No test failed. Hence, we can proceed to step 2."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4435aa5f3a8f55b9",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "## Step 2: Check query identifiability\n",
- "\n",
- "The causal query of interest is the average treatment effect of $X$ on $Y$, defined as:\n",
- "$E[Y|do(X=1)] - E[Y|do(X=0)]$."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "6a1d0da707ca1d2f",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:36:23.240315Z",
- "start_time": "2024-01-25T14:36:23.157745Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "text/latex": [
- "$\\sum\\limits_{M_1, Z_1, Z_2, Z_3} P(M_1 | X, Z_1) P(Y | M_1, X, Z_1, Z_2, Z_3) P(Z_2 | Z_1) P(Z_3 | Z_1, Z_2) \\sum\\limits_{M_1, X, Y, Z_2, Z_3} \\sum\\limits_{R_1, R_2, R_3} P(M_1, R_1, R_2, R_3, X, Y, Z_1, Z_2, Z_3)$"
- ],
- "text/plain": [
- "Sum[M1, Z1, Z2, Z3](P(M1 | X, Z1) * P(Y | M1, X, Z1, Z2, Z3) * P(Z2 | Z1) * P(Z3 | Z1, Z2) * Sum[M1, X, Y, Z2, Z3](Sum[R1, R2, R3](P(M1, R1, R2, R3, X, Y, Z1, Z2, Z3))))"
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "identify_outcomes(graph=graph, treatments=X, outcomes=Y)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5e27342ad3164b42",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "The query is identifiable. Hence we can proceed to step 3."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2ed3073afbc7030a",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "## Step 3: Find nuisance variables and mark them as latent"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b7661cedf357ad01",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "This function finds the nuisance variables for the input graph."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "c094ba6186dfecf6",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:36:23.241863Z",
- "start_time": "2024-01-25T14:36:23.174588Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "{R1, R2, R3}"
- ]
- },
- "execution_count": 10,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "find_nuisance_variables(graph, treatments=X, outcomes=Y)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "79a36765bb1640f6",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "The nuisance variables are $R_1$, $R_2$, and $R_3$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b86a9300fa3d3881",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "## Step 4: Simplify the network"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "92e386966d986cec",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "The following function finds the nuisance variable (step 3), marks them as latent and then applies Evan's simplification rules to remove the nuisance variables. As a result, running the 'find_nuisance_variables' and 'mark_nuisance_variables_as_latent' functions in step 3 is not necessary to get the value of step 4. However, we called them to illustrate the results. The new graph obtained in step 4 does not contain the nuisance variables. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "id": "e1d0f93c0d993112",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:36:23.712654Z",
- "start_time": "2024-01-25T14:36:23.192581Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAGFCAYAAABg2vAPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABE3klEQVR4nO3deVhUZf8/8Dc77oh7KKioj4m4gLg9mT6uYJmmuZQLw2KalrlrLon7LlbmrtWTfivz6sk0NDXT6qu5oCiM+VOwwNyV0VQYZpi5f3/45TyMLM7AzJyD5/26rrkuBmbO+Rz9zOHNfZ/FRQghQERERKrlKncBREREJC+GASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilWMYICIiUjmGASIiIpVjGCAiIlI5hgEiIiKVYxggIiJSOYYBIiIilXOXuwAqOZ1Oh9OnTyMxMRFJSUm4e/cu9Ho9DAYDPD094e3tjWrVqqFVq1YIDQ1FSEgIqlatKnfZJCP2DNmKPaMOLkIIIXcRZB2TyYS9e/di+/btOHHiBC5fvmzzMho2bIi2bdti6NChiIiIgJubmwMqJaVgz5Ct2DPqxDBQBty+fRtbtmzB+vXrkZ6ebrflBgQEYPTo0YiJiUGNGjXstlySH3uGbMWeUTeGAQXLyMjAzJkzsWPHDhgMBoufeXt7S8Nybdq0QWhoKPz9/eHt7Q1PT08YDAbo9Xqkp6cjMTFReiQlJUGv11ssy9PTE4MHD8aCBQvg7+/vzE0kO2PPkK3YMwQAEKQ4ZrNZbNy4UVSqVEkAkB4uLi4iIiJC7N69WxiNxhIt22g0it27d4uIiAjh4uJisfxKlSqJTZs2CbPZbOctIkdjz5Ct2DOUH8OAwqSnp4uePXtafHh8fX3FlClTRFpaml3XlZqaKqZMmSJ8fX0t1tezZ0+Rnp5u13WR47BnyFbsGXoSw4CCbNu2rUBKj46OFvfu3XPoenU6nYiOji6Q3rdv3+7Q9VLpsWfIVuwZKgzDgEKsWrXK4kNSt25dsXfvXqfWkJCQIPz8/CzqiI+Pd2oNZD32DNmKPUNFYRhQgHnz5ll8MCIjIx2e0oui0+nEiBEjLOqZP3++LLVQ0dgzZCv2DBWHYUBmTyb1uXPnyn5gjdlsFnFxcUzuCsWeIVuxZ+hpGAZktG3bNosPwooVK+QuycKKFSss6uPcnvzYM2Qr9gxZg2FAJunp6RYH8cTFxcldUqHmzJljcbAPj/6VD3uGbMWeIWsxDMjAbDaLHj16WMzdyT1kVxSz2Wwxt9erVy/F1vosY8+QrdgzZAuGARls3LjR4mheuQ7isZZOp7M4+nfTpk1yl6Q67BmyFXuGbMEw4GRPDtslJCTIXZJVEhISOIwnE/YM2Yo9Q7ZiGHCy4cOHW1zooyyJioqSah8+fLjc5agGe4ZsxZ4hWzEMONHNmzeFp6enAB5f+tPaYbtFixZZHG1r6+PDDz+0S/06nU66pKinp6e4deuWXZZLRStpzxQlKytLVK9eXQAQrVu3furrf//9d1GlShUBQFSpUkX8/vvvNq2PPeN83M9QSbiCnGbr1q3SXcFiYmJQpUoVq953+vTpUq23RYsWpXp/Hh8fH0RHRwMADAYDtm7dapflUtFK2jNFKVeuHMaOHQsAOHPmDH788cciX5uZmYk+ffrg/v37cHNzw5dffommTZvatD72jPNxP0MlwVsYO4nJZEJgYCDS09Ph4uKCS5cuITAw0Kr3Xr58GVlZWVa99sGDBxg8eDCuXLkCAGjfvj1++ukneHt7l7j2/NLS0tCoUSMAQP369ZGamgo3Nze7LJsslaZninPnzh34+/sjOzsbvXr1wr59+wq8Jjc3F7169cKhQ4cAAKtXr8a7775bovWxZ5yH+xkqMbmHJtRi9+7d0nBaRESEQ9aRnZ0tunTpIq2nRYsWIjMz0+7rCQ8Pl9axZ88euy+fHnNkz4wZM0Za9tmzZ4v9+ciRI0u9PvaMc3A/QyXFMOAkQ4YMkRp79+7ddl++wWAQvXv3ltbRuHFjcePGDbuvRwghvvvuO2k9Q4YMccg6yLE9k5qaKtzc3Ao9SGvdunXSejt37iwMBkOp18eecQ7uZ6ikGAacpGHDhgKA8Pb2Fkaj0a7LNplMYvDgwdIHx9/f36Gn5BiNRuHt7S0AiMDAQIetp6xJS0sTw4YNE3PmzBF37twp9fIc2TNCCDFw4EABQHh4eIiMjAwhhBCHDh0S7u7uAoBo2LChXbZDCPZMUc6dOyeGDBkilixZIu7fv1/q5XE/QyXFMOAEmZmZ0geoffv2dl9+bGystPxatWqJixcv2n0dT2rXrp20Tp1O5/D1lQWjRo2S/k0qVqwoZsyYUeJfpo7uGSGEOHHihLSOiRMnitTUVFGtWjUBQFSuXFlotVq7ro89U1C/fv2kfxNfX1+xcOHCEocC7meoNHg2gRPkP0o3NDTUrsueOHEiNm/eDACoWrUq9u/fj8aNG9t1HYXJvx2lPQr5WZGZmSl9/fDhQyxatAj169fHzJkzcffuXZuW5cieyRMWFoYuXboAADZt2oQ+ffrg7t270pkDzZo1s+v62DMF5e+ZzMxMzJw5Ew0aNMCiRYvw999/27Qs7meoNBgGnCAxMVH6uk2bNnZb7ty5cxEfHw8AqFixIhISEux2es/T5N+O/NtHlkoaChzVM0+aMmUKgMdHh//+++8AgOXLlyMiIsLu62LPWKekoYD7GSoNhgEnSEpKkr62V2JfvXo14uLiAABeXl7YtWsX2rdvb5dlWyP/dpw5c8Zp6y2rbA0FjuiZwvTu3Rv+/v7S85iYGEyYMMEh62LP2MbWUMD9DJWGu9wFqEH+HX/+HW9JbdmyBRMnTgQAuLu7Y8eOHejatWupl2uL/NuRN9SZlZWFPXv2QKj00hV551wXJy8UrFq1Cr169cLbb7+N6tWrF3jdn3/+KX1tj54pysaNG5GRkSE979ixo8PWlX870tPTLX55qdXDhw+f+pq8ULBs2TKMGDEC8+bNg4+PT4HXqWU/Qw4i90ELavDiiy9KB8Ho9fpSLeurr74Srq6uAoBwdXUV27dvt/q9n3/+uXjzzTdFaGiodLnSTz75pER1ZGdnW5x+ptPppFPV+LDvo7Q9U5Qff/xROnMg7/H888877Nax+XuGj5I/ypUrV+jpnkrYz/z1118iPj5e9OjRQ9SrV094eHiIWrVqif79+4vffvvN5jqe3M+Q4zAMOEH79u2lhi7Njvb7778XHh4e0rLWr19v0/sDAgIEAFG9enXp65KGAbPZLNXRvn17iyPT+bDvwxG/nC9evCiqVq0qAAgfHx8xcuRIaX3fffed3dcnhGXP8FG6R2FnqShhPzNt2jQBPD4VMCYmRkyfPl0MGDBAuLm5CVdXV/Hll1/aVMuT+xlyHE4TOIGnp6f0tcFggJeXl83LOHLkCF577TUYjUYAwLJlyzBq1CiblrF582Y0btwYAQEBWLJkCd577z2b68iTk5Mjfe3l5YWwsDDMmDEDu3btUvU0wYMHD6x+vbu7O4KCgtChQ4cCP9u1axeuX78OoOQ9U5R79+6hT58+0Ol0cHd3x9dff42wsDB8+eWXePDgAZYtW4Y+ffrYbX158vdMnTp10LdvX7uvo6zJ//9sjcqVK2PGjBmoVq1agZ8pYT/Ttm1bHD58GJ07d7b4/i+//IJu3brhrbfeQr9+/ayu7cn9DDmQ3GlEDXr27Cml25Lcde7kyZMW9yafOXNmqWtavHixAEo+MqDT6aR6evXqVep6ngV5F/F52sPPz0+sWbNGZGdnF7ms0vZMUYxGo+jevbu07DVr1kg/mzRpkvT9Y8eO2W2dedgzBeUf2i/uERgYKD799NNiLySkxP1MYfWdPHnS6vewZ5yHZxM4Qf4Un/9gLWtotVqEh4dLf3G+8847WLBggV3rK4n82+Hr6ytjJWWHn58f1qxZg9TUVIwdO7bYm7qUpmeK8+677+LgwYMAgDFjxkh3MASACRMmwMPDA8Dj0wvtjT1ju8DAQHz66ae4cOECIiMj4e5e9GCu0vczeb1V3DY8iT3jPAwDTtCqVSvpa1vOlU1LS0OPHj2ko4QjIyPxwQcf2Lu8Esm/Ha1bt5axEuWzJQTkKWnPFGfNmjVYu3YtAKB79+4FesnPzw9vvPEGAODbb7/FpUuX7LLePOwZ69kSAvIoeT+TkZGBgwcPok6dOggODrb6fewZ52EYcIL858qeOnXKqvdcvXoV3bt3l+YTBwwYgC1btsDFxcUhNdoq/3Y48jz4ssTV1fLjVJIQkKckPVOc/fv3Y/z48QCAJk2aYMeOHYX+gpkyZQpcXFxgNpuxcuXKUq83P/ZMQU/2TElCQB6l7meMRiOGDx+OnJwcLF261KZbEbNnnIcHEDpBSEiI9LU1iV2n06F79+7SueZBQUGYMWOGdIU4a/j7+6Ny5co212qt/NuRf/vUrH///ti5cydq166N9957DzExMSW+v7utPVOcCxcuYNCgQTCZTKhatSp2796NqlWrFvraoKAg9O7dG99//z0+++wzzJs3DzVr1izV+vOwZwoaNGgQDh8+jMDAQMyePRtDhw61KQDkp8T9jNlshkajwc8//4yRI0di+PDhVi8bYM84ldwHLaiFLXcT++KLL0p96tGJEyeKXUdpDiDk3cSKdv/+fWEymeyyLHvcge7u3buiUaNGAoBwd3cXBw4ceOp7jhw5IvXRrFmzSrTeJ7Fninbv3j27nT6qpP2MyWQSkZGRAoAYNmyYzZ8L9oxzcZrASdq2bQsA0Ov12LdvX7GvTU5OLtW6PDw8HHrt8L1790Kv1wN4fLMb+q/KlSsXGPotKVt6pjBGoxEDBgxAamoqAOCDDz5A9+7dn/q+F198Ubrk7Nq1a/Ho0SOb1/0k9kzRqlSpYrdheaXsZ8xmM6KiovDZZ5/h9ddfx6effmrz54I942RypxG12L17t5SmIyIi5C6nVCMD4eHh0rbs2bPH/sWREEJ5PVMa7BnnUELPmEwmMWLECAFADB48WOTm5pZoOewZ52IYcJLc3Fzpqn8uLi4iNTVV1npKGgZSU1OlD2j9+vVL/EGnp1Naz5QUe8Z55O6Z/FMDAwcOLPH0FnvG+ThN4CRubm4YPXo0AEAIgQ0bNji9hs2bN0Oj0UCj0eDrr78u8L28+5UXZ/369dLXo0ePtunIYLKNEnrGHtgzziN3z8ybNw+fffYZKlasiCZNmmDBggWIi4uzeFhzgyr2jAxkDiOqcuvWLekGQb6+vna9spw18hJ7UY/IyMhi36/T6YSvr68AIDw9PcWtW7ecU7iKyd0zpcWecT45e+Zp+xhYMRrJnpEHw4CTDR8+XPpQREdHy12OTaKioqTahw8fLnc5qsGeIVuxZ8hWDANOlp6ebnH974SEBLlLssr3338v1VypUiWRkZEhd0mqwZ4hW7FnyFYMAzLYuHGj1PB+fn6KH/rV6XTCz89PqnnTpk1yl6Q67BmyFXuGbMEwIAOz2Wxxh7HIyEiH3LPeHsxms3SaEPD4zmFKrfVZxp4hW7FnyBYMAzJ5chgvLi5O7pIKNWfOHA7bKQR7hmzFniFrMQzIaNu2bRZH2a5YsULukiysWLHCor7t27fLXZLqsWfIVuwZsgbDgMzi4+MtPghxcXGyD4+ZzWaLpA5ArF69Wtaa6L/YM2Qr9gw9DcOAAsyfP9/iAzFixAih0+lkqUWn01nM3QEQ8+fPl6UWKhp7hmzFnqHiMAwoxKpVqyw+GH5+fk4/HSghIcHiaF4AIj4+3qk1kPXYM2Qr9gwVhWFAQbZt22ZxsE/eBUMcnd51Op3FhT4AiMqVK3Purgxgz5Ct2DNUGIYBhUlPT7c4HQh4fEnRyZMn2/2mI6mpqWLy5MnSpT/zn9bDo3nLDvYM2Yo9Q09iGFAgs9ksNm3aVCC9u7i4iPDwcPHdd9+V+G5gRqNRfPfddxa3B82f0jdv3iz7gUVkHZPJJPbu3SuuXLnCniGrGAwGsWfPHnHz5k32DFlwEUIIkCJlZGRg1qxZ+Oqrr2AwGCx+5u3tjZYtWyI0NBRt2rRBaGgo/P394e3tDS8vL+Tk5ECv1yMjIwOJiYk4deoUEhMTcfbsWej1eotleXp6YvDgwVi4cCHq1avnzE2kUhgxYgQ+//xzNGjQAJcuXYKbmxt7hopkNpvRu3dv/PDDD2jbti2OHz8OgPsZ+j9ypxF6ulu3boklS5ZI9ym316N+/fpiyZIlvCtYGbRz507p/7Fu3boFfs6eoSd9/PHH0v9jaGhogZ+zZ9SNIwNliMlkwr59+7Bt2zacPHkSaWlpNi8jMDAQYWFhGDZsGMLDw3mf8DLo9u3bCAoKwu3btwEA27dvxxtvvFHoa9kzBACXL19GixYt8OjRIwDAgQMH0L1790Jfy55RJ4aBMuzevXs4ffo0EhMTcebMGWRmZkKv1yMnJwdeXl7w9vaGr68vWrdujdDQUISEhMDHx0fusqmUBg0ahK+//hoA0K9fP3zzzTdwcXGx6r3sGfUxm83o2rUrjhw5AgAYNWoU1q9fb/X72TPqwDBAVIbs2LEDgwcPBgBUq1YNWq0WtWrVkrkqUrKPPvoI48aNAwAEBAQgOTkZlSpVkrkqUhqGAaIy4ubNmwgKCsLdu3cBAF9++aUUDIgKk5qaihYtWiA7OxsA8OOPP6Jr164yV0VK5Cp3AUT0dEIIvPXWW1IQeO211zBo0CCZqyIlM5vNiIqKkoLA2LFjGQSoSBwZICoDvvjiC+kgwerVq0Or1aJmzZoyV0VKtnr1akyYMAEA0KBBA5w7dw4VK1aUuSpSKoYBIoW7fv06goKCoNPpADw+bmDgwIEyV0VKdvHiRbRs2VI61//w4cPo3LmzzFWRknGagEjBhBAYNWqUFAQGDx7MIEDFMplM0Gg0UhAYN24cgwA9FUcGiBTs888/x4gRIwAANWvWhFarRfXq1WWuipRsxYoVmDJlCgCgUaNGSEpKQoUKFWSuipSOYYBIoa5du4agoCDcu3cPAPDNN9/g1VdflbcoUrQLFy6gVatWyMnJgYuLC37++We88MILcpdFZQCnCYgUSAiBN998UwoCb7zxBoMAFSs3NxeRkZHIyckBAIwfP55BgKzGkQEiBfr0008RFRUFAKhduzZSUlJQrVo1masiJVu6dCmmT58OAGjSpAnOnDmD8uXLy1wVlRUMA0QK89dffyEoKAh///03AGDXrl145ZVXZK6KlEyr1SIkJAQGgwGurq749ddf0aFDB7nLojKE0wRECiKEQGxsrBQERowYwSBAxcrNzYVGo5FuPzxp0iQGAbIZRwaIFGTLli2IjY0FADz33HNISUlB1apVZa6KlGzRokWYOXMmAKBp06Y4c+YMvL29Za6KyhqGASKFyMjIQPPmzfHgwQMAwJ49e/DSSy/JXBUpWXJyMkJDQ2E0GuHq6oqjR4+iXbt2cpdFZRCnCYgUQAiBmJgYKQhERUUxCFCxjEYjIiMjYTQaAQBTp05lEKAS48gAkQJs2LABo0ePBgDUrVsXycnJvCc8FWvevHmYM2cOACAoKAiJiYnw8vKSuSoqqxgGiGT2559/Ijg4GA8fPgQA7Nu3D7169ZK5KlKypKQkhIWFITc3F25ubvjtt9/Qpk0bucuiMozTBEQyMpvNiImJkYJAbGwsgwAVy2AwQKPRIDc3FwAwffp0BgEqNYYBIhmtX78ehw4dAgDUq1cPK1eulLkiUrqFCxfi7NmzAIDg4GDMnj1b5oroWcBpAiKZXL58GcHBwcjKygIAHDhwAN27d5e5KlKy06dPo23btjCZTHB3d8fx48cREhIid1n0DODIAJEMzGYzoqOjpSAwevRoBgEqVk5ODjQaDUwmEwBg5syZDAJkNwwDRDL4+OOPceTIEQBAQEAAli1bJnNFpHTz589HcnIyAKBly5aYMWOGzBXRs4TTBEROlpqaihYtWiA7OxsA8OOPP6Jr164yV0VKdvLkSXTo0EGaHjh16hRatmwpd1n0DOHIAJETmUwmaDQaKQiMHTuWQYCKpdfrLaYH3n//fQYBsjuGASIn+vDDD/G///u/AICGDRtiyZIlMldEShcXF4fz588DAEJCQqTbFBPZE6cJiJzk4sWLaNmyJfR6PQDgyJEjePHFF2WuipTs+PHj6NixI8xmMzw8PHD69Gk0b95c7rLoGcSRASInyJseyAsC48aNYxCgYmVnZ0Oj0cBsNgN4PELAIECOwjBA5ATx8fE4duwYAKBRo0ZYtGiRzBWR0r3//vu4cOECAKBNmzaYOnWqzBXRs4zTBEQO9vvvv6N169bIycmBi4sLfv75Z7zwwgtyl0UKdvToUbzwwgsQQsDT0xNnzpxBs2bN5C6LnmEcGSByoNzcXGg0GuTk5AAAJkyYwCBAxcrKyoJGo0He32nz589nECCHYxggcqCVK1fixIkTAIAmTZpgwYIFMldESjdr1ixcunQJANCuXTtMmjRJ5opIDThNQOQgWq0WISEhMBgMcHV1xa+//ooOHTrIXRYp2C+//ILOnTtDCAEvLy8kJSWhadOmcpdFKsCRASIHMBqNiIyMhMFgAABMmjSJQYCK9ejRI0RFRUnTAwsXLmQQIKdhGCBygGXLliExMREA0LRpU8ybN0/mikjp3nvvPaSlpQEAOnbsiPHjx8tbEKkKpwmI7Cw5ORmhoaEwGo1wdXXFsWPH0LZtW7nLIgU7cuQIunTpAgAoV64ckpKS0KRJE3mLIlXhyACRHeVNDxiNRgDA1KlTGQSoWA8fPkRUVJT0fNGiRQwC5HQMA0R2tHjxYpw5cwYAEBQUhLi4OHkLIsWbNm0a/vjjDwBAp06dMG7cOJkrIjXiNAGRnSQlJSEsLAy5ublwc3PDb7/9hjZt2shdFinYoUOH0K1bNwBA+fLlcfbsWTRq1EjmqkiNODJAZAcGgwEajQa5ubkAHh8MxiBAxXnw4AGio6Ol50uXLmUQINkwDBDZwcKFC3H27FkAQHBwMGbPni1zRaR0U6ZMQXp6OgCgS5cuGDNmjMwVkZpxmoColE6fPo22bdvCZDLB3d0dx48fR0hIiNxlkYLt378fvXr1AgBUqFAB586dQ8OGDWWuitSMIwNEpZCTk4PIyEiYTCYAwMyZMxkEqFj3799HbGys9Hz58uUMAiQ7hgGiUpg/fz5SUlIAAK1atcKMGTNkroiUbvLkybhy5QoAoFu3bhg1apTMFRFxmoCoxE6ePIkOHTrAZDLBw8MDJ0+eRMuWLeUuixRs3759iIiIAABUrFgRKSkpCAgIkLkqIo4MEJWIXq+HRqORpgdmz57NIEDFunfvnsX0wMqVKxkESDEYBohKIC4uDufPnwcAhISEYPr06TJXREo3YcIEXL16FQDQo0cPjBw5UuaKiP6L0wRENvrtt9/wz3/+E2azGR4eHjh9+jSaN28ud1mkYN9//z1efvllAEDlypWRkpKCevXqyVwV0X9xZIDIBtnZ2YiKioLZbAYAzJ07l0GAiqXT6SxGAeLj4xkESHEYBohs8P777+PChQsAgLCwMEyZMkXmikjp3n33XVy/fh0AEBERYXFTIiKl4DQBkZWOHj2KF154AUIIeHp64syZM2jWrJncZZGC7dq1C/369QMAVKlSBSkpKahbt668RREVgiMDRFbIysqCRqNBXnaeP38+gwAV6+7duxbXEPjggw8YBEixGAaIrDBr1ixcunQJANC+fXtMmjRJ5opI6caNG4ebN28CAF5++WWMGDFC5oqIisZpAqKn+OWXX9C5c2cIIeDt7Y2kpCT84x//kLssUrBvvvkGAwYMAABUrVoVKSkpeO6552SuiqhoHBkgKsajR48QFRUlTQ8sWLCAQYCKdfv2bYwePVp6/uGHHzIIkOIxDBAV47333kNaWhoAoGPHjhg/fry8BZHivf3227h9+zYAoG/fvhg6dKjMFRE9HacJiIpw+PBh/Otf/wIAlCtXDklJSWjSpInMVZGSff311xg0aBAAwNfXF1qtFrVr15a5KqKn48gAUSEePnyI6Oho6fnixYsZBKhYt27dwpgxY6TnH3/8MYMAlRkMA0SFmDZtGv744w8AQKdOnfDOO+/IXBEpmRACY8aMwZ07dwAA/fv3x+DBg2Wuish6nCYgesKPP/6I7t27AwDKly+Ps2fPolGjRjJXRUr25Zdf4vXXXwcAVK9eHVqtFjVr1pS5KiLrcWSAKJ8HDx4gJiZGer506VIGASrWjRs3MHbsWOn52rVrGQSozGEYIMpnypQpSE9PBwB06dLFYg6Y6ElCCIwePRqZmZkAgEGDBmHgwIEyV0VkO04TEP2f/fv3o1evXgCAChUqIDk5GQ0aNJC5KlKy7du3Y9iwYQCAmjVrQqvVonr16jJXRWQ7jgwQAbh//z5iY2Ol58uXL2cQoGJdu3bN4sDSdevWMQhQmcUwQARg0qRJuHLlCgCgW7duFjeYIXqSEAKjRo2CTqcDALz++uvo37+/zFURlRynCUj19u3bh4iICABAxYoVkZKSgoCAAJmrIiX797//jcjISABArVq1oNVqUa1aNZmrIio5jgyQqt27d89iemDVqlUMAlSsq1evYty4cdLzDRs2MAhQmccwQKo2YcIEXL16FQDQs2dPi2BA9CQhBEaOHIn79+8DAIYPH46+ffvKXBVR6XGagFRrz5496NOnDwCgcuXKSElJQb169WSuipRs69at0nUo6tSpA61Wi6pVq8pcFVHpcWSAVEmn0+HNN9+UnsfHxzMIULGuXLmCCRMmSM83btzIIEDPDIYBUqV3330X169fBwBEREQgKipK5opIyYQQiI2Nxd9//w0A0Gg0ePnll2Wuish+OE1AqrNr1y7069cPAFClShVotVr4+fnJWxQp2qZNm6SRJD8/P6SkpMDHx0feoojsiGGAVOXu3bsICgrCzZs3AQCffvqpdIoYUWHS09PRvHlzPHz4EACQkJAgnYpK9KzgNAGpyjvvvCMFgZdffhkjRoyQuSJSMiEEYmJipCAQExPDIEDPJI4MkGp88803GDBgAACgatWqSElJwXPPPSdzVaRk69evx1tvvQUAqFevHpKTk1GlShWZqyKyP4YBUoXbt28jKCgIt2/fBgBs27YNQ4cOlbkqUrI//vgDwcHBePToEYDHN7Lq0aOHzFUROQanCUgV3n77bSkI9OvXD2+88YbMFZGSmc1mREdHS0Fg1KhRDAL0TOPIAD3zvv76awwaNAgA4OvrC61Wi9q1a8tcFSnZmjVrpDsSBgQEIDk5GZUqVZK5KiLHYRigZ9qtW7cQFBSEO3fuAAC++OILDBkyROaqSMnS0tLQokULZGVlAQAOHjyIbt26yVwVkWNxmoCeWUIIjBkzRgoCAwYMwODBg2WuipTMbDYjKipKCgJjxoxhECBV4MgAPbO+/PJLvP766wCA6tWrQ6vVombNmjJXRUr2wQcfYPz48QCABg0a4Ny5c6hYsaK8RRE5AcMAPZNu3LiBoKAgZGZmAgB27NiBgQMHylwVKdnFixfRqlUrZGdnAwB++ukndOnSRd6iiJyE0wT0zBFCYPTo0VIQGDRoEIMAFctkMiEqKkoKAu+88w6DAKkKRwbombN9+3YMGzYMAFCzZk1otVpUr15d5qpIyVatWoVJkyYBAAIDA3H27FlUqFBB5qqInIdhgJ4p165dQ/PmzaHT6QA8vurgq6++KnNVpGQXLlxA69atodfr4eLigiNHjqBTp05yl0XkVJwmoGeGEAKjRo2SgsAbb7zBIEDFMplM0Gg00Ov1AIDx48czCJAqcWSAnhmfffYZNBoNAKBWrVrQarWoVq2avEWRoi1btgzTpk0DADRu3BhJSUkoX768zFUROR/DAD0Trl69iqCgINy/fx8A8O2336Jv374yV0VKdv78eYSEhCAnJwcuLi749ddf0bFjR7nLIpIFpwmozBNCYOTIkVIQGD58OIMAFSs3NxcajQY5OTkAgEmTJjEIkKpxZIDKvK1btyImJgYAUKdOHWi1WlStWlXmqkjJFi9ejBkzZgAAmjZtitOnT6NcuXIyV0UkH4YBKtMyMjIQHByMv//+GwCwZ88evPTSSzJXRUqWnJyM0NBQGI1GuLq64ujRo2jXrp3cZRHJitMEVGYJIRAbGysFAY1GwyBAxTIajdBoNDAajQCAKVOmMAgQgSMDVIZt2rQJb775JgDAz88PKSkp8PHxkbcoUrQFCxZg9uzZAIBmzZohMTER3t7eMldFJD+GASqT0tPT0bx5czx8+BAAsHfvXoSHh8tcFSnZ2bNnERYWBqPRCDc3Nxw7dgxhYWFyl0WkCJwmoDLHbDYjOjpaCgKxsbEMAlQsg8FgMT0wffp0BgGifBgGqMzZsGEDDh06BACoV68eVq5cKXNFpHSLFi1CUlISAKB58+bSVAERPcZpAipT/vjjDwQHB+PRo0cAgP3796NHjx4yV0VKdubMGbRt2xa5ublwc3PDiRMnEBISIndZRIrCkQEqM/KmB/KCwKhRoxgEqFgGgwGRkZHIzc0FAMycOZNBgKgQDANUZqxduxaHDx8GAAQEBGD58uXyFkSKN3/+fCQnJwMAWrZsiZkzZ8pcEZEycZqAyoS0tDS0aNECWVlZAIAff/wRXbt2lbkqUrJTp06hffv2MJlMcHd3x8mTJ9GqVSu5yyJSJI4MkOKZzWZERUVJQWDMmDEMAlSsnJwcaDQamEwmAMDs2bMZBIiKwTBAivfRRx/hl19+AQA0aNAAS5culbkiUrq5c+dCq9UCAFq3bo333ntP5oqIlI3TBKRoFy9eRKtWrZCdnQ0AOHz4MDp37ixzVaRkJ06cQIcOHWA2m+Hh4YHExEQEBwfLXRaRonFkgBTLZDIhKipKCgLjxo1jEKBi6fV6REZGwmw2AwDi4uIYBIiswDBAirV69WocPXoUABAYGIhFixbJXBEp3fvvv48LFy4AANq0aYOpU6fKXBFR2cBpAlKkCxcuoHXr1tDr9XBxccGRI0fQqVMnucsiBTt27BheeOEFmM1meHp64vTp0wgKCpK7LKIygSMDpDgmkwkajQZ6vR4AMH78eAYBKlZ2djY0Go00PTBv3jwGASIbMAyQ4qxcuRLHjx8HADRp0gQLFiyQuSJSulmzZuHixYsAgHbt2mHSpEkyV0RUtnCagBTl/PnzaN26NQwGA1xcXPDrr7+iY8eOcpdFCvbrr7/ixRdfhBACXl5eSEpKQtOmTeUui6hM4cgAKUZubi40Gg0MBgMAYNKkSQwCVKysrCxERUUh72+aBQsWMAgQlQDDACnG8uXLcfLkSQBA06ZNMW/ePJkrIqWbMWMGUlNTAQAdOnTAhAkTZK6IqGziNAEpQnJyMkJDQ2E0GuHq6oqjR4+iXbt2cpdFCnbkyBF06dIFAODt7Y2zZ8+iSZMm8hZFVEZxZIBkZzQaodFoYDQaAQBTp05lEKBiPXz4ENHR0dLzxYsXMwgQlQLDAMluyZIlOH36NACgWbNmiIuLk7cgUrzp06fj8uXLAIAXXngB48aNk7kiorKN0wQkq7NnzyIsLAxGoxFubm44duwYwsLC5C6LFOynn36S7lpZrlw5nDt3Do0aNZK5KqKyjSMDJBuDwWAxPTB9+nQGASrWgwcPLKYHli5dyiBAZAcMAySbRYsWISkpCQAQHByM2bNny1sQKd7UqVPx559/AgA6d+6MsWPHylsQ0TOC0wQkizNnzqBt27bIzc2Fm5sbTpw4gZCQELnLIgU7ePAgevToAQCoUKECzp07h4YNG8pcFdGzgSMD5HQGgwGRkZHIzc0FAMycOZNBgIr1999/IyYmRnq+bNkyBgEiO2IYIKebP38+kpOTAQAtW7bEzJkzZa6IlG7y5MnIyMgAAHTt2hWjR4+WuSKiZwunCcipTp06hfbt28NkMsHd3R2nTp1Cy5Yt5S6LFOyHH35AeHg4AKBixYpITk5G/fr15S2K6BnDkQFympycHGg0GphMJgDA+++/zyBAxbp//z5iY2Ol5ytXrmQQIHIAhgFymrlz50Kr1QIAQkJCMH36dJkrIqWbOHEi/vrrLwBAjx49MHLkSJkrIno2cZqAnOLEiRPo0KEDzGYzPDw8kJiYiODgYLnLIgVLSEjASy+9BACoVKkSUlJS4O/vL3NVRM8mjgyQw+n1ekRGRsJsNgMA4uLiGASoWDqdzmIUID4+nkGAyIEYBsjh3n//fVy4cAEA0KZNG0ydOlXmikjpxo8fj2vXrgEAwsPDLa46SET2x2kCcqhjx47hn//8J4QQ8PT0xJkzZ9CsWTO5yyIF2717N1555RUAQJUqVZCSkoK6devKXBXRs40jA+Qw2dnZ0Gg0yMub8+bNYxCgYmVmZuLNN9+Unq9evZpBgMgJGAbIYWbNmoWLFy8CANq1a4dJkybJXBEp3bhx43Djxg0AwEsvvYTIyEiZKyJSB04TkEP8+uuvePHFFyGEgJeXF5KSktC0aVO5yyIF+89//oP+/fsDAHx8fKDVavHcc8/JXBWROnBkgOwuKysLUVFR0vTAwoULGQSoWHfu3LG4xPBHH33EIEDkRAwDZHczZsxAamoqAKBjx44YP368vAWR4r3zzju4desWAOCVV17B0KFDZa6ISF04TUB2deTIEXTp0gUA4O3tjbNnz6JJkybyFkWKtnPnTgwcOBAA4OvrC61Wi9q1a8tcFZG6cGSA7Obhw4cW54MvXryYQYCKdevWLbz11lvS8zVr1jAIEMmAYYDsZvr06bh8+TIAoFOnThg3bpzMFZGSCSEwZswY3LlzBwDQv39/DBkyROaqiNSJ0wRkF4cOHUK3bt0AAOXKlcO5c+fQqFEjmasiJfvqq6+kX/7VqlWDVqtFrVq1ZK6KSJ04MkCl9uDBA8TExEjPly5dyiBAxbp58ybGjh0rPV+7di2DAJGMGAao1KZOnYo///wTANC5c2eLnTzRk4QQeOutt3D37l0AwMCBAzFo0CCZqyJSN04TUKkcOHAAPXv2BABUqFAB586dQ8OGDWWuipTsf/7nf6RTB2vUqAGtVosaNWrIXBWRunFkgErs77//tpgeWL58OYMAFev69et4++23pefr1q1jECBSAIYBKtbJkyfRoUMHjBkzBjqdzuJnkydPxpUrVwAAXbt2xahRo+QokRTm4MGDaNu2LaZOnYpHjx5J3xdCYNSoUVIfDRkyBAMGDJCrTCLKh9MEVKz+/fvjP//5DwDgueeew6ZNm9C7d2/88MMPCA8PBwBUrFgRycnJqF+/voyVklL885//xNGjRwEAgYGB+OSTT9CpUyd8/vnnGDFiBACgVq1a0Gq1qFatmpylEtH/YRigYjVv3hxardbie0OHDsWhQ4dw/fp1AMCGDRssbjtL6lajRg3p2gEA4OLigtjYWOzYsQP3798HAHz77bfo27evXCUS0RMYBqhIQghUrFgRWVlZRb6mR48e+OGHH+Di4uLEykipHj58iEqVKhX7mmHDhuHzzz93UkVEZA0eM0BFun37drFBAACqV6+OBw8eOKkiUro//vjjqa/x8fFBdna2E6ohImsxDFCRrNmxf/HFFwgODsbBgwedUBEpnTU9s2bNGoSEhOD48eNOqIiIrMEwQEXKu5DQ02RkZKBHjx4YP348OOukbtb2zIULF9CxY0csWLDAsQURkVUYBqhI1vyVl98HH3zAv/ZUzpaeMZvNmD17ts19RkT2xzBARbJ1Jx0QEMB7EqicrT0THBzMexIQKQDDABXJ2h27h4cHJk+ejKSkJFSvXt3BVZGSWdsz5cuXx7x58/Dbb7+hfPnyDq6KiJ7GXe4CSLmsmf/t378/li1bhsDAQMcXRIomhLCqZyIjI7Fw4UL4+fk5vigisgqvM0CFMpvNKFeuHAwGQ6E/b926NeLj49G5c2cnV0ZKlZmZWewVBTt16oT4+HiEhoY6sSoisganCahQ165dKzQI1K5dG1u3bsXJkycZBMhCWlpaod9v0KABdu7ciSNHjjAIECkUpwmoUImJiRbPvb29MXnyZEybNg0VK1aUqSpSsp9//tnieaVKlTBr1iyMGzcO3t7eMlVFRNZgGCjDdDodTp8+jcTERCQlJeHu3bvQ6/UwGAzw9PSEt7c3qlWrhlatWiE0NBQhISGoWrWqVctu1aoVXF1dYTab0alTJ2zbtg3+/v4O3iJyNEf2TMeOHaWv+/Tpg82bN6NmzZqO2hRyEkf2DCkHjxkoQ0wmE/bu3Yvt27fjxIkTuHz5ss3LaNiwIdq2bYuhQ4ciIiICbm5uRb72xo0buHPnDpo3b16asklGzu6Z9PR05OTkoEmTJqUpm2Tk7J4hZWAYKANu376NLVu2YP369UhPT7fbcgMCAjB69GjExMSgRo0adlsuyY89Q7Ziz6gbw4CCZWRkYObMmdixY0eBg/m8vb2lYbk2bdogNDQU/v7+8Pb2hqenJwwGA/R6PdLT05GYmCg9kpKSoNfrLZbl6emJwYMHY8GCBZwKKOPYM2Qr9gwBAAQpjtlsFhs3bhSVKlUSAKSHi4uLiIiIELt37xZGo7FEyzYajWL37t0iIiJCuLi4WCy/UqVKYtOmTcJsNtt5i8jR2DNkK/YM5ccwoDDp6emiZ8+eFh8eX19fMWXKFJGWlmbXdaWmpoopU6YIX19fi/X17NlTpKen23Vd5DjsGbIVe4aexDCgINu2bSuQ0qOjo8W9e/ccul6dTieio6MLpPft27c7dL1UeuwZshV7hgrDMKAQq1atsviQ1K1bV+zdu9epNSQkJAg/Pz+LOuLj451aA1mPPUO2Ys9QURgGFGDevHkWH4zIyEiHp/Si6HQ6MWLECIt65s+fL0stVDT2DNmKPUPFYRiQ2ZNJfe7cubIfWGM2m0VcXByTu0KxZ8hW7Bl6GoYBGW3bts3ig7BixQq5S7KwYsUKi/o4tyc/9gzZij1D1mAYkEl6errFQTxxcXFyl1SoOXPmWBzsw6N/5cOeIVuxZ8haDAMyMJvNokePHhZzd3IP2RXFbDZbzO316tVLsbU+y9gzZCv2DNmCYUAGGzdutDiaV66DeKyl0+ksjv7dtGmT3CWpDnuGbMWeIVswDDjZk8N2CQkJcpdklYSEBA7jyYQ9Q7Ziz5CtGAacbPjw4RYX+ihLoqKipNqHDx8udzmqwZ4hW7FnyFYMA0508+ZN4enpKYDHl/60Zthu3bp1Fkfa+vv7W7WuPXv2WLwPgPjrr79KVb9Op5MuKerp6Slu3bpVquXR05WkZ4QQYtGiRQX+/215fPjhh3apnz3jfCXtmSdlZmaKihUrStMM1tynwGg0SscpuLi4iK+++srm9bJn5OEKcpqtW7dKdwWLiYlBlSpVnvqepKQki+dXrlzBw4cPi32PyWTC1KlTLb5Xo0YN+Pn52VbwE3x8fBAdHQ0AMBgM2Lp1a6mWR09Xkp4BgNOnT5dqvS1atCjV+/OwZ5yvpD3zpKpVqyI2NhYA8Ndff2Hnzp1Pfc/48eNx4MABAMCcOXMwaNAgm9fLnpGJ3GlELXJzc0VAQICUmFNTU616X7t27QQAUb58eemvtuPHjxf7ng0bNkipOu893bt3t8dmiNTUVGmZ9evXF7m5uXZZLhVU0p4RQoi0tDSRnJxs1ePo0aOiXr160v9r+/btRXZ2tt22gz3jPKXpmcKkp6cLd3d3AUC0a9eu2NeuXbtW+n8ePHhwqdbLnnE+hgEn2b17t9TcERERVr3HZDJJIaBfv37S11u3bi3yPQ8ePBC1a9cWAMRrr70mrXPy5Mn22hQRHh4uLXfPnj12Wy5ZKknP2Co7O1t06dJFWk+LFi1EZmam3dfDnnEOR/TMsGHDpGUeO3as0NccPHhQCg1hYWEiKyur1OtlzzgXpwmcZPv27dLXY8aMseo9Fy9eRFZWFgCgdevWCAoKAgCcP3++yPcsX74cN27cgK+vL1566SXp+61atSpB1YXLX/+2bdvstlyyVJKesYXRaMSAAQNw+PBhAEDjxo2xf/9+VK1a1e7rYs84hyN6Jv+U4+rVqwv8/OLFixg4cCByc3Ph5+eHXbt2oVy5cqVeL3vGyeROI2rRsGFDAUB4e3tbdSCOEEJ88cUXUjLetWuXdPvPohL/1atXRYUKFQTw+Brf+a/qpdVq7bYtRqNReHt7CwAiMDDQbsslSyXpGWuZTCYxePBgiwNTHXkaF3umcDt37hT+/v4iPDxcXLlypdTLc1TPRERECADC3d3dok6dTieaNGkiTWUmJibabZ3sGediGHCCzMxMi/lYa02bNk16X3p6uli9enWxZxTExMRIHxyDwSD69u0rAIhy5crZfc4t71gGAEKn09l12VTynrFWbGystPxatWqJixcv2n0dT2LPFFSnTh3p38TFxUX06NGjxKHAkT3z008/ScueNm2aEOLxL+vu3btLte/cudOu6xSCPeNMnCZwgvxHdoeGhlr9vrwzCapWrQp/f3/pCO/CzihISUnBp59+CgBYsmQJPDw8pPc3b94cbm5uJd+AQuTfjtIeuU4FlbRnrDFx4kRs3rwZwOPe2r9/Pxo3bmzXdRSGPVNQ3jQgAAghcODAAfj7+6Nnz57466+/bFqWI3umS5cuaNu2LQBg06ZNyMrKwrvvvouDBw8CAObNm4cBAwbYdZ0Ae8aZGAacIDExUfq6TZs2Vr8v75d5y5YtAfz3dC8hRIHjBiZPngyTyYQOHTrgtddew71795Ceng7AvscL5Mm/Hfm3j+yjpD3zNHPnzkV8fDwAoGLFikhISLDbaYRPw56xTklDgaN6Js+UKVMAAJmZmejTpw/Wrl0LAHj99dcxa9Ysu68PYM84E8OAE+S/VoC1if3GjRu4efMmgP/+Mq9WrRrq1KkDANBqtdJrDxw4gB9++AEAsGLFigLrdEQYyL8dZ86csfvy1a4kPfM0q1evRlxcHADAy8sLu3btQvv27e2ybGuwZ2xjayhwRM/k179/fzRq1AgAcOjQIQBAu3btHHodAPaM87jLXYAa3L17V/ra39/fqvfk/2DnjQwAj0cHrl+/Lo0MmM1mKbEPGDAAHTt2LPD+1q1bl7T0IuXfjszMTACPj04/f/48hBB2X5/a/Pnnn9LX1vZMcbZs2YKJEycCANzd3bFjxw507dq11Mu1Rf7tSE9PR1JSEvR6vTSCpUZ5FwcqTl4oqFevHlq0aIFdu3ahfv36BV5Xkv2MLVxdXTF58mSMHj0aAFC3bl18++238Pb2tvu68hS2nyHHYBhwAr1eL31t7QenqL/sW7RogR9++EEaGfj3v/+Ns2fPwsPDA0uWLCnwfldX1wLDwNu2bcMvv/yCxMREJCcnw2Aw4JNPPoFGo7F6m/Jvh16vh16vR9OmTVW9Y3eU0u5sd+zYgTfffBNCCLi6uuKzzz7DK6+88tT36fV6zJgxA6dOnUJqaioyMzPh4+ODwMBAxMbGYtiwYfDw8LC6jvzbcfToUYeE1GfduXPn0KBBAzx69Ajly5e3+FlJ9jO2CgwMlL4ePXo0ateu7ZD15HlyP0OOw2kCJ8if/j09Pa16T94vcw8PDzRr1kz6fnBwMIDH0wTZ2dnSXN2YMWOkIbz872/UqBEqVKhgsexZs2Zh48aNSE9Pl6YdbOXl5SV9nZOTgzt37iAjI6NEy6LiWdszhUlISMCwYcNgNpsBAGvXrsUbb7xh1XsfPnyIdevWwcXFBS+99BImTpyIV199FVevXkV0dDRefvllabnWyN8zVDqFTRmUZD9jK0dPPz7pyf0MOQ5HBpwg/wfTYDBYtVPM+9A9//zzFu/Pf0bB3LlzcfXqVfj4+GD27NkW68ibRijsA7t582Y0btwYAQEBWLJkCd577z2btyn/B9PLy0saMty3bx+nCexg165duH79OgDre+ZJR44cwWuvvQaj0QgAWLZsGUaNGmX1+319fXH//v0Cv1hyc3PRo0cP7N+/H3v37rW4uFVx8vdMnTp10LdvX1y6dEnaTjW6cOGCTYEKAMLDw9GkSZMC3y/JfsZWZ8+elb52Rhh4cj9DjsMw4ARPDnU9ramzsrJw6dIlAAU/cM8//zw8PDxgNBqxfPlyAMCMGTNQrVo16TXnz5+XfgEU9oHt3r17STbDQmFDkq+88opVw8/0dJcvX5Z+SVrTM086deoU+vTpg+zsbADAzJkzpWNLrOXq6lroX5ju7u549dVXcfjwYaSmplq9vPw906JFC6xbt86mep5FPj4+uH///lNf5+7uDo1Gg48++qjIKQBb9zMlkfdHij1ufGYNZ0x90GOcJnCC/L+orRlKP3funPTXwpO/zD09PfGPf/wDwOODBwMCAjBu3DiL1zj64EHAcjt8fX0dsg41s7Vn8tNqtQgPD8eDBw8AAO+88w4WLFhgt9rMZjP27dsH4PE1LKzFnrGdu7s7YmNj8eDBA2zatKnYX4il6Rlr5OTk4MKFCwAsD2p2JPaM8zAMOEH+X+jWnCtb1JkEeUJCQuDl5QUvLy8sXry4wF8AzpjXy78dPBDM/mztmTxpaWno0aOHdGR5ZGQkPvjgg1LVYjAYEBcXhzlz5uDtt99GUFAQ9u7di6ioKHTr1s3q5bBnrGdLCMhT0p6xllarRW5uboF1ORJ7xnkYBpwg/7myp06deurrn/bL/LPPPpOO4H/99deLfH+tWrUcdrRv/u1wxDnNamdrzwDA1atX0b17d2l6YcCAAdiyZQtcXFxKVYvBYMDcuXMxb948fPzxx/h//+//YfLkydi4caNNy2HPFPTk/01JQkCekvSMLZx98CDAnnEmHjPgBCEhIdLXtowM1KtXr0RDY3kH+TjyA5t/O/JvH9mHrT2j0+nQvXt36foEQUFBmDFjBn7//Xer1+nv74/KlSsX+H7FihUhhIDZbMa1a9ewe/duzJgxA8eOHUNCQkKh7ykMe6agXr164auvvrLqmICnsbVnbOXsgwcB9oxTyXZXBJWx9m5iJpNJlC9fXgAQL7/8ss3r+eOPPwrcUKQ4ixcvFgDEJ598YvU6eDcx57DlDnT573BZ0seJEyesrm3Hjh0CgJg6dapVr2fPFC09PV2YTCa7LMuRd7p88cUXpWXb+8ZnhWHPOBenCZwk7yYfer1eOviqMBcvXpRuXlKS9O2Moby9e/dKR/mGhYU5ZB1kfc8AQHJycqnW5eHhYdM9Cnr27AkAOHz4sFWvZ88Uzd/fH66u9tkV29Iztjp37hwAx9z4rDDsGediGHCSoUOHSl/n3eCjME2bNoV4fGtpzJ8/3+b19OvXT3r/kCFDSlTr0+Svf9iwYQ5ZB1nfMwCwcOFC6f+9JA9bz0u/du0aAFh9BUL2jHPY0jO20ul0EELg5MmTdl1uUdgzzsUw4CQREREICAgAAOzbtw9paWkyV1QyaWlp0l8c9evXR3h4uMwVPbvk7pnz589b3GI3T1ZWlnSfg969ez91OewZ55G7Z+yFPeN8DANO4ubmJt3gQwiBDRs2yFbL5s2bodFooNFo8PXXXxf4Xt697guzfv166evRo0c7ZbhQreTumR07dqB27dro3bs3xowZg+nTp2P48OHw9/fHvn370KlTJ0yYMOGpy2HPOI/cPWMv7BkZOPH4BNW7deuW8PT0FACEr6+vuHfvnix1REZGFnsgWWRkZKHv0+l0wtfXVwAQnp6e4tatW84tXIXk7JmTJ0+KkSNHiqCgIOHj4yPc3d1FtWrVxL/+9S+xYcMGqw5QY884n1L2MyXFnpEHw4CTDR8+XPqlGx0dLXc5NomKipJqHz58uNzlqAZ7hmzFniFbMQw4WXp6uqhUqZLU7AkJCXKXZJXvv/9eqrlSpUoiIyND7pJUgz1DtmLPkK0YBmSwceNGqeH9/PwUP4yn0+mEn5+fVPOmTZvkLkl12DNkK/YM2YJhQAZms1n07NnTYo7ebDbLXVahzGazGDFihFRrr169FFvrs4w9Q7Ziz5AtGAZk8uQwXlxcnNwlFWrOnDkctlMI9gzZij1D1mIYkNG2bdssjuJfsWKF3CVZWLFihUV927dvl7sk1WPPkK3YM2QNhgGZxcfHW3wQ4uLiZB8eM5vNFkkdgFi9erWsNdF/sWfIVuwZehqGAQWYP3++xQdixIgRQqfTyVKLTqezmLsDIObPny9LLVQ09gzZij1DxWEYUIhVq1ZZfDD8/PycfjpQQkKCxdG8AER8fLxTayDrsWfIVuwZKgrDgIJs27bN4mCfvAuGODq963Q6iwt9ABCVK1fm3F0ZwJ4hW7FnqDAMAwqTnp5ucToQ8PiSopMnTxapqal2XVdqaqqYPHmydOnP/Kf18GjesoM9Q7Ziz9CTGAYUyGw2i02bNhVI7y4uLiI8PFx89913Vl0XvjBGo1F89913Ijw83GLZeSl98+bNsh9YRLZjz5Ct2DOUn4sQQoAUKSMjA7NmzcJXX30Fg8Fg8TNvb2+0bNkSoaGhaNOmDUJDQ+Hv7w9vb294eXkhJycHer0eGRkZSExMxKlTp5CYmIizZ89Cr9dbLMvT0xODBw/GwoULUa9ePWduItkZe4ZsxZ4hAGAYKANu376NrVu3Yt26dUhPT7fbcuvXr4/Ro0cjOjoaNWrUsNtySX7sGbIVe0bdGAbKEJPJhH379mHbtm04efIk0tLSbF5GYGAgwsLCMGzYMISHh/M+4c849gzZij2jTgwDZdi9e/dw+vRpJCYm4syZM8jMzIRer0dOTg68vLzg7e0NX19ftG7dGqGhoQgJCYGPj4/cZZOM2DNkK/aMOjAMEBERqZyr3AUQERGRvBgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhUjmGAiIhI5RgGiIiIVI5hgIiISOUYBoiIiFSOYYCIiEjlGAaIiIhU7v8DnZn7rJtbzvsAAAAASUVORK5CYII=",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "new_graph = remove_nuisance_variables(graph, treatments=X, outcomes=Y)\n",
- "new_graph.draw()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f1b8adaf922f3096",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "## Step 5: Estimate the query"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "id": "55a147fe3a3ee98d",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:36:23.977103Z",
- "start_time": "2024-01-25T14:36:23.690832Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "1.1927594503118613"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "ATE_value = estimate_ace(graph=new_graph, treatments=X, outcomes=Y, data=data)\n",
- "ATE_value"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8b1fbdecffb16328",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "The ATE amounts to 0.21 meaning that the average effect that $X$ has on $Y$ is negative."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e362c47381bc281f",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "# Evaluation Criterion\n",
- "\n",
- "As we used synthetic data set, we were able to generate two interventional data sets where in\n",
- "one X was set to 1, and the other one X is set to 0. The ATE was calculated by subtracting the average value of Y obtained from each interventional data,\n",
- "resulting in the ground truth ATE=0.01. The ATE indicates that increase in X can increase Y levels."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "id": "7aa1b23565adae07",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:36:24.001406Z",
- "start_time": "2024-01-25T14:36:23.981263Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The true ATE is 0.49\n"
- ]
- }
- ],
- "source": [
- "def get_background_ace(seed=None) -> float:\n",
- " data_1 = example.generate_data(1000, {X: 1.0}, seed=seed)\n",
- " data_0 = example.generate_data(1000, {X: 0.0}, seed=seed)\n",
- " return data_1.mean()[Y.name] - data_0.mean()[Y.name]\n",
- "\n",
- "\n",
- "true_ate = get_background_ace(seed=SEED)\n",
- "print(f\"The true ATE is {true_ate:.04}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1de8f8091cc1f8c9",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "### Random Sampling Evaluation"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6949dceb5dff8017",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "Instead of relying on a single point estimate of ATE, such as what we did in the previous section, in this part, we sample from the observational data (D) several times. For each sub-sampled data, we calculate the ATE for the new graph and original graph to examine the effect of removing the nuisance variables from the graph."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "id": "8290a6c96fb3b26a",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:36:24.064421Z",
- "start_time": "2024-01-25T14:36:23.993866Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [],
- "source": [
- "# Population => Generate D = 10000 data points\n",
- "D = example.generate_data(10000, seed=SEED)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "7335a9ba0f7818cd",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:36:24.587225Z",
- "start_time": "2024-01-25T14:36:24.016244Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [],
- "source": [
- "# Samples => Generate 1000 datasets with 1000 points each (d) using random sampling\n",
- "\n",
- "d_count = 1000\n",
- "d_size = 1000\n",
- "d = [D.sample(d_size) for _ in range(d_count)]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "f7e71ed8912c89bb",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:40:55.102266Z",
- "start_time": "2024-01-25T14:36:24.597029Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [],
- "source": [
- "# Estimate the query in the form of ATE for the new graph\n",
- "ate_new_graph = [estimate_ace(new_graph, treatments=X, outcomes=Y, data=data) for data in d]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "id": "a9bf7f5040ce2225",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:41:11.077521Z",
- "start_time": "2024-01-25T14:40:55.124037Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [],
- "source": [
- "# Estimate the query in the form of expected value (E[Y|do(X=0)]\n",
- "expected_value_new_graph = [\n",
- " estimate_query(\n",
- " new_graph, data, treatments=X, outcome=Y, interventions={X: 0}, query_type=\"expected_value\"\n",
- " )\n",
- " for data in d\n",
- "]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "id": "2bf7dde37179a515",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:41:11.109398Z",
- "start_time": "2024-01-25T14:41:11.080185Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "1.138909517506263"
- ]
- },
- "execution_count": 18,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "np.var(ate_new_graph)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "id": "2193cf2bfe7deb09",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:41:11.188745Z",
- "start_time": "2024-01-25T14:41:11.094143Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "0.774284708230741"
- ]
- },
- "execution_count": 19,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "np.var(expected_value_new_graph)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "id": "337f8cacfd6cd318",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:48:32.685840Z",
- "start_time": "2024-01-25T14:41:11.131360Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [],
- "source": [
- "# Estimate the query in the form of ATE for the original graph\n",
- "ate_graph = [estimate_ace(graph, treatments=X, outcomes=Y, data=data) for data in d]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "id": "90e8e51eacfcd61d",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:48:46.693915Z",
- "start_time": "2024-01-25T14:48:32.693355Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [],
- "source": [
- "# Estimate the query in the form of expected value (E[Y|do(X=0)]\n",
- "expected_value_graph = [\n",
- " estimate_query(\n",
- " graph, data, treatments=X, outcome=Y, interventions={X: 0}, query_type=\"expected_value\"\n",
- " )\n",
- " for data in d\n",
- "]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "id": "afcf227ffe06440",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:48:46.719692Z",
- "start_time": "2024-01-25T14:48:46.695854Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "1.1389095175062627"
- ]
- },
- "execution_count": 22,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "np.var(ate_graph)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "id": "c244aaf67982988",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:48:46.720306Z",
- "start_time": "2024-01-25T14:48:46.708205Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "0.774284708230741"
- ]
- },
- "execution_count": 23,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "np.var(expected_value_graph)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4f8902e34011dba9",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "We can see that the variance of the estimated ATEs from the new graph where the nuisance variables are removed is smaller than the original graph. However, the difference between the two is extremly small. This can be shown in the box plots below, where the difference is not visible by eye. Despite the marginal improvement in precision favoring the simplified graph, the process of detecting and removing nuisance variables can result in more interpretable and less complex networks."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "id": "d3a6d3c4e69adfbb",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2024-01-25T14:20:46.692063Z",
- "start_time": "2024-01-25T14:20:46.184537Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
+ }
+ ],
+ "source": [
+ "if reduced_graph is None:\n",
+ " print(\"Graph was not reduced, no need to re-check identifiability\")\n",
+ "else:\n",
+ " eliater.step_2_notebook(graph=reduced_graph, treatment=treatment, outcome=outcome)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "id": "5830ffa8-a794-4e05-b450-26c5e621d889",
+ "metadata": {},
+ "outputs": [
{
"data": {
+ "text/markdown": [
+ "## Step 5: Estimate the Query\n",
+ "\n",
+ "### Calculating the True Average Treatment Effect (ATE)\n",
+ "\n",
+ "We first generated synthetic observational data. Now, we generate two interventional datasets:\n",
+ "one where we set $X$ to $0.0$ and one where we set $X$ to $1.0$.\n",
+ "We can then calculate the \"true\" average treatment effect (ATE) as the difference of the means\n",
+ "for the outcome variable $Y$ in each. The ATE is formulated as:\n",
+ "\n",
+ "$ATE = \\mathbb{E}[Y \\mid do(X = 1)] - \\mathbb{E}[Y \\mid do(X = 0)]$\n",
+ "\n",
+ "After generating 10,000 samples for each distribution, we took 500 subsamples of size\n",
+ "of size 1,000 and calculated the\n",
+ "ATE for each. The variance comes to 1.7e-30, which shows that the ATE is very stable with respect\n",
+ "to random generation. We therefore calculate the _true_ ATE as the average value from these samplings,\n",
+ "which comes to 4.9e-01.\n",
+ "\n",
+ "The ATE can be interpreted in the following way:\n",
+ "\n",
+ "1. If the ATE is positive, it suggests that the treatment $X$ has a negative effect on the outcome $Y$\n",
+ "2. If the ATE is negative, it suggests that the treatment $X$ has a positive effect on the outcome $Y$\n",
+ "\n",
+ "### Estimating the Average Treatment Effect (ATE)\n",
+ "\n",
+ "In practice, we are often unable to get the appropriate interventional data, and therefore want to estimate\n",
+ "the average treatment effect (ATE) from observational data. Because we're using synthetic data, we generate\n",
+ "10,000 samples, then took 500 subsamples of size 1,000 through which we calculated\n",
+ "the following:\n",
+ "\n",
+ "1. The ATE, using the y0/ananke implementation\n",
+ "2. The ATE, using the Eliater linear regression implementation\n"
+ ],
"text/plain": [
- ""
+ ""
]
},
- "execution_count": 24,
"metadata": {},
- "output_type": "execute_result"
+ "output_type": "display_data"
},
{
"data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGdCAYAAADqsoKGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAz0klEQVR4nO3de1yUdd7/8feAAiIwiiKHDQ94PixuaXljZVp5q7WueXfbCTdxt3po2kGz0kcUYppWdl5TK80Tq50sa/Nwm2VUa2aWh3ItLUw2wQMoCCTYzPX7ox+zjiDO4IzXzMXr+Xhcj7iu6zvX9QH7Dm+u6zvfy2YYhiEAAIAgF2J2AQAAAL5AqAEAAJZAqAEAAJZAqAEAAJZAqAEAAJZAqAEAAJZAqAEAAJZAqAEAAJbQyOwCzien06kDBw4oOjpaNpvN7HIAAIAHDMPQ8ePHlZSUpJCQM1+PaVCh5sCBA0pOTja7DAAAUA/5+fm64IILzri/QYWa6OhoSb/9UGJiYkyuBgAAeKK0tFTJycmu3+Nn0qBCTfUtp5iYGEINAABB5mxDRxgoDAAALIFQAwAALIFQAwAALKFBjakBADQchmHo119/lcPhMLsUnEVoaKgaNWp0ztOtEGoAAJZTVVWlgoICVVRUmF0KPBQZGanExESFhYXV+xiEGgCApTidTuXl5Sk0NFRJSUkKCwtjwtUAZhiGqqqqdPjwYeXl5aljx451TrBXF0INAMBSqqqq5HQ6lZycrMjISLPLgQeaNGmixo0b66efflJVVZUiIiLqdZygGSg8depU2Ww2t6VLly5mlwUACFD1/Wsf5vDFv1dQXanp3r27PvjgA9d6o0ZBVT7OI4fDoR07dqi4uFixsbFKTU1VaGio2WUB8AH6N84kqFJBo0aNlJCQYHYZCHC5ubl68cUXVVhY6NqWkJCgO++8U/369TOxMgDniv6NugTVtbk9e/YoKSlJKSkpSk9P1/79++tsX1lZqdLSUrcF1pabm6usrCylpKRozpw5Wr16tebMmaOUlBRlZWUpNzfX7BIB1BP9G2cTNKGmT58+WrRokdauXau5c+cqLy9Pl19+uY4fP37G18ycOVN2u9218IRua3M4HHrxxReVlpam6dOnq3v37oqMjFT37t01ffp0paWlae7cucxZAQShhtC/Tx83evoyderU817T8uXLFRoaqnHjxrm29e/fv846+/fvL0lq27ZtrftnzZrlt3qDJtQMGTJEI0aMUGpqqgYNGqTVq1fr2LFjev3118/4milTpqikpMS15Ofnn8eKcb7t2LFDhYWFSk9PrzHgLCQkROnp6SooKNCOHTtMqhBAfTWE/l1QUOBann32WcXExLhtmzRpkqtt9cSC/rZgwQI98MADWr58uU6cOCFJWrlypaumL774QpL0wQcfuLatXLnS9fpp06a5fQ8FBQW66667/FZv0ISa0zVr1kydOnXS3r17z9gmPDzc9URunsxtfcXFxZKkdu3a1bq/ent1OwDB45z7t2FI5eXmLIbh0feYkJDgWux2u2w2m2t99+7dio6O1po1a9SrVy+Fh4fr008/VUZGhq677jq349x7772uqyXSb/P2zJw5U+3atVOTJk3Us2dPvfnmm2etJy8vT//85z81efJkderUyRVWYmNjXXXFxcVJklq0aOHaFhsb6zpGdHS02/eVkJCgpk2bevTzqI+gGih8qrKyMv3www/685//bHYpCBDVHSkvL0/du3evsT8vL8+tHYDgcc79u6JCioryW311KiuTfPSLfPLkyZo9e7ZSUlLUvHlzj14zc+ZMLVu2TPPmzVPHjh2Vm5urkSNHKi4uTldcccUZX/fqq6/q2muvld1u18iRI7VgwQLdcsstPvk+/CVortRMmjRJH3/8sfbt26d//vOfGj58uEJDQ3XzzTebXRoCRGpqqhISEpSTkyOn0+m2z+l0KicnR4mJiUpNTTWpQgD1Rf/+zbRp0zRw4EC1b9/eoz/QKisr9dhjj2nhwoUaNGiQUlJSlJGRoZEjR2r+/PlnfJ3T6dSiRYs0cuRISdJNN92kTz/91BUePfXggw8qKirKbfnkk0+8OoY3guZKzb///W/dfPPNKioqUlxcnC677DJ9/vnnrktfQGhoqO68805lZWUpMzNT6enpateunfLy8pSTk6NNmzYpOzub+SyAIORN/z558mTNA0RG/nbFxAw+nNW4d+/eXrXfu3evKioqNHDgQLftVVVVuvDCC8/4uvXr16u8vFzXXHONJKlly5YaOHCgFi5cqEcffdTj899///3KyMhw2/a73/3O82/AS0ETalasWGF2CQgC/fr1U3Z2tl588UW30fqJiYnKzs5mHgsgiJ1T/7bZfHYLyEynj0cJCQmRcdqYnVNDXdn/D3Lvv/9+jTARHh5+xvMsWLBAxcXFatKkiWub0+nUjh07lJ2d7fHsvy1btlSHDh08ausLQRNqAE/169dPl156KTOOAhZE/3YXFxenb775xm3btm3b1LhxY0lSt27dFB4erv3799c5fuZURUVFWrVqlVasWOE2fsnhcOiyyy7T//3f/2nw4MG++yZ8iFADSwoNDa3z0iqA4EX//o8rr7xSTz75pJYsWaK0tDQtW7ZM33zzjevnEx0drUmTJmnChAlyOp267LLLVFJSos8++0wxMTEaNWpUjWMuXbpULVq00A033FDj6ebXXHONFixY4HGoOX78uNvsz5IUGRnpt08jB81AYQAA4G7QoEF6+OGH9cADD+jiiy/W8ePHdeutt7q1efTRR/Xwww9r5syZ6tq1qwYPHqz333//jB+PX7hwoYYPH14j0EjS9ddfr3fffVdHjhzxqL5HHnlEiYmJbssDDzzg/TfqIZtx+s04CystLZXdbldJSQlz1gCARZ04cUJ5eXlq166dIiIizC4HHqrr383T399cqQEAAJZAqAEAAJZAqAEAAJZAqAEAAJZAqAEAWFID+hyMJfji34tQAwCwlOqJ5yoqKkyuBN6o/veq/verDybfAwBYSmhoqJo1a6ZDhw5J+m2yt9rmXEFgMAxDFRUVOnTokJo1a3ZOs0MTagAAlpOQkCBJrmCDwNesWTPXv1t9EWoAAJZjs9mUmJioVq1a1f7UbgSUxo0b++T5XYQaAIBlhYaGNtiHXTZEDBQGAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWQKgBAACWELShZtasWbLZbLr33nvNLgUAAASAoAw1W7Zs0fz585Wammp2KQAAIEAEXagpKytTenq6Xn75ZTVv3tzscgAAQIAIulAzbtw4XXvttbr66qvP2rayslKlpaVuCwAAsKZGZhfgjRUrVuirr77Sli1bPGo/c+ZMZWdn+7kqAAAQCILmSk1+fr7uuece5eTkKCIiwqPXTJkyRSUlJa4lPz/fz1UCAACz2AzDMMwuwhPvvPOOhg8frtDQUNc2h8Mhm82mkJAQVVZWuu2rTWlpqex2u0pKShQTE+PvkgEAgA94+vs7aG4/XXXVVdq5c6fbttGjR6tLly568MEHzxpoAACAtQVNqImOjlaPHj3ctjVt2lQtWrSosR0AADQ8QTOmBgAAoC5Bc6WmNhs3bjS7BAAAECC4UgMAACyBUAMAACyBUAMAACyBUAMAACwhqAcKwz9OnDih/fv3m10GTtG6dWuPZ9IG6kL/Djz0b98h1KCG/fv364477jC7DJzipZdeUqdOncwuAxZA/w489G/fCZrHJPgCj0nwjFX+kvvpp580Y8YMPfTQQ2rTpo3Z5ZwT/pKDr9C/Aw/9++ws95gEnD8RERGW+quhTZs2lvp+gHNB/4aVMVAYAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYAqEGAABYQtCEmrlz5yo1NVUxMTGKiYlRWlqa1qxZY3ZZAAAgQARNqLngggs0a9Ysbd26VV9++aWuvPJKDRs2TN9++63ZpQEAgADQyOwCPDV06FC39RkzZmju3Ln6/PPP1b17d5OqAgAAgSJoQs2pHA6H3njjDZWXlystLe2M7SorK1VZWelaLy0tPR/lAQAAEwTN7SdJ2rlzp6KiohQeHq4xY8bo7bffVrdu3c7YfubMmbLb7a4lOTn5PFYLAADOp6AKNZ07d9a2bdu0efNmjR07VqNGjdKuXbvO2H7KlCkqKSlxLfn5+eexWgAAcD4F1e2nsLAwdejQQZLUq1cvbdmyRc8995zmz59fa/vw8HCFh4efzxIBAIBJgupKzemcTqfbmBkAANBwBc2VmilTpmjIkCFq3bq1jh8/rr///e/auHGj1q1bZ3ZpAAAgAARNqDl06JBuvfVWFRQUyG63KzU1VevWrdPAgQPNLg0AAASAoAk1CxYsMLsEAAAQwIJ6TA0AAEA1Qg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEQg0AALAEj0PNF198IYfDccb9lZWVev31131SFAAAgLc8DjVpaWkqKipyrcfExOjHH390rR87dkw333yzb6sDAADwkMehxjCMOtfPtA0AAOB88OmYGpvN5svDAQAAeIyBwgAAwBIaedN4165dKiwslPTbrabdu3errKxMknTkyBHfVwcAAOAhr0LNVVdd5TZu5o9//KOk3247GYbB7ScAAGAaj0NNXl6eP+sAAAA4Jx6HmsWLF2vSpEmKjIz0Zz0AAAD14vFA4ezsbNf4GTPMnDlTF198saKjo9WqVStdd911+u6770yrBwAABJZ6z1Nzvn388ccaN26cPv/8c61fv14nT57Uf//3f6u8vNzUugAAQGDwaqCwmQOB165d67a+aNEitWrVSlu3blW/fv1MqgoAAAQKr0JNp06dzhpsiouLz6kgT5WUlEiSYmNjz9imsrJSlZWVrvXS0lK/1wUAAMzhVajJzs6W3W73Vy0eczqduvfee3XppZeqR48eZ2w3c+ZMZWdnn8fKAACAWbwKNTfddJNatWrlr1o8Nm7cOH3zzTf69NNP62w3ZcoUTZw40bVeWlqq5ORkf5cHAABM4HGoCZSJ9caPH69//OMfys3N1QUXXFBn2/DwcIWHh5+nygAAgJl88umn0tJSzZ07V7179/ZJUWc6//jx4/X222/rww8/VLt27fx2LgAAEHw8vlLjdDprbPvoo4+0cOFCrVy5Una7XcOHD/dpcacaN26c/v73v2vVqlWKjo52PYPKbrerSZMmfjsvAAAIDl6NqZGkn3/+WYsWLdKrr76qY8eO6ejRo/r73/+uG264wa+3qObOnStJ6t+/v9v2V199VRkZGX47LwAACA4e33566623dM0116hz587atm2bnnrqKR04cEAhISH6/e9/7/cxN4Zh1LoQaAAAgOTFlZobb7xRDz74oF577TVFR0f7syYAAACveXyl5q9//avmzJmjwYMHa968eTp69Kg/6wIAAPCKx6Fm/vz5Kigo0B133KHly5crMTFRw4YNk2EYtQ4iBgAAOJ88DjWS1KRJE40aNUoff/yxdu7cqe7duys+Pl6XXnqpbrnlFq1cudJfdQIAANTJq1Bzqo4dO+qxxx5Tfn6+li1bpoqKCt18882+rA0AAMBjXn+k+3QhISEaOnSohg4dqkOHDvmiJgAAAK/V+0pNbQLhuVAAAKBh8mmoAQAAMAuhBgAAWMI5j6lBTQcPHlRJSYnZZTR4P/30k9t/YS673a74+Hizyzhn9O/AQP8OLIHSv21GXY/ftpjS0lLZ7XaVlJQoJibGL+c4ePCgRv75Vp2sqvTL8YFg1TgsXMuWLgmIN776OnjwoG7980hVVp00uxQgoISHNdaSpcv81r89/f3t0ZWa5s2be/xsp+LiYs8qtKiSkhKdrKrULylXyBlhN7scICCEnCiRfvxYJSUlQR1qSkpKVFl1UmO6HVdSU4fZ5QAB4UB5qObtig6I/u1RqHn22WddXxcVFWn69OkaNGiQ0tLSJEmbNm3SunXr9PDDD/ulyGDkjLDL2bSl2WUA8IOkpg61jSbUAIHGo1AzatQo19fXX3+9pk2bpvHjx7u23X333frb3/6mDz74QBMmTPB9lQAAAGfh9aef1q1bp8GDB9fYPnjwYH3wwQc+KQoAAMBbXoeaFi1aaNWqVTW2r1q1Si1atPBJUQAAAN7y+iPd2dnZuu2227Rx40b16dNHkrR582atXbtWL7/8ss8LBAAA8ITXoSYjI0Ndu3bV888/73oqd9euXfXpp5+6Qg4AAMD5Vq/J9/r06aOcnBxf1wIAAFBv9XpMwg8//KDMzEzdcsstridzr1mzRt9++61PiwMAAPCU16Hm448/1u9//3tt3rxZb731lsrKyiRJ27dvV1ZWls8LBAAA8ITXoWby5MmaPn261q9fr7CwMNf2K6+8Up9//rlPiwMAAPCU16Fm586dGj58eI3trVq10pEjR3xSFAAAgLe8DjXNmjVTQUFBje1ff/21fve73/mkKAAAAG95HWpuuukmPfjggyosLJTNZpPT6dRnn32mSZMm6dZbb/VHjQAAAGfldah57LHH1KVLFyUnJ6usrEzdunVTv3791LdvX2VmZvqjRgAAgLPyep6asLAwvfzyy3rkkUe0c+dOlZWV6cILL1THjh39UR8AAIBHvL5SM23aNFVUVCg5OVnXXHONbrjhBnXs2FG//PKLpk2b5o8aAQAAzsrrUJOdne2am+ZUFRUVys7O9klRAAAA3vI61BiGIZvNVmP79u3bFRsb65OiAAAAvOXxmJrmzZvLZrPJZrOpU6dObsHG4XCorKxMY8aM8UuRAAAAZ+NxqHn22WdlGIb+8pe/KDs7W3a73bUvLCxMbdu2VVpaml+KBAAAOBuPQ82oUaMkSe3atVPfvn3VuHFjvxUFAADgLa8/0n3FFVe4vj5x4oSqqqrc9sfExJx7VQAAAF7yeqBwRUWFxo8fr1atWqlp06Zq3ry52+JPubm5Gjp0qJKSkmSz2fTOO+/49XwAACB4eB1q7r//fn344YeaO3euwsPD9corryg7O1tJSUlasmSJP2p0KS8vV8+ePTVnzhy/ngcAAAQfr28/vffee1qyZIn69++v0aNH6/LLL1eHDh3Upk0b5eTkKD093R91SpKGDBmiIUOG+O34AAAgeHl9paa4uFgpKSmSfhs/U1xcLEm67LLLlJub69vqzlFlZaVKS0vdFgAAYE1eX6lJSUlRXl6eWrdurS5duuj111/XJZdcovfee0/NmjXzQ4n1N3PmTNNmOQ755Zgp5wUCkdX6w4HyULNLAAJGIPUHr0PN6NGjtX37dl1xxRWaPHmyhg4dqr/97W86efKknn76aX/UWG9TpkzRxIkTXeulpaVKTk4+L+dukhdYV60A+M68XdFmlwCgFl6HmgkTJri+vvrqq7V7925t3bpVHTp0UGpqqk+LO1fh4eEKDw835dy/tOsnZ5NmppwbCDQhvxyzVNAf0+24kpo6zC4DCAgHykMDJuh7HWpO16ZNG7Vp08YXtViKs0kzOZu2NLsMAH6Q1NShttGEGiDQ1CvUbNmyRR999JEOHTokp9Ppts+ft6DKysq0d+9e13peXp62bdum2NhYtW7d2m/nBQAAgc/rUPPYY48pMzNTnTt3Vnx8vNuDLWt7ercvffnllxowYIBrvXq8zKhRo7Ro0SK/nhsAAAQ2r0PNc889p4ULFyojI8MP5dStf//+MgzjvJ8XAAAEPq/nqQkJCdGll17qj1oAAADqzetQM2HCBB5TAAAAAo7Xt58mTZqka6+9Vu3bt1e3bt3UuHFjt/0rV670WXEAAACe8jrU3H333froo480YMAAtWjRwu+DgwEAADzhdahZvHix3nrrLV177bX+qAcAAKBevB5TExsbq/bt2/ujFgAAgHrzOtRMnTpVWVlZqqio8Ec9AAAA9eL17afnn39eP/zwg+Lj49W2bdsaA4W/+uornxUHAADgKa9DzXXXXeeHMgAAAM6N16EmKyvLH3UAAACcE6/H1AAAAAQij67UxMbG6vvvv1fLli3VvHnzOuemKS4u9llxAAAAnvIo1DzzzDOKjo52fc2EewAAINB4FGpGjRrl+tqMp3MDAACcjdcDhUNDQ1VQUKBWrVq5bS8qKlKrVq3kcDh8VpzflJdLoaF+ObStokIRDoccJyvlrKr0yzmAYBNyslIRDodsFRW/9b8gVd2/Q6ocslU5zS4HCAghVfJ///bwuDbDMAxvjhsSEqLCwsIaoebAgQNq3769fvnlF28Od16VlpbKbrerRFKM2cUAAACPlEqySyopKVFMzJl/g3t8peb555+XJNlsNr3yyiuKiopy7XM4HMrNzVWXLl3qXTAAAMC58DjUPPPMM5IkwzA0b948hZ5y+yYsLExt27bVvHnzfF+hPxw4INWR9M7Fnj17dNddd6m8yzVyRrb0yzmAYBNScURNd6/WCy+8oI4dO5pdTr1V9+/Mi46pTTS3nwBJ+ul4iKZ/1cy//bu0VEpKOmszj0NNXl6eJGnAgAFauXKlmjdvXv/izNa06W+LHxiRkToRGqoTjcPlDAv3yzmAYBNyMlyhoaEyIiP91vfOh+r+7QwLlRFmdjVAYHCGheqEv/u3h+N1vZ5876OPPnILNA6HQ9u2bdPRo0e9PRQAAIDPeB1q7r33Xi1YsEDSb4GmX79+uuiii5ScnKyNGzf6uj4AAACPeB1q3njjDfXs2VOS9N5772nfvn3avXu3JkyYoIceesjnBQIAAHjC61BTVFSkhIQESdLq1as1YsQIderUSX/5y1+0c+dOnxcIAADgCa9DTXx8vHbt2iWHw6G1a9dq4MCBkqSKigq3T0QBAACcT17PKDx69GjdcMMNSkxMlM1m09VXXy1J2rx5M/PUAAAA03gdaqZOnaoePXooPz9fI0aMUHj4bx9bDg0N1eTJk31eIAAAgCe8DjWS9L//+781tp360EsAAIDzzeMxNddcc41KSkpc67NmzdKxY8dc60VFRerWrZtPiwMAAPCUx6Fm3bp1qqz8z1OnH3vsMRUXF7vWf/31V3333Xe+rQ4AAMBDHt9+Ov1h3l4+3LvBCTlRcvZGQANhtf5woJxPegLVAqk/1GtMDc7MbrercVi49OPHZpcCBJTGYeGy2+1ml3FO7Ha7wsMaa96uaLNLAQJKeFjjgOjfHocam80mm81WYxvcxcfHa9nSJW7jj2COn376STNmzNBDDz2kNm3amF1Og2e32xUfH292GeckPj5eS5Yuo38HAPp3YAmU/u3V7aeMjAzXR7hPnDihMWPGqOn/fyLnqeNtGrr4+PiA+MfFb9q0aaNOnTqZXQYsgv4dWOjfOJXHoeb0j2yPHDmyRptbb7313CsCAACoB49DzauvvurPOjw2Z84cPfnkkyosLFTPnj31wgsv6JJLLjG7LAAAYDKvn/1kptdee00TJ05UVlaWvvrqK/Xs2VODBg3SoUOHzC4NAACYLKhCzdNPP63bb79do0ePVrdu3TRv3jxFRkZq4cKFZpcGAABMFjShpqqqSlu3bnU9QFOSQkJCdPXVV2vTpk21vqayslKlpaVuCwAAsKagCTVHjhyRw+Go8amD+Ph4FRYW1vqamTNnym63u5bk5OTzUSoAADBB0ISa+pgyZYpKSkpcS35+vtklAQAAPwmaGYVbtmyp0NBQHTx40G37wYMHlZCQUOtrwsPDXfPqAAAAawuaKzVhYWHq1auXNmzY4NrmdDq1YcMGpaWlmVgZAAAIBEFzpUaSJk6cqFGjRql379665JJL9Oyzz6q8vFyjR482uzQAAGCyoAo1N954ow4fPqxHHnlEhYWF+sMf/qC1a9cyZTkAAAiuUCNJ48eP1/jx480uAwAABJigGVMDAABQF0INAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwBEINAACwhKAJNTNmzFDfvn0VGRmpZs2amV0OAAAIMEETaqqqqjRixAiNHTvW7FIAAEAAamR2AZ7Kzs6WJC1atMjcQgAAQEAKmlBTH5WVlaqsrHStl5aWmlgNAADwp6C5/VQfM2fOlN1udy3JyclmlwQAAPzE1FAzefJk2Wy2Opfdu3fX+/hTpkxRSUmJa8nPz/dh9QAAIJCYevvpvvvuU0ZGRp1tUlJS6n388PBwhYeH1/v1AAAgeJgaauLi4hQXF2dmCQAAwCKCZqDw/v37VVxcrP3798vhcGjbtm2SpA4dOigqKsrc4gAAgOmCJtQ88sgjWrx4sWv9wgsvlCR99NFH6t+/v0lVAQCAQBE0n35atGiRDMOosRBoAACAFEShBgAAoC6EGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmEGgAAYAmNzC7AE/v27dOjjz6qDz/8UIWFhUpKStLIkSP10EMPKSwszOzyLOfEiRPav3+/2WWcs59++sntv8GsdevWioiIMLsMWAD9O/DQv30nKELN7t275XQ6NX/+fHXo0EHffPONbr/9dpWXl2v27Nlml2c5+/fv1x133GF2GT4zY8YMs0s4Zy+99JI6depkdhmwAPp34KF/+47NMAzD7CLq48knn9TcuXP1448/evya0tJS2e12lZSUKCYmxo/VBbdg/0vuq6++0htvvKGioiLXthYtWmjEiBG66KKLTKys/vhLDr5C/w489O+z8/T3d1BcqalNSUmJYmNj62xTWVmpyspK13ppaam/y7KEiIiIoP2rITc3V/Pnz1daWprS09PVrl075eXlKScnR/Pnz1d2drb69etndpmAaejfsLKgvFKzd+9e9erVS7Nnz9btt99+xnZTp05VdnZ2je1cqbEmh8Oh9PR0paSkaPr06QoJ+c84eKfTqczMTOXl5WnZsmUKDQ01sVIA3qJ/N2yeXqkx9dNPkydPls1mq3PZvXu322t+/vlnDR48WCNGjKgz0EjSlClTVFJS4lry8/P9+e3AZDt27FBhYaHS09Pd3vAkKSQkROnp6SooKNCOHTtMqhBAfdG/4QlTbz/dd999ysjIqLNNSkqK6+sDBw5owIAB6tu3r1566aWzHj88PFzh4eHnWiaCRHFxsSSpXbt2te6v3l7dDkDwoH/DE6aGmri4OMXFxXnU9ueff9aAAQPUq1cvvfrqqzWSOlA9xiovL0/du3evsT8vL8+tHYDgQf+GJ4IiGfz888/q37+/WrdurdmzZ+vw4cMqLCxUYWGh2aUhgKSmpiohIUE5OTlyOp1u+5xOp3JycpSYmKjU1FSTKgRQX/RveCIoQs369eu1d+9ebdiwQRdccIESExNdC1AtNDRUd955pzZt2qTMzEx9++23qqio0LfffqvMzExt2rRJY8eOZRAhEITo3/BEUH76qb6Yp6ZhyM3N1Zw5c3Tw4EHXtoSEBN1555183BMIcvTvhikoPv0E+MOuXbt0+PBht22HDh3Srl27TKoIgK/Qv1GXoJ18D6jNvHnztGLFCjVv3lwDBw5UUlKSDhw4oPXr12vFihWSpDFjxphcJYD6oH/jbLj9BMuoqqrSkCFDFBERoaioKLfL0/Hx8SorK9OJEye0Zs0aHoQKBBn6d8PG7Sc0OKtWrZLD4VB5ebnat2+vOXPmaPXq1ZozZ47at2+v8vJyORwOrVq1yuxSAXiJ/g1PEGpgGT///LMkqXfv3po+fbq6d++uyMhIde/eXdOnT1fv3r3d2gEIHvRveIJQA8vp1KlTrdOod+zY0aSKAPgK/Rt1IdTAMrp27SpJWr16tX799Ve3fb/++qvWrFnj1g5A8KB/wxOEGlhGq1atJEnHjh3TiBEj9N577+nIkSN67733NGLECB07dsytHYDgQf+GJ/hINyyjehr1kJAQFRYW6qmnnnLtCw0NVVJSkgzDYBp1IAjRv+EJQg0so3oa9aysLPXp00e/+93vVFlZqfDwcP3888/avHmzsrOzmUYdCEL0b3iCeWpgObm5uXrxxRfdHniamJiosWPHMo06EOTo3w2Tp7+/CTWwpKqqKq1atUoHDhxQUlKShg0bxoRcgEXQvxseQk0tCDUNQ21/yfHAO8Aa6N8NEzMKo0HKzc1VVlaWUlJS3GYcTUlJUVZWlnJzc80uEUA90b9xNlypgWU4HA6lp6crJSVF06dPd5ugy+l0KjMzU3l5eVq2bBmDCYEgQ/9u2LhSgwZnx44dKiwsVHp6eq0zjqanp6ugoEA7duwwqUIA9UX/hicINbCM4uJiSVK7du1q3V+9vbodgOBB/4YnCDWwjNjYWElSXl5erfurt1e3AxA86N/wBKEGllE942hOTo6cTqfbPqfTqZycHCUmJjLjKBCE6N/wBKEGllE94+imTZuUmZmpb7/9VhUVFfr222+VmZmpTZs2aezYsQwiBIIQ/Rue4NNPsBxmHAWsi/7dMDH5Xi0INQ2Hw+HQjh07VFxcrNjYWKWmpvIXHGAR9O+Gx9Pf3zzQEpYUGhqqCy+80OwyAPgB/RtnwpgaAABgCYQaAABgCYQaAABgCYQaAABgCYQaAABgCYQaAABgCYQaAABgCYQaAABgCYQaAABgCQ1qRuHqJ0KUlpaaXAkAAPBU9e/tsz3ZqUGFmuPHj0uSkpOTTa4EAAB46/jx47Lb7Wfc36AeaOl0OnXgwAFFR0fLZrOZXQ78rLS0VMnJycrPz+cBpoDF0L8bFsMwdPz4cSUlJSkk5MwjZxrUlZqQkBBdcMEFZpeB8ywmJoY3PcCi6N8NR11XaKoxUBgAAFgCoQYAAFgCoQaWFR4erqysLIWHh5tdCgAfo3+jNg1qoDAAALAurtQAAABLINQAAABLINQAAABLINQAfrRo0SI1a9bM7DKAM7LZbHrnnXf8fp7+/fvr3nvvda23bdtWzz77rGu9sLBQAwcOVNOmTV19xhe1ZWRk6LrrrjunY/jTxo0bZbPZdOzYMbNLsQRCDc6JWW8YhAXg7A4fPqyxY8eqdevWCg8PV0JCggYNGqTPPvvM1aagoEBDhgw577Vt2bJFd9xxh2v9mWeeUUFBgbZt26bvv//e1NoQvBrUjMKAJ6qqqhQWFmZ2GcA5u/7661VVVaXFixcrJSVFBw8e1IYNG1RUVORqk5CQYEptcXFxbus//PCDevXqpY4dO7q2mVXb2fAeEbi4UgOf6t+/v+6++2498MADio2NVUJCgqZOnerWxmazae7cuRoyZIiaNGmilJQUvfnmm679tV2O3bZtm2w2m/bt26eNGzdq9OjRKikpkc1mk81mq3GOU02fPl2tWrVSdHS0brvtNk2ePFl/+MMfXPurrzbNmDFDSUlJ6ty5syRp6dKl6t27t6Kjo5WQkKBbbrlFhw4dqlHn+++/r9TUVEVEROi//uu/9M0339SoYd26deratauioqI0ePBgFRQUePeDBbx07NgxffLJJ3r88cc1YMAAtWnTRpdccommTJmiP/3pT652p97i2bdvn2w2m15//XVdfvnlatKkiS6++GJ9//332rJli3r37q2oqCgNGTJEhw8fdh2jug9lZ2crLi5OMTExGjNmjKqqqs5Y36m3n9q2bau33npLS5Yskc1mU0ZGRo3aJCk/P1833HCDmjVrptjYWA0bNkz79u1z7Xc4HJo4caKaNWumFi1a6IEHHjjrU50l6eWXX1ZycrIiIyM1fPhwPf30025XgqdOnao//OEPeuWVV9SuXTtFRERIktauXavLLrvMdb4//vGP+uGHH1yvq/55rlixQn379lVERIR69Oihjz/+uEYNW7duVe/evRUZGam+ffvqu+++O2vdqIlQA59bvHixmjZtqs2bN+uJJ57QtGnTtH79erc2Dz/8sK6//npt375d6enpuummm/Svf/3Lo+P37dtXzz77rGJiYlRQUKCCggJNmjSp1rY5OTmaMWOGHn/8cW3dulWtW7fW3Llza7TbsGGDvvvuO61fv17/+Mc/JEknT57Uo48+qu3bt+udd97Rvn37XG+2p7r//vv11FNPacuWLYqLi9PQoUN18uRJ1/6KigrNnj1bS5cuVW5urvbv33/GegFfiYqKUlRUlN555x1VVlZ69dqsrCxlZmbqq6++UqNGjXTLLbfogQce0HPPPadPPvlEe/fu1SOPPOL2mg0bNuhf//qXNm7cqOXLl2vlypXKzs726HxbtmzR4MGDdcMNN6igoEDPPfdcjTYnT57UoEGDFB0drU8++USfffaZ64+E6vD01FNPadGiRVq4cKE+/fRTFRcX6+23367z3J999pnGjBmje+65R9u2bdPAgQM1Y8aMGu327t2rt956SytXrtS2bdskSeXl5Zo4caK+/PJLbdiwQSEhIRo+fLicTqfba++//37dd999+vrrr5WWlqahQ4e6XS2TpIceekhPPfWUvvzySzVq1Eh/+ctfPPrZ4TQGcA5GjRplDBs2zLV+xRVXGJdddplbm4svvth48MEHXeuSjDFjxri16dOnjzF27FjDMAzjo48+MiQZR48ede3/+uuvDUlGXl6eYRiG8eqrrxp2u/2s9fXp08cYN26c27ZLL73U6Nmzp9v3EB8fb1RWVtZ5rC1bthiSjOPHj7vVuWLFCleboqIio0mTJsZrr73mqlOSsXfvXlebOXPmGPHx8WetHThXb775ptG8eXMjIiLC6Nu3rzFlyhRj+/btbm0kGW+//bZhGIaRl5dnSDJeeeUV1/7ly5cbkowNGza4ts2cOdPo3Lmza33UqFFGbGysUV5e7to2d+5cIyoqynA4HIZh/PbecM8997j2t2nTxnjmmWdc68OGDTNGjRp1xtqWLl1qdO7c2XA6na79lZWVRpMmTYx169YZhmEYiYmJxhNPPOHaf/LkSeOCCy5we4863Y033mhce+21btvS09Pd3l+ysrKMxo0bG4cOHTrjcQzDMA4fPmxIMnbu3GkYxn9+nrNmzapR0+OPP24Yxn/eRz744ANXm/fff9+QZPzyyy91ng81caUGPpeamuq2npiY6HbbRpLS0tJqrHt6pcYb3333nS655BK3baevS9Lvf//7GvfIt27dqqFDh6p169aKjo7WFVdcIUnav3+/W7tTv5fY2Fh17tzZ7XuJjIxU+/btXeu1/TwAf7j++ut14MABvfvuuxo8eLA2btyoiy66SIsWLarzdaf24fj4eEm/9ZFTt53+/3DPnj0VGRnpWk9LS1NZWZny8/N98J1I27dv1969exUdHe26ChUbG6sTJ07ohx9+UElJiQoKCtSnTx/Xaxo1aqTevXvXeVxP3yPatGlTYxzQnj17dPPNNyslJUUxMTFq27atpLrfI6prOv397tSfeWJioiTxPlEPDBSGzzVu3Nht3Waz1bgcW5eQkN+ytnHKvfBTb+f4Q9OmTd3Wy8vLNWjQIA0aNEg5OTmKi4vT/v37NWjQoDrHCdSmtp+HwdNJcJ5ERERo4MCBGjhwoB5++GHddtttysrKqvVWarVT/5+12Wy1bvOmT/tCWVmZevXqpZycnBr7Tg8b/nD6e4QkDR06VG3atNHLL7+spKQkOZ1O9ejRw+v3CKn2n/n5/hlbAVdqYIrPP/+8xnrXrl0l/ecN6tTBtNX3sKuFhYXJ4XCc9TydO3fWli1b3Ladvl6b3bt3q6ioSLNmzdLll1+uLl26nPGvplO/l6NHj+r77793fS9AoOnWrZvKy8t9ftzt27frl19+ca1//vnnioqKUnJysk+Of9FFF2nPnj1q1aqVOnTo4LbY7XbZ7XYlJiZq8+bNrtf8+uuv2rp1a53Hre97RFFRkb777jtlZmbqqquuUteuXXX06NFa2576HlFdE+8R/kGogSneeOMNLVy4UN9//72ysrL0xRdfaPz48ZKkDh06KDk5WVOnTtWePXv0/vvv66mnnnJ7fdu2bVVWVqYNGzboyJEjqqioqPU8d911lxYsWKDFixdrz549mj59unbs2OH6S+hMWrdurbCwML3wwgv68ccf9e677+rRRx+tte20adO0YcMGffPNN8rIyFDLli0DerIvNAxFRUW68sortWzZMu3YsUN5eXl644039MQTT2jYsGE+P19VVZX++te/ateuXVq9erWysrI0fvx415XXc5Wenq6WLVtq2LBh+uSTT5SXl6eNGzfq7rvv1r///W9J0j333KNZs2bpnXfe0e7du3XnnXeedVK7u+66S6tXr9bTTz+tPXv2aP78+VqzZs1Z3yOaN2+uFi1a6KWXXtLevXv14YcfauLEibW2nTNnjt5++23t3r1b48aN09GjRxkI7CeEGpgiOztbK1asUGpqqpYsWaLly5erW7dukn67DLt8+XLt3r1bqampevzxxzV9+nS31/ft21djxozRjTfeqLi4OD3xxBO1nic9PV1TpkzRpEmTdNFFFykvL08ZGRmuj2SeSVxcnBYtWqQ33nhD3bp106xZszR79uxa286aNUv33HOPevXqpcLCQr333nvMYQHTRUVFqU+fPnrmmWfUr18/9ejRQw8//LBuv/12/e1vf/P5+a666ip17NhR/fr104033qg//elPdU614K3IyEjl5uaqdevW+p//+R917dpVf/3rX3XixAnFxMRIku677z79+c9/1qhRo5SWlqbo6GgNHz68zuNeeumlmjdvnp5++mn17NlTa9eu1YQJE876HhESEqIVK1Zo69at6tGjhyZMmKAnn3yy1razZs3SrFmz1LNnT3366ad699131bJly/r9IFAnm8HNfZxnNptNb7/9tmlXMwYOHKiEhAQtXbr0nI6zceNGDRgwQEePHmV2YzRoGRkZOnbs2Hl53ML5cPvtt2v37t365JNPzuk4+/btU7t27fT111+7zY0F/2GgMCytoqJC8+bN06BBgxQaGqrly5frgw8+qDFvDoCGa/bs2a7nTq1Zs0aLFy/Wiy++aHZZqAdCDSzNZrNp9erVmjFjhk6cOKHOnTvrrbfe0tVXX212aQACxBdffKEnnnhCx48fV0pKip5//nnddtttZpeFeuD2EwAAsAQGCgMAAEsg1AAAAEsg1AAAAEsg1AAAAEsg1AAAAEsg1AAAAEsg1AAAAEsg1AAAAEsg1AAAAEv4fwQhYif317m8AAAAAElFTkSuQmCC",
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "",
+ "version_major": 2,
+ "version_minor": 0
+ },
"text/plain": [
- ""
+ "Subsampling: 0%| | 0/500 [00:00\n",
+ "\n",
+ "\n"
+ ],
"text/plain": [
- " 0%| | 0/500 [00:00"
]
},
"metadata": {},
@@ -1699,13 +4498,27 @@
},
{
"data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "",
- "version_major": 2,
- "version_minor": 0
- },
+ "text/markdown": [
+ "Notes on this plot:\n",
+ "\n",
+ "1. We show the _true_ ATE as a red vertical line\n",
+ "2. We show both this process done with the original ADMG and the reduced ADMG. This shows that the reduction\n",
+ " on the ADMG does not affect estimation. However, reduction is still valuable for simplifying visual exploration.\n",
+ "\n",
+ "#### Interpreting $Y_0$/Ananke Estimation of ACE\n",
+ "\n",
+ "The p-value for testing if ananke/y0 estimation of the ACE is non-zero is 7.13e-65, which is below the the significance threshold of 0.01. Therefore, we reject the null hypothesis of the 1 Sample T-test and conclude that the distribution is significantly different from zero. This means that the treatment $X$ has *a negative effect* on the outcome $Y$. \n",
+ "\n",
+ "The p-value for testing if ananke/y0 estimation of the ACE is different from the reference ATE is 2.30e-20, which is below the the significance threshold of 0.01. Therefore, we reject the null hypothesis of the 1-Sample T-test and conclude that the distribution is significantly different from the ground truth calculated by synthetic generation of interventional data (ATE=4.90e-01). \n",
+ "\n",
+ "#### Interpreting Linear Regression Estimation of ACE\n",
+ "\n",
+ "The p-value for testing if Eliater linear regression estimation of the ACE is non-zero is 1.57e-69, which is below the the significance threshold of 0.01. Therefore, we reject the null hypothesis of the 1 Sample T-test and conclude that the distribution is significantly different from zero. This means that the treatment $X$ has *a negative effect* on the outcome $Y$. \n",
+ "\n",
+ "The p-value for testing if Eliater linear regression estimation of the ACE is different from the reference ATE is 5.64e-18, which is below the the significance threshold of 0.01. Therefore, we reject the null hypothesis of the 1-Sample T-test and conclude that the distribution is significantly different from the ground truth calculated by synthetic generation of interventional data (ATE=4.90e-01). \n"
+ ],
"text/plain": [
- "No Workflow ATEs: 0%| | 0/500 [00:00"
]
},
"metadata": {},
@@ -1713,112 +4526,1525 @@
},
{
"data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "",
- "version_major": 2,
- "version_minor": 0
- },
+ "text/markdown": [
+ "### Estimating the Expected Value\n",
+ "\n",
+ "We now estimate the query in the form of the expected value:\n",
+ "\n",
+ "$\\mathbb{E}[Y \\mid do(X = 0)]$\n"
+ ],
"text/plain": [
- "Workflow ATEs: 0%| | 0/500 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
- }
- ],
- "source": [
- "repeats = 500\n",
- "datasets = [\n",
- " example.generate_data(num_samples=500, seed=seed)\n",
- " for seed in trange(repeats, desc=\"Generating datasets\", leave=False)\n",
- "]\n",
- "background_ates = [get_background_ace(seed=seed) for seed in trange(repeats, leave=False)]\n",
- "unreduced_aces = [\n",
- " estimate_ace(graph=graph, treatments=X, outcomes=Y, data=data)\n",
- " for data in tqdm(datasets, desc=\"No Workflow ATEs\", leave=False)\n",
- "]\n",
- "reduced_aces = [\n",
- " estimate_ace(\n",
- " graph=remove_nuisance_variables(graph, treatments=X, outcomes=Y),\n",
- " treatments=X,\n",
- " outcomes=Y,\n",
- " data=data,\n",
- " )\n",
- " for data in tqdm(datasets, desc=\"Workflow ATEs\", leave=False)\n",
- "]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fd17b15b-4a40-4894-92a4-11e2dfe6ca8e",
- "metadata": {},
- "source": [
- "## Does Reducing Nuissance Variables Matter?\n",
- "\n",
- "In the experiment below, I create several random datasets and for each see if adding the removal of nuissance variables changes the resulting ACE by taking the difference between the ACE calculated on the original graph vs the reduced graph."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 26,
- "id": "a2b7ea36-99ba-4d13-8a42-241e364b41e7",
- "metadata": {},
- "outputs": [
+ },
{
"data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjoAAAHHCAYAAAC2rPKaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABT70lEQVR4nO3dd1QU198G8GcRWEBgkY5Ks4JdsQQVK4o9RtRoLGiwJCK2mGKMNRpiEktMUGN+CrHFRBONMbFiT9AoihWwRMUoRVRApcN9//Awr+uCwrK4MD6fc/Yc987cmS8zO7uPs3dmFUIIASIiIiIZMtB3AURERETlhUGHiIiIZItBh4iIiGSLQYeIiIhki0GHiIiIZItBh4iIiGSLQYeIiIhki0GHiIiIZItBh4iIiGTrlQ46c+fOhUKhUGvLy8vDBx98AGdnZxgYGKB///4AgEePHmHMmDFwdHSEQqHAlClTXn7BlUh4eDgUCgVOnTql71JIB4o7LqjsDh06BIVCgUOHDum7lGK5ublh1KhR+i5DUtR798umUCgwd+5cvdbwPKNGjYKbm5u+y9Cam5sb+vTpo5NlySboFH6wFj5MTExQvXp1+Pn5Yfny5Xj48GGJlrN27Vp8+eWXGDhwIH744QdMnToVAPDZZ58hPDwc7777LtavX48RI0aU559DRfjzzz8r9BuLLsXExEiv49TU1GLny8rKwtKlS9GmTRuoVCqYmJigXr16mDhxIi5fvizNV/jBUNwjMTHxufUUd1wQEVV0hvouQNfmz58Pd3d35ObmIjExEYcOHcKUKVOwZMkS7NixA02aNJHm/eSTT/DRRx+p9T9w4ABq1KiBpUuXarS/9tprmDNnzkv5O0jTn3/+idDQ0Fci7GzYsAGOjo548OABtm7dijFjxmjMk5KSgh49eiAqKgp9+vTBW2+9BXNzc8TFxWHz5s1YvXo1cnJy1PqsXLkS5ubmGsuysrJ6bj3FHRdERBWd7IJOz5490bJlS+n5jBkzcODAAfTp0wf9+vVDTEwMTE1NAQCGhoYwNFTfBMnJyUW+6ScnJ6NBgwY6q7OgoAA5OTkwMTHR2TJJHoQQ2LRpE9566y1cv34dGzduLDLojBo1CmfOnMHWrVvh7++vNu3TTz/FzJkzNfoMHDgQtra2pa6puOPiWXl5eSgoKICxsXGp11HZ8BiunDIyMmBmZqbvMmTh8ePHqFq1qr7LeCHZfHX1PF26dMGsWbNw8+ZNbNiwQWp/+nveGzduQKFQ4ODBg7h48aJ0Sr/w+/Pr16/jjz/+kNpv3LgBAMjOzsacOXNQp04dKJVKODs744MPPkB2drZaDQqFAhMnTsTGjRvRsGFDKJVK7N69GwBw+/ZtvP3223BwcIBSqUTDhg2xdu1atf6Fdfz8889YuHAhatasCRMTE3Tt2hVXr17V+JtPnDiBXr16oVq1aqhatSqaNGmCr7/+Wm2e2NhYDBw4ENbW1jAxMUHLli2xY8cOtXlyc3Mxb9481K1bFyYmJrCxsUH79u2xb9++Em37jIwMjB8/HjY2NrC0tMTIkSPx4MEDjfl27doFHx8fVK1aFRYWFujduzcuXrwoTR81ahRCQ0OlbVn4AIAWLVpgwIABastr3LgxFAoFzp07J7X99NNPUCgUiImJkdpKsu2B0u/n7du3o1GjRtIyC/d1Sfz111+4ceMGhgwZgiFDhuDIkSP477//1OY5ceIE/vjjDwQGBmqEHABQKpX46quvSrzO4jzvuCic9tVXX2HZsmWoXbs2lEolLl26BODJWaDCfWplZYXXX39dbdsD/38MXr58GcOHD4dKpYKdnR1mzZoFIQRu3bqF119/HZaWlnB0dMTixYtLXHN4eLjGtGfHVRSu/+rVqxg1ahSsrKygUqkwevRoZGRkaPQtyzEMAP/99x/69++PqlWrwt7eHlOnTtV4DQHFj4np1KkTOnXqpNaWlZWFuXPnol69ejAxMYGTkxMGDBiAa9euSfMUFBRg2bJlaNiwIUxMTODg4IDx48drHItCCCxYsAA1a9aEmZkZOnfurHYcvshXX32Ftm3bwsbGBqampvDy8sLWrVs15ivNcXLs2DG0atUKJiYmqF27Nr777rsS19OpUyc0atQIUVFR6NChA8zMzPDxxx8DKPkxnZ2djalTp8LOzg4WFhbo16+fxvEIFD8mprjxRBs2bEDr1q1hZmaGatWqoUOHDti7d6/aPC96XyxUuB1NTEzQqFEjbNu2rcTbqKCgAHPnzkX16tWlfX7p0iWN12Dh8JDDhw9jwoQJsLe3R82aNQEAN2/exIQJE1C/fn2YmprCxsYGgwYNkj4nn13GkSNHSvS5ADzZ/61bt4aJiQlq1aqFdevWlfhvKyS7MzrFGTFiBD7++GPs3bsXY8eO1ZhuZ2eH9evXY+HChXj06BFCQkIAAJ6enli/fj2mTp2KmjVr4r333pPmLygoQL9+/XDs2DGMGzcOnp6eOH/+PJYuXYrLly9j+/btaus4cOAAfv75Z0ycOBG2trZwc3NDUlISXnvtNenAt7Ozw65duxAYGIj09HSNQc+ff/45DAwMMH36dKSlpeGLL77AsGHDcOLECWmeffv2oU+fPnBycsLkyZPh6OiImJgY7Ny5E5MnTwYAXLx4Ee3atUONGjXw0UcfoWrVqvj555/Rv39//PLLL3jjjTcAPDlIQ0JCMGbMGLRu3Rrp6ek4deoUTp8+jW7dur1wu0+cOBFWVlaYO3cu4uLisHLlSty8eVMKbgCwfv16BAQEwM/PD4sWLUJGRgZWrlyJ9u3b48yZM3Bzc8P48eNx584d7Nu3D+vXr1dbh4+PD3788Ufp+f3793Hx4kUYGBjg6NGj0teVR48ehZ2dHTw9PQGgxNu+tPv52LFj+PXXXzFhwgRYWFhg+fLl8Pf3R3x8PGxsbF64zTZu3IjatWujVatWaNSoEczMzPDjjz/i/fffl+YpDKSlHSt2//59jTZDQ8Niz9Y877jIzMwEAISFhSErKwvjxo2DUqmEtbU19u/fj549e6JWrVqYO3cuMjMz8c0336Bdu3Y4ffq0xgfCm2++CU9PT3z++ef4448/sGDBAlhbW+O7775Dly5dsGjRImzcuBHTp09Hq1at0KFDh1L93S8yePBguLu7IyQkBKdPn8b//vc/2NvbY9GiRWrzleUYzszMRNeuXREfH49JkyahevXqWL9+PQ4cOKB13fn5+ejTpw8iIiIwZMgQTJ48GQ8fPsS+fftw4cIF1K5dGwAwfvx4hIeHY/To0Zg0aRKuX7+Ob7/9FmfOnMFff/0FIyMjAMDs2bOxYMEC9OrVC7169cLp06fRvXt3ja9Ai/P111+jX79+GDZsGHJycrB582YMGjQIO3fuRO/evdXmLclxcv78eXTv3h12dnaYO3cu8vLyMGfOHDg4OJR4G927dw89e/bEkCFDMHz4cDg4OJTqmB4zZgw2bNiAt956C23btsWBAwc0/pbSmjdvHubOnYu2bdti/vz5MDY2xokTJ3DgwAF0794dQMneFwFg79698Pf3R4MGDRASEoJ79+5h9OjRUgh5kRkzZuCLL75A37594efnh7Nnz8LPzw9ZWVlFzj9hwgTY2dlh9uzZePz4MQDg5MmT+PvvvzFkyBDUrFkTN27cwMqVK9GpUydcunRJ4wxaST4XAODq1asYOHAgAgMDERAQgLVr12LUqFHw8vJCw4YNS77BhUyEhYUJAOLkyZPFzqNSqUTz5s2l53PmzBHPboKOHTuKhg0bavR1dXUVvXv3Vmtbv369MDAwEEePHlVrX7VqlQAg/vrrL6kNgDAwMBAXL15UmzcwMFA4OTmJlJQUtfYhQ4YIlUolMjIyhBBCHDx4UAAQnp6eIjs7W5rv66+/FgDE+fPnhRBC5OXlCXd3d+Hq6ioePHigtsyCggLp3127dhWNGzcWWVlZatPbtm0r6tatK7U1bdpU4+8uicL94eXlJXJycqT2L774QgAQv/32mxBCiIcPHworKysxduxYtf6JiYlCpVKptQcFBWnsLyGE2LJliwAgLl26JIQQYseOHUKpVIp+/fqJN998U5qvSZMm4o033pCel3Tbl3Y/Gxsbi6tXr0ptZ8+eFQDEN99884KtJkROTo6wsbERM2fOlNreeust0bRpU7X53njjDQFAYx8Xp/C1XtSjfv36L+xf1HFx/fp1AUBYWlqK5ORktWnNmjUT9vb24t69e1Lb2bNnhYGBgRg5cqRGXePGjZPa8vLyRM2aNYVCoRCff/651P7gwQNhamoqAgICnltrYV1hYWEa0wCIOXPmaKz/7bffVpvvjTfeEDY2Nhp9y3IML1u2TAAQP//8szTP48ePRZ06dQQAcfDgQand1dW1yL+zY8eOomPHjtLztWvXCgBiyZIlGvMWHu9Hjx4VAMTGjRvVpu/evVutPTk5WRgbG4vevXurvVd8/PHHAsALt7sQQvpbC+Xk5IhGjRqJLl26qLWX9Djp37+/MDExETdv3pTaLl26JKpUqVLke8GzOnbsKACIVatWqbWX9JiOjo4WAMSECRPU5nvrrbc0XksBAQHC1dVVo4ZnP2euXLkiDAwMxBtvvCHy8/PV5i3c7qV5X2zWrJlwcnISqampUtvevXsFgCLreXZ5hoaGon///mrtc+fO1djnhe/p7du3F3l5eWrzP7vfhRAiMjJSABDr1q3TWMaLPheEeHIMABBHjhyR2pKTk4VSqRTvvffec/+uZ70SX10VMjc3L/HVVyWxZcsWeHp6wsPDAykpKdKjS5cuAICDBw+qzd+xY0e1cT5CCPzyyy/o27cvhBBqy/Dz80NaWhpOnz6ttozRo0erjX/w8fEBAPz7778AgDNnzuD69euYMmWKxv/SC5Py/fv3ceDAAQwePBgPHz6U1nnv3j34+fnhypUruH37NoAng1QvXryIK1euaLWNxo0bJ/1vEQDeffddGBoa4s8//wTw5OxTamoqhg4dqvb3V6lSBW3atNHYhkUp3AZHjhwB8OTMTatWrdCtWzccPXoUAJCamooLFy5I85Zm25d2P/v6+kr/kwaAJk2awNLSUtpHz7Nr1y7cu3cPQ4cOldqGDh2Ks2fPqp2yTk9PBwBYWFi8cJlP++WXX7Bv3z61R1hYWKmW8Sx/f3/Y2dlJzxMSEhAdHY1Ro0bB2tpaam/SpAm6desm7funPT0GqUqVKmjZsiWEEAgMDJTaraysUL9+/RJtx9J655131J77+Pjg3r170nYuVJZj+M8//4STkxMGDhwo9TczM8O4ceO0rvuXX36Bra0tgoODNaYVHu9btmyBSqVCt27d1Orz8vKCubm59Prdv38/cnJyEBwcrPa/6tLcSqNw/CMAPHjwAGlpafDx8dF4HwNefJzk5+djz5496N+/P1xcXKT5PD094efnV+KalEolRo8erdZW0mO68LU6adIktf5lub3I9u3bUVBQgNmzZ8PAQP0juHC7l/R9sfBYCwgIgEqlkpbTrVu3Eo0pjYiIQF5eHiZMmKDWXtTrqdDYsWNRpUoVtban93tubi7u3buHOnXqwMrKqsh9/6LPhUINGjSQ3rOBJ2eYtXkPeGW+ugKe3AvH3t5eZ8u7cuUKYmJi1N7kn5acnKz23N3dXe353bt3kZqaitWrV2P16tUlWsbTBzwAVKtWDQCk7zcLv5dv1KhRsXVfvXoVQgjMmjULs2bNKna9NWrUwPz58/H666+jXr16aNSoEXr06IERI0aoXb32PHXr1lV7bm5uDicnJ+m728IAVfgG8yxLS8sXrsPBwQF169bF0aNHMX78eBw9ehSdO3dGhw4dEBwcjH///RcxMTEoKCiQDprSbPvS7udn9xHwZD8V9x300zZs2AB3d3colUpp7FXt2rVhZmaGjRs34rPPPgPw/9vl4cOHJRokXKhDhw5aDUZ+nmdf1zdv3gQA1K9fX2NeT09P7NmzR2MQ47PbrPBS+WdrValUuHfvnq5KL3b9Tx9XT78Gy3IM37x5E3Xq1NEYr1HUdiqpa9euoX79+hoXVTztypUrSEtLK/a97+n6AM1j1s7OTtoeL7Jz504sWLAA0dHRamNdihqj8qLj5O7du8jMzNSoB3iyzYoKzEWpUaOGxuD4kh7TN2/ehIGBgVogK1y/tq5duwYDA4PnBpGSvi8Wt88KaywqZDytsH+dOnXU2q2trYvd588eA8CTr2VDQkIQFhaG27dvQwghTUtLS9OY/0WfC4XK8l76tFcm6Pz3339IS0vT2KFlUVBQgMaNG2PJkiVFTnd2dlZ7/nTqLewPAMOHD0dAQECRy3g2UDybpAs9/cJ6kcL1Tp8+vdj/GRVupw4dOuDatWv47bffsHfvXvzvf//D0qVLsWrVqiKvBCqtwlrWr18PR0dHjenPewN/Wvv27REREYHMzExERUVh9uzZaNSoEaysrHD06FHExMTA3NwczZs3V1tvSbZ9afeztvsoPT0dv//+O7Kysop849q0aRMWLlwIhUIBDw8PAE/GMDz9Px59ePZ1rY2itpm227G4G8nl5+eXav1FrUsXx3BJPO9vKK7W4hQUFMDe3h4bN24scnpxH/aldfToUfTr1w8dOnTAihUr4OTkBCMjI4SFhWHTpk0a8+vivawkinp9lvaYLgltXnfF0dX7YnkoansGBwcjLCwMU6ZMgbe3N1QqFRQKBYYMGSL9LdrQ1WvklQk6hQNYS3PK80Vq166Ns2fPomvXrlrdpbNwFH9+fj58fX11VhMAXLhwodhl1qpVCwBgZGRUovVaW1tj9OjRGD16NB49eoQOHTpg7ty5JQo6V65cQefOnaXnjx49QkJCAnr16qVWr729/Qtred429vHxQVhYGDZv3oz8/Hy0bdsWBgYGaN++vRR02rZtKx04pdn2Zd3PJfXrr78iKysLK1eu1DiTERcXh08++QR//fUX2rdvj759+yIkJAQbNmzQe9B5lqurK4AnNT8rNjYWtra25XpJauH/RJ+90WLh/151qTSvI1dXV1y4cAFCCLXXUVHbqVq1akXeKPLmzZvS8Qs8eW2eOHECubm5al8FPK127drYv38/2rVr99xQWrjfrly5oraOu3fvluh/0L/88gtMTEywZ88eKJVKqV3br0bt7Oxgampa5NfmRW2z0ijpMe3q6oqCggLpzNnz1v+8ffbsugsKCnDp0iU0a9as2PqAF78vPr3PnlWSbVTY/+rVq2pnau7du1eqsyZbt25FQECA2lWRWVlZxd7s9EWfC7r2SozROXDgAD799FO4u7tj2LBhOlvu4MGDcfv2bXz//fca0zIzM6UR6cWpUqUK/P398csvv+DChQsa0+/evVvqmlq0aAF3d3csW7ZM40VWmILt7e3RqVMnfPfdd0hISHjuep/9msDc3Bx16tQp8pLYoqxevRq5ubnS85UrVyIvLw89e/YE8CR4Wlpa4rPPPlObr6haCj8cizp4Cj/sFy1ahCZNmkjfV/v4+CAiIgKnTp1SCwSl2fZl3c8ltWHDBtSqVQvvvPMOBg4cqPaYPn06zM3Npf+Ve3t7o0ePHvjf//6ncdUXAOTk5GD69Ok6qau0nJyc0KxZM/zwww9q++rChQvYu3dvub2ZFbK0tIStra00ZqvQihUrdL6u0ryOevXqhTt37qhdbp2RkVHkV161a9fG8ePH1a522rlzJ27duqU2n7+/P1JSUvDtt99qLKPweB88eDDy8/Px6aefasyTl5cn7SNfX18YGRnhm2++Ufsf87Jly4r569VVqVIFCoVC7QzGjRs3inx9lnR5fn5+2L59O+Lj46X2mJgY7NmzR6tlFirpMV34PrV8+XK1eYraJrVr10ZaWpraLS0SEhI0LvXu378/DAwMMH/+fI2zHYXbvaTvi08fa09/RbRv3z7pNg/P07VrVxgaGmLlypVq7UW9np6nSpUqGmdZvvnmm2LPZr3oc0HXZHdGZ9euXYiNjUVeXh6SkpJw4MAB7Nu3D66urtixY4dOb+41YsQI/Pzzz3jnnXdw8OBBtGvXDvn5+YiNjcXPP/+MPXv2qN28sCiff/45Dh48iDZt2mDs2LFo0KAB7t+/j9OnT2P//v1FXg78PAYGBli5ciX69u2LZs2aYfTo0XByckJsbCwuXrwovUGEhoaiffv2aNy4McaOHYtatWohKSkJkZGR+O+//3D27FkATwaDderUCV5eXrC2tsapU6ewdetWTJw4sUT15OTkoGvXrhg8eDDi4uKwYsUKtG/fHv369QPw5ENp5cqVGDFiBFq0aIEhQ4bAzs4O8fHx+OOPP9CuXTvpoPPy8gLwZGCgn58fqlSpgiFDhgB48lWbo6Mj4uLi1AbSdejQAR9++CEAaJz5KOm218V+fpE7d+7g4MGDGoMeCymVSvj5+WHLli1Yvnw5jIyMsG7dOnTv3h0DBgxA37590bVrV1StWhVXrlzB5s2bkZCQoHEvna1btxZ5Z+Ru3bqV6pLdF/nyyy/Rs2dPeHt7IzAwULq8XKVSvZQ7W48ZMwaff/45xowZg5YtW+LIkSNqP4mhSyV9HY0dOxbffvstRo4ciaioKDg5OWH9+vVF3rxuzJgx2Lp1K3r06IHBgwfj2rVr2LBhg8ZYkZEjR2LdunWYNm0a/vnnH/j4+ODx48fYv38/JkyYgNdffx0dO3bE+PHjERISgujoaHTv3h1GRka4cuUKtmzZgq+//hoDBw6EnZ0dpk+fjpCQEPTp0we9evXCmTNnsGvXrhKN6+rduzeWLFmCHj164K233kJycjJCQ0NRp04dtQ//0pg3bx52794NHx8fTJgwAXl5efjmm2/QsGFDrZcJlPyYbtasGYYOHYoVK1YgLS0Nbdu2RURERJH3LhsyZAg+/PBDvPHGG5g0aZJ0OXi9evXUxsrUqVMHM2fOxKeffgofHx8MGDAASqUSJ0+eRPXq1RESElKq98WQkBD07t0b7du3x9tvv4379+9L2+jRo0fP3Q4ODg6YPHkyFi9ejH79+qFHjx44e/astM9Lega7T58+WL9+PVQqFRo0aIDIyEjs37+/2NtpvOhzQedKdY1WBVZ42Vrhw9jYWDg6Oopu3bqJr7/+WqSnp2v0Kevl5UI8uXxy0aJFomHDhkKpVIpq1aoJLy8vMW/ePJGWlibNB0AEBQUVWXtSUpIICgoSzs7OwsjISDg6OoquXbuK1atXS/MUXl6+ZcsWtb7FXUp77Ngx0a1bN2FhYSGqVq0qmjRponF587Vr18TIkSOFo6OjMDIyEjVq1BB9+vQRW7duleZZsGCBaN26tbCyshKmpqbCw8NDLFy4UO3SwKIU7o/Dhw+LcePGiWrVqglzc3MxbNgwtUuOn/77/Pz8hEqlEiYmJqJ27dpi1KhR4tSpU9I8eXl5Ijg4WNjZ2QmFQqGx7wYNGiQAiJ9++klqy8nJEWZmZsLY2FhkZmZqrLck275wOWXZz8VdLlxo8eLFAoCIiIgodp7w8HCNSzAzMjLEV199JVq1aiXMzc2FsbGxqFu3rggODla7dPd5l5fjmUubi/K8y8u//PLLIvvs379ftGvXTpiamgpLS0vRt29f6RYAz9Z19+5dtfaAgABRtWrVEtVRlIyMDBEYGChUKpWwsLAQgwcPFsnJycVeXv7s+gtfv9evX5faynoMCyHEzZs3Rb9+/YSZmZmwtbUVkydPli7zfnYfLF68WNSoUUMolUrRrl07cerUKY3Lywv/1pkzZwp3d3dp3QMHDhTXrl1Tm2/16tXCy8tLmJqaCgsLC9G4cWPxwQcfiDt37kjz5Ofni3nz5gknJydhamoqOnXqJC5cuPDC12+hNWvWiLp16wqlUik8PDxEWFhYke+zpTlODh8+LLy8vISxsbGoVauWWLVqVZHLLMrzXi8lPaYzMzPFpEmThI2Njahataro27evuHXrlsZrSYgnl3U3atRIGBsbi/r164sNGzYUW+vatWtF8+bNpXV37NhR7Nu3T22ekrwvCiHEL7/8Ijw9PYVSqRQNGjQQv/76a7GXuz8rLy9PzJo1Szg6OgpTU1PRpUsXERMTI2xsbMQ777wjzfe8W7g8ePBAjB49Wtja2gpzc3Ph5+cnYmNjNfZnaT4XivvMLeoYeBGFEDoe+UVERESVVmpqKqpVq4YFCxYU+VMy2iq8aeXJkyfLfBa8NF6JMTpERESkqfAO508rHIP07M+NVFayG6NDREREJfPTTz8hPDwcvXr1grm5OY4dO4Yff/wR3bt3R7t27fRdnk4w6BAREb2imjRpAkNDQ3zxxRdIT0+XBigvWLBA36XpDMfoEBERkWxxjA4RERHJFoMOERERyZbsx+gUFBTgzp07sLCwKNfb9xMREZHuCCHw8OFDVK9eXeOX3ktD9kHnzp07Wv1AGxEREenfrVu3ULNmTa37yz7oWFhYAHiyoQp/2p6IiIgqtvT0dDg7O0uf49qSfdAp/LrK0tKSQYeIiKiSKeuwE70ORnZzc4NCodB4BAUFAXjyM+9BQUGwsbGBubk5/P39kZSUpM+SiYiIqBLRa9A5efIkEhISpMe+ffsAAIMGDQIATJ06Fb///ju2bNmCw4cP486dOxgwYIA+SyYiIqJKpELdMHDKlCnYuXMnrly5gvT0dNjZ2WHTpk0YOHAgACA2Nhaenp6IjIzEa6+9VqJlpqenQ6VSIS0tjV9dERERVRK6+vyuMPfRycnJwYYNG/D2229DoVAgKioKubm58PX1lebx8PCAi4sLIiMji11OdnY20tPT1R5ERET0aqowQWf79u1ITU3FqFGjAACJiYkwNjaGlZWV2nwODg5ITEwsdjkhISFQqVTSg5eWExERvboqTNBZs2YNevbsierVq5dpOTNmzEBaWpr0uHXrlo4qJCIiosqmQlxefvPmTezfvx+//vqr1Obo6IicnBykpqaqndVJSkqCo6NjsctSKpVQKpXlWS4RERFVEhXijE5YWBjs7e3Ru3dvqc3LywtGRkaIiIiQ2uLi4hAfHw9vb299lElERESVjN7P6BQUFCAsLAwBAQEwNPz/clQqFQIDAzFt2jRYW1vD0tISwcHB8Pb2LvEVV0RERPRq03vQ2b9/P+Lj4/H2229rTFu6dCkMDAzg7++P7Oxs+Pn5YcWKFXqokoiIiCqjCnUfnfLA++gQERFVPrK7jw4RERGRrjHoEBERkWwx6BAREZFs6X0wMhFRWcTHxyMlJUWrvra2tnBxcdFxRURUkTDoEFGlFR8fDw8PT2RmZmjV39TUDLGxMQw7RDLGoENElVZKSgoyMzPQ5u05sHRyK1Xf9IQbOLF2HlJSUhh0iGSMQYeIKj1LJzdYu9TXdxlEVAFxMDIRERHJFs/oENErLSYmRqt+HMhMVDkw6BDRKykz7R4ABYYPH65Vfw5kJqocGHSI6JWUm/EQgECztz6EnbtHqfpyIDNR5cGgQ0SvNHN7Fw5kJpIxDkYmIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItnSe9C5ffs2hg8fDhsbG5iamqJx48Y4deqUNF0IgdmzZ8PJyQmmpqbw9fXFlStX9FgxERERVRZ6DToPHjxAu3btYGRkhF27duHSpUtYvHgxqlWrJs3zxRdfYPny5Vi1ahVOnDiBqlWrws/PD1lZWXqsnIiIiCoDQ32ufNGiRXB2dkZYWJjU5u7uLv1bCIFly5bhk08+weuvvw4AWLduHRwcHLB9+3YMGTLkpddMRERElYdez+js2LEDLVu2xKBBg2Bvb4/mzZvj+++/l6Zfv34diYmJ8PX1ldpUKhXatGmDyMjIIpeZnZ2N9PR0tQcRERG9mvQadP7991+sXLkSdevWxZ49e/Duu+9i0qRJ+OGHHwAAiYmJAAAHBwe1fg4ODtK0Z4WEhEClUkkPZ2fn8v0jiIiIqMLSa9ApKChAixYt8Nlnn6F58+YYN24cxo4di1WrVmm9zBkzZiAtLU163Lp1S4cVExERUWWi16Dj5OSEBg0aqLV5enoiPj4eAODo6AgASEpKUpsnKSlJmvYspVIJS0tLtQcRERG9mvQadNq1a4e4uDi1tsuXL8PV1RXAk4HJjo6OiIiIkKanp6fjxIkT8Pb2fqm1EhERUeWj16uupk6dirZt2+Kzzz7D4MGD8c8//2D16tVYvXo1AEChUGDKlClYsGAB6tatC3d3d8yaNQvVq1dH//799Vk6ERERVQJ6DTqtWrXCtm3bMGPGDMyfPx/u7u5YtmwZhg0bJs3zwQcf4PHjxxg3bhxSU1PRvn177N69GyYmJnqsnIiIiCoDvQYdAOjTpw/69OlT7HSFQoH58+dj/vz5L7EqIiIikgO9/wQEERERUXlh0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItky1HcBRESVVUxMjFb9bG1t4eLiouNqiKgoDDpERKWUmXYPgALDhw/Xqr+pqRliY2MYdoheAgYdIqJSys14CECg2Vsfws7do1R90xNu4MTaeUhJSWHQIXoJGHSIiLRkbu8Ca5f6+i6DiJ6Dg5GJiIhIthh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi29Bp05s6dC4VCofbw8Pj/26lnZWUhKCgINjY2MDc3h7+/P5KSkvRYMREREVUmej+j07BhQyQkJEiPY8eOSdOmTp2K33//HVu2bMHhw4dx584dDBgwQI/VEhERUWWi99+6MjQ0hKOjo0Z7Wloa1qxZg02bNqFLly4AgLCwMHh6euL48eN47bXXXnapREREVMnoPehcuXIF1atXh4mJCby9vRESEgIXFxdERUUhNzcXvr6+0rweHh5wcXFBZGRksUEnOzsb2dnZ0vP09PRy/xuICIiPj0dKSopWfW1tbflL3kRULvQadNq0aYPw8HDUr18fCQkJmDdvHnx8fHDhwgUkJibC2NgYVlZWan0cHByQmJhY7DJDQkIwb968cq6ciJ4WHx8PDw9PZGZmaNXf1NQMsbExDDtEpHN6DTo9e/aU/t2kSRO0adMGrq6u+Pnnn2FqaqrVMmfMmIFp06ZJz9PT0+Hs7FzmWomoeCkpKcjMzECbt+fA0smtVH3TE27gxNp5SElJYdAhIp3T+1dXT7OyskK9evVw9epVdOvWDTk5OUhNTVU7q5OUlFTkmJ5CSqUSSqXyJVRLRM+ydHKDtUt9fZdBRCTR+1VXT3v06BGuXbsGJycneHl5wcjICBEREdL0uLg4xMfHw9vbW49VEhERUWWh1zM606dPR9++feHq6oo7d+5gzpw5qFKlCoYOHQqVSoXAwEBMmzYN1tbWsLS0RHBwMLy9vXnFFREREZWIXoPOf//9h6FDh+LevXuws7ND+/btcfz4cdjZ2QEAli5dCgMDA/j7+yM7Oxt+fn5YsWKFPksmIiKiSkSvQWfz5s3PnW5iYoLQ0FCEhoa+pIqIiIhITirUGB0iIiIiXWLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2aowQefzzz+HQqHAlClTpLasrCwEBQXBxsYG5ubm8Pf3R1JSkv6KJCIiokqlQgSdkydP4rvvvkOTJk3U2qdOnYrff/8dW7ZsweHDh3Hnzh0MGDBAT1USERFRZaP3oPPo0SMMGzYM33//PapVqya1p6WlYc2aNViyZAm6dOkCLy8vhIWF4e+//8bx48f1WDERERFVFnoPOkFBQejduzd8fX3V2qOiopCbm6vW7uHhARcXF0RGRr7sMomIiKgSMtTnyjdv3ozTp0/j5MmTGtMSExNhbGwMKysrtXYHBwckJiYWu8zs7GxkZ2dLz9PT03VWLxEREVUuejujc+vWLUyePBkbN26EiYmJzpYbEhIClUolPZydnXW2bCIiIqpc9BZ0oqKikJycjBYtWsDQ0BCGhoY4fPgwli9fDkNDQzg4OCAnJwepqalq/ZKSkuDo6FjscmfMmIG0tDTpcevWrXL+S4iIiKii0ttXV127dsX58+fV2kaPHg0PDw98+OGHcHZ2hpGRESIiIuDv7w8AiIuLQ3x8PLy9vYtdrlKphFKpLNfaiYiIqHLQW9CxsLBAo0aN1NqqVq0KGxsbqT0wMBDTpk2DtbU1LC0tERwcDG9vb7z22mv6KJmIiIgqGa2CTq1atXDy5EnY2NiotaempqJFixb4999/dVLc0qVLYWBgAH9/f2RnZ8PPzw8rVqzQybKJqGKJiYl5KX2I6NWiVdC5ceMG8vPzNdqzs7Nx+/ZtrYs5dOiQ2nMTExOEhoYiNDRU62USUcWWmXYPgALDhw/Xehm52Tm6K4iIZKVUQWfHjh3Sv/fs2QOVSiU9z8/PR0REBNzc3HRWHBHJX27GQwACzd76EHbuHqXqm3A+Ehd2rEZeXl75FEdElV6pgk7//v0BAAqFAgEBAWrTjIyM4ObmhsWLF+usOCJ6dZjbu8DapX6p+qQn3CifYohINkoVdAoKCgAA7u7uOHnyJGxtbculKCIiIiJd0GqMzvXr13VdBxEREZHOaX15eUREBCIiIpCcnCyd6Sm0du3aMhdGREREVFZaBZ158+Zh/vz5aNmyJZycnKBQKHRdFxEREVGZaRV0Vq1ahfDwcIwYMULX9RARERHpjFa/dZWTk4O2bdvquhYiIiIindIq6IwZMwabNm3SdS1EREREOqXVV1dZWVlYvXo19u/fjyZNmsDIyEht+pIlS3RSHBEREVFZaBV0zp07h2bNmgEALly4oDaNA5OJiIiootAq6Bw8eFDXdRARERHpnFZjdIiIiIgqA63O6HTu3Pm5X1EdOHBA64KIiIiIdEWroFM4PqdQbm4uoqOjceHCBY0f+yQiIiLSF62CztKlS4tsnzt3Lh49elSmgoiIiIh0RadjdIYPH87fuSIiIqIKQ6dBJzIyEiYmJrpcJBEREZHWtPrqasCAAWrPhRBISEjAqVOnMGvWLJ0URkRERFRWWgUdlUql9tzAwAD169fH/Pnz0b17d50URkRERFRWWgWdsLAwXddBREREpHNaBZ1CUVFRiImJAQA0bNgQzZs310lRRERERLqgVdBJTk7GkCFDcOjQIVhZWQEAUlNT0blzZ2zevBl2dna6rJGIiIhIK1pddRUcHIyHDx/i4sWLuH//Pu7fv48LFy4gPT0dkyZN0nWNRERERFrR6ozO7t27sX//fnh6ekptDRo0QGhoKAcjE1Vi8fHxSElJKXW/wq+wiYgqGq2CTkFBAYyMjDTajYyMUFBQUOaiiOjli4+Ph4eHJzIzM7ReRm52jg4rIiIqO62CTpcuXTB58mT8+OOPqF69OgDg9u3bmDp1Krp27arTAono5UhJSUFmZgbavD0Hlk5upeqbcD4SF3asRl5eXvkUR0SkJa2Czrfffot+/frBzc0Nzs7OAIBbt26hUaNG2LBhg04LJKKXy9LJDdYu9UvVJz3hRvkUQ0RURloFHWdnZ5w+fRr79+9HbGwsAMDT0xO+vr46LY6IiIioLEp11dWBAwfQoEEDpKenQ6FQoFu3bggODkZwcDBatWqFhg0b4ujRo+VVKxEREVGplCroLFu2DGPHjoWlpaXGNJVKhfHjx2PJkiU6K46IiIioLEoVdM6ePYsePXoUO7179+6Iiooqc1FEREREulCqoJOUlFTkZeWFDA0Ncffu3TIXRURERKQLpQo6NWrUwIULF4qdfu7cOTg5OZW5KCIiIiJdKNVVV7169cKsWbPQo0cPmJiYqE3LzMzEnDlz0KdPH50WSEQkR9reTdrW1hYuLi46roZIvkoVdD755BP8+uuvqFevHiZOnIj69Z/cayM2NhahoaHIz8/HzJkzy6VQIiI5yEy7B0CB4cOHa9Xf1NQMsbExDDtEJVSqoOPg4IC///4b7777LmbMmAEhBABAoVDAz88PoaGhcHBwKJdCiYjkIDfjIQCBZm99CDt3j1L1TU+4gRNr5yElJYVBh6iESn3DQFdXV/z555948OABrl69CiEE6tati2rVqpVHfUREsmRu71LqO1ATUelpdWdkAKhWrRpatWqly1qIiIiIdErroENEFVN8fDxSUlJK3U/bwbFERBUZgw6RjMTHx8PDwxOZmRlaLyM3O0eHFRER6Zdeg87KlSuxcuVK3LhxAwDQsGFDzJ49Gz179gQAZGVl4b333sPmzZuRnZ0NPz8/rFixggOeiYqRkpKCzMwMtHl7Diyd3ErVN+F8JC7sWI28vLzyKY6ISA/0GnRq1qyJzz//HHXr1oUQAj/88ANef/11nDlzBg0bNsTUqVPxxx9/YMuWLVCpVJg4cSIGDBiAv/76S59lE1V4lk5upR7omp5wo3yKISLSI70Gnb59+6o9X7hwIVauXInjx4+jZs2aWLNmDTZt2oQuXboAAMLCwuDp6Ynjx4/jtdde00fJREREVImU6icgylN+fj42b96Mx48fw9vbG1FRUcjNzYWvr680j4eHB1xcXBAZGVnscrKzs5Genq72ICIioleT3oPO+fPnYW5uDqVSiXfeeQfbtm1DgwYNkJiYCGNjY1hZWanN7+DggMTExGKXFxISApVKJT2cnZ3L+S8gIiKiikrvQad+/fqIjo7GiRMn8O677yIgIACXLl3SenkzZsxAWlqa9Lh165YOqyUiIqLKRO+XlxsbG6NOnToAAC8vL5w8eRJff/013nzzTeTk5CA1NVXtrE5SUhIcHR2LXZ5SqYRSqSzvsomIiKgS0PsZnWcVFBQgOzsbXl5eMDIyQkREhDQtLi4O8fHx8Pb21mOFREREVFno9YzOjBkz0LNnT7i4uODhw4fYtGkTDh06hD179kClUiEwMBDTpk2DtbU1LC0tERwcDG9vb15xRURERCWi16CTnJyMkSNHIiEhASqVCk2aNMGePXvQrVs3AMDSpUthYGAAf39/tRsGEhEREZWEXoPOmjVrnjvdxMQEoaGhCA0NfUkVERERkZxUuDE6RERERLrCoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyZajvAoiIqHRiYmK06pednQ2lUqlVX1tbW7i4uGjVl0ifGHSIiCqJzLR7ABQYPny4dgtQKAAhtOpqamqG2NgYhh2qdBh0iIgqidyMhwAEmr31IezcPUrVN+F8JC7sWK1V3/SEGzixdh5SUlIYdKjSYdAhIqpkzO1dYO1Sv1R90hNuaN2XqDLjYGQiIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki29Bp2QkBC0atUKFhYWsLe3R//+/REXF6c2T1ZWFoKCgmBjYwNzc3P4+/sjKSlJTxUTERFRZaLXoHP48GEEBQXh+PHj2LdvH3Jzc9G9e3c8fvxYmmfq1Kn4/fffsWXLFhw+fBh37tzBgAED9Fg1ERERVRZ6/QmI3bt3qz0PDw+Hvb09oqKi0KFDB6SlpWHNmjXYtGkTunTpAgAICwuDp6cnjh8/jtdee00fZRMREVElUaHG6KSlpQEArK2tAQBRUVHIzc2Fr6+vNI+HhwdcXFwQGRlZ5DKys7ORnp6u9iAiIqJXU4UJOgUFBZgyZQratWuHRo0aAQASExNhbGwMKysrtXkdHByQmJhY5HJCQkKgUqmkh7Ozc3mXTkRERBVUhQk6QUFBuHDhAjZv3lym5cyYMQNpaWnS49atWzqqkIiIiCobvY7RKTRx4kTs3LkTR44cQc2aNaV2R0dH5OTkIDU1Ve2sTlJSEhwdHYtcllKphFKpLO+SiYiIqBLQ6xkdIQQmTpyIbdu24cCBA3B3d1eb7uXlBSMjI0REREhtcXFxiI+Ph7e398sul4iIiCoZvZ7RCQoKwqZNm/Dbb7/BwsJCGnejUqlgamoKlUqFwMBATJs2DdbW1rC0tERwcDC8vb15xRURERG9kF6DzsqVKwEAnTp1UmsPCwvDqFGjAABLly6FgYEB/P39kZ2dDT8/P6xYseIlV0pERESVkV6DjhDihfOYmJggNDQUoaGhL6EiIiIikpMKc9UVERERka4x6BAREZFsMegQERGRbFWI++gQkbr4+HikpKSUul9MTEw5VENEVHkx6BBVMPHx8fDw8ERmZobWy8jNztFhRURElReDDlEFk5KSgszMDLR5ew4sndxK1TfhfCQu7FiNvLy88imOiKiSYdAhqqAsndxg7VK/VH3SE26UTzFERJUUByMTERGRbPGMDhERlYi2g91tbW3h4uKi42qISoZBh4iInisz7R4ABYYPH65Vf1NTM8TGxjDskF4w6BAR0XPlZjwEINDsrQ9h5+5Rqr7pCTdwYu08pKSkMOiQXjDoEBFRiZjbu5R6gDyRvnEwMhEREckWgw4RERHJFoMOERERyRaDDhEREckWgw4RERHJFoMOERERyRaDDhEREckWgw4RERHJFoMOERERyRaDDhEREckWgw4RERHJFoMOERERyRaDDhEREckWgw4RERHJFoMOERERyRaDDhEREckWgw4RERHJFoMOERERyRaDDhEREckWgw4RERHJFoMOERERyRaDDhEREcmWob4LIJKr+Ph4pKSklLpfTExMOVRDRPRqYtAhKgfx8fHw8PBEZmaG1svIzc7RYUVERK8mBh2icpCSkoLMzAy0eXsOLJ3cStU34XwkLuxYjby8vPIpjojoFcKgQ1SOLJ3cYO1Sv1R90hNulE8xRESvIA5GJiIiItli0CEiIiLZYtAhIiIi2dJr0Dly5Aj69u2L6tWrQ6FQYPv27WrThRCYPXs2nJycYGpqCl9fX1y5ckU/xRIREVGlo9eg8/jxYzRt2hShoaFFTv/iiy+wfPlyrFq1CidOnEDVqlXh5+eHrKysl1wpERERVUZ6veqqZ8+e6NmzZ5HThBBYtmwZPvnkE7z++usAgHXr1sHBwQHbt2/HkCFDXmapREREVAlV2DE6169fR2JiInx9faU2lUqFNm3aIDIysth+2dnZSE9PV3sQERHRq6nCBp3ExEQAgIODg1q7g4ODNK0oISEhUKlU0sPZ2blc6yQiIqKKq8IGHW3NmDEDaWlp0uPWrVv6LomIiIj0pMIGHUdHRwBAUlKSWntSUpI0rShKpRKWlpZqDyIiIno1Vdig4+7uDkdHR0REREht6enpOHHiBLy9vfVYGREREVUWer3q6tGjR7h69ar0/Pr164iOjoa1tTVcXFwwZcoULFiwAHXr1oW7uztmzZqF6tWro3///vormoiIiCoNvQadU6dOoXPnztLzadOmAQACAgIQHh6ODz74AI8fP8a4ceOQmpqK9u3bY/fu3TAxMdFXyURERFSJ6DXodOrUCUKIYqcrFArMnz8f8+fPf4lVERERkVxU2DE6RERERGXFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLKl1/voEBHRqyEmJkarfra2tnBxcdFxNfQqYdAhIqJyk5l2D4ACw4cP16q/qakZYmNjGHZIaww6RERUbnIzHgIQaPbWh7Bz9yhV3/SEGzixdh5SUlIYdEhrDDpERFTuzO1dYO1SX99l0CuIg5GJiIhIthh0iIiISLb41RUREdEz4uPjkZKSolVfXilWsTDoEBERPSU+Ph4eHp7IzMzQqj+vFKtYGHSIiIiekpKSgszMDLR5ew4sndxK1ZdXilU8DDpERERFsHRy45ViMsDByERERCRbDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFu8vJyIiCq0mJgYrfrxDsUEMOgQEVEFlZl2D4ACw4cP16o/71BMAIMOERFVULkZDwEINHvrQ9i5e5SqL+9QTIUYdIiIqEIzt3fhHYpJaxyMTERERLLFMzoke/Hx8UhJSdGqb3Z2NpRKZan7aTt4koiIdItBh2QtPj4eHh6eyMzM0G4BCgUghNbrz83O0bovERGVHYMOyVpKSgoyMzPQ5u05sHRyK1XfhPORuLBjtVYDIQv75uXllaofERHpFoMOvRIsndxKPZgxPeEGAO0GQhb2JSIi/eJgZCIiIpItntEhIiLSMW0vSND2AgiAd4IuDoMOERGRjpT1bs5luQCCd4IuGoMOERGRjpTlbs5luQCCd4IuHoMOERGRjpXlIgbeCVq3OBiZiIiIZItBh4iIiGSLX12VgT5+WqBQWUbXl6XuyrheInp1aXP106v4Ey5yfn+uFEEnNDQUX375JRITE9G0aVN88803aN26tV5r0vdPC2g7ur6sdVe29RLRq6nMVz/h1fkJF7m/P1f4oPPTTz9h2rRpWLVqFdq0aYNly5bBz88PcXFxsLe311td+vppAaBso+vLUndlXC8RvZp0cfXTq/ITLnJ/f67wQWfJkiUYO3YsRo8eDQBYtWoV/vjjD6xduxYfffSRnqt7+T8toCva1F2Z10tEryb+hEvJyfX9uUIPRs7JyUFUVBR8fX2lNgMDA/j6+iIyMlKPlREREVFlUKHP6KSkpCA/Px8ODg5q7Q4ODoiNjS2yT3Z2NrKzs6XnaWlpAID09HSd1vbo0SMAwP2bccjLzixV3/SEm09qu30FRoaKUq87PTEeABAVFSXVUVJxcXEAtKz7VVtvGfYT+7Iv+7LvS+2r5/fnR48e6fxztnB5ogzjWQsXUGHdvn1bABB///23Wvv7778vWrduXWSfOXPmCAB88MEHH3zwwYcMHrdu3SpTlqjQZ3RsbW1RpUoVJCUlqbUnJSXB0dGxyD4zZszAtGnTpOcFBQW4f/8+bGxsoFCU/uzJqyw9PR3Ozs64desWLC0t9V1OpcftqXvcprrHbapb3J7aE0Lg4cOHqF69epmWU6GDjrGxMby8vBAREYH+/fsDeBJcIiIiMHHixCL7KJVKjfvTWFlZlXOl8mZpackDVIe4PXWP21T3uE11i9tTOyqVqszLqNBBBwCmTZuGgIAAtGzZEq1bt8ayZcvw+PFj6SosIiIiouJU+KDz5ptv4u7du5g9ezYSExPRrFkz7N69W2OAMhEREdGzKnzQAYCJEycW+1UVlR+lUok5c+aU6acq6P9xe+oet6nucZvqFren/imEKOt1W0REREQVU4W+YSARERFRWTDoEBERkWwx6BAREZFsMegQERGRbDHo0AvduHEDgYGBcHd3h6mpKWrXro05c+YgJydH36VVagsXLkTbtm1hZmbGm1pqITQ0FG5ubjAxMUGbNm3wzz//6LukSu3IkSPo27cvqlevDoVCge3bt+u7pEotJCQErVq1goWFBezt7dG/f3/pN6Xo5WLQoReKjY1FQUEBvvvuO1y8eBFLly7FqlWr8PHHH+u7tEotJycHgwYNwrvvvqvvUiqdn376CdOmTcOcOXNw+vRpNG3aFH5+fkhOTtZ3aZXW48eP0bRpU4SGhuq7FFk4fPgwgoKCcPz4cezbtw+5ubno3r07Hj9+rO/SXjm8vJy08uWXX2LlypX4999/9V1KpRceHo4pU6YgNTVV36VUGm3atEGrVq3w7bffAnjy0zDOzs4IDg7GRx99pOfqKj+FQoFt27ZJP71DZXf37l3Y29vj8OHD6NChg77LeaXwjA5pJS0tDdbW1voug15BOTk5iIqKgq+vr9RmYGAAX19fREZG6rEyouKlpaUBAN839YBBh0rt6tWr+OabbzB+/Hh9l0KvoJSUFOTn52v8DIyDgwMSExP1VBVR8QoKCjBlyhS0a9cOjRo10nc5rxwGnVfYRx99BIVC8dxHbGysWp/bt2+jR48eGDRoEMaOHaunyisubbYpEclbUFAQLly4gM2bN+u7lFdSpfitKyof7733HkaNGvXceWrVqiX9+86dO+jcuTPatm2L1atXl3N1lVNptymVnq2tLapUqYKkpCS19qSkJDg6OuqpKqKiTZw4ETt37sSRI0dQs2ZNfZfzSmLQeYXZ2dnBzs6uRPPevn0bnTt3hpeXF8LCwmBgwJOBRSnNNiXtGBsbw8vLCxEREdJg2YKCAkRERPDHf6nCEEIgODgY27Ztw6FDh+Du7q7vkl5ZDDr0Qrdv30anTp3g6uqKr776Cnfv3pWm8X/Q2ouPj8f9+/cRHx+P/Px8REdHAwDq1KkDc3Nz/RZXwU2bNg0BAQFo2bIlWrdujWXLluHx48cYPXq0vkurtB49eoSrV69Kz69fv47o6GhYW1vDxcVFj5VVTkFBQdi0aRN+++03WFhYSOPHVCoVTE1N9Vzdq4WXl9MLhYeHF/sBwpeP9kaNGoUffvhBo/3gwYPo1KnTyy+okvn222/x5ZdfIjExEc2aNcPy5cvRpk0bfZdVaR06dAidO3fWaA8ICEB4ePjLL6iSUygURbaHhYW98Ott0i0GHSIiIpItDrQgIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQIapk3NzcsGzZMr3WcOPGDSgUCuluzqRJoVBg+/bteq3h0KFDUCgUSE1N1WsdVDEcOXIEffv2RfXq1V/K67O063vnnXegUCh0/v7GoENUjE6dOmHKlCka7eHh4bCysnrp9VRWP/74I6pUqYKgoKAip6enp2PmzJnw8PCAiYkJHB0d4evri19//VW683anTp2K/CX4d955p0y1hYeHS8syMDCAk5MT3nzzTcTHx5dpuZVRSEgIqlSpgi+//LLI6YmJiQgODkatWrWgVCrh7OyMvn37IiIiQprHzc2tyP30+eefv6w/g57j8ePHaNq0KUJDQyvc+rZt24bjx4+jevXqOq+Dv3VFpAc5OTkwNjbWdxkvxZo1a/DBBx/gu+++w+LFi2FiYiJNS01NRfv27ZGWloYFCxagVatWMDQ0xOHDh/HBBx+gS5cuUqgcO3Ys5s+fr7ZsMzOzMtdnaWmJuLg4CCFw/fp1TJgwAYMGDcKJEyfKvOzKZO3atfjggw+wdu1avP/++2rTbty4gXbt2sHKygpffvklGjdujNzcXOzZswdBQUGIjY2V5p0/fz7Gjh2r1t/CwuKl/A30fD179kTPnj2LnZ6dnY2ZM2fixx9/RGpqKho1aoRFixZp/ZM0L1pfodu3byM4OBh79uxB7969tVrX8/CMDlEZjRo1Cv3798dXX30FJycn2NjYICgoCLm5udI8bm5u+PTTTzFy5EhYWlpi3LhxAIBjx47Bx8cHpqamcHZ2xqRJk/D48WOpX3JyMvr27QtTU1O4u7tj48aNausu6iuk1NRUKBQKHDp0SGq7ePEi+vTpA0tLS1hYWMDHxwfXrl2Tpv/vf/+Dp6cnTExM4OHhgRUrVqit559//kHz5s1hYmKCli1b4syZMyXaNtevX8fff/+Njz76CPXq1cOvv/6qNv3jjz/GjRs3cOLECQQEBKBBgwaoV68exo4di+joaLUfNzUzM4Ojo6Paw9LSskR1PI9CoYCjoyOcnJzQtm1bBAYG4p9//kF6ero0z2+//YYWLVrAxMQEtWrVwrx585CXlydNv3LlCjp06AATExM0aNAA+/btU1tHUV8hRUdHQ6FQ4MaNG1LbX3/9hU6dOsHMzAzVqlWDn58fHjx4AODJL7SHhITA3d0dpqamaNq0KbZu3aq2nj///BP16tWDqakpOnfurLbs5zl8+DAyMzMxf/58pKen4++//1abPmHCBCgUCvzzzz/w9/dHvXr10LBhQ0ybNg3Hjx9Xm9fCwkJjP1WtWrVEdZB+TZw4EZGRkdi8eTPOnTuHQYMGoUePHrhy5Uq5rbOgoAAjRozA+++/j4YNG5bLOhh0iHTg4MGDuHbtGg4ePIgffvgB4eHhGj+E+NVXX6Fp06Y4c+YMZs2ahWvXrqFHjx7w9/fHuXPn8NNPP+HYsWOYOHGi1GfUqFG4desWDh48iK1bt2LFihVITk4uVW23b99Ghw4doFQqceDAAURFReHtt9+WPqg3btyI2bNnY+HChYiJicFnn32GWbNmST84+ujRI/Tp0wcNGjRAVFQU5s6di+nTp5do3WFhYejduzdUKhWGDx+ONWvWSNMKCgqwefNmDBs2rMjT1ebm5jA0fLknnZOTk7Ft2zZUqVIFVapUAQAcPXoUI0eOxOTJk3Hp0iV89913CA8Px8KFC6W/Y8CAATA2NsaJEyewatUqfPjhh6Ved3R0NLp27YoGDRogMjISx44dQ9++fZGfnw/gyVdL69atw6pVq3Dx4kVMnToVw4cPx+HDhwEAt27dwoABA9C3b19ER0djzJgx+Oijj0q07jVr1mDo0KEwMjLC0KFD1fbT/fv3sXv3bgQFBRUZWPg1rjzEx8cjLCwMW7ZsgY+PD2rXro3p06ejffv2CAsLK7f1Llq0CIaGhpg0aVK5rQOCiIrUsWNHMXnyZI32sLAwoVKppOcBAQHC1dVV5OXlSW2DBg0Sb775pvTc1dVV9O/fX205gYGBYty4cWptR48eFQYGBiIzM1PExcUJAOKff/6RpsfExAgAYunSpUIIIa5fvy4AiDNnzkjzPHjwQAAQBw8eFEIIMWPGDOHu7i5ycnKK/Dtr164tNm3apNb26aefCm9vbyGEEN99952wsbERmZmZ0vSVK1dqrPdZ+fn5wtnZWWzfvl0IIcTdu3eFsbGx+Pfff4UQQiQlJQkAYsmSJcUuo1DHjh2FkZGRqFq1qtpjw4YNL+z7PGFhYQKAqFq1qjAzMxMABAAxadIkaZ6uXbuKzz77TK3f+vXrhZOTkxBCiD179ghDQ0Nx+/ZtafquXbsEALFt2zYhhBAHDx4UAMSDBw+kec6cOSMAiOvXrwshhBg6dKho165dkXVmZWUJMzMz8ffff6u1BwYGiqFDhwohnuznBg0aqE3/8MMPNdb7rLS0NGFqaiqio6OluszNzcXDhw+FEEKcOHFCABC//vprscso5OrqKoyNjTX205EjR17Yl16up1+fQgixc+dO6Vh4+mFoaCgGDx4shPj/95/nPT788MMSrU8IIU6dOiUcHBzUjh1XV1fp/U1XOEaHSAcaNmwonQEAACcnJ5w/f15tnpYtW6o9P3v2LM6dO6f2dZQQAgUFBbh+/TouX74MQ0NDeHl5SdM9PDxK/T/o6Oho+Pj4wMjISGPa48ePce3aNQQGBqqNq8jLy4NKpQIAxMTEoEmTJmpja7y9vV+43n379uHx48fo1asXAMDW1hbdunXD2rVr8emnn0oDjUtq2LBhmDlzplqbg4NDkfNu3LgR48ePl57v2rULPj4+Rc5rYWGB06dPIzc3F7t27cLGjRulszXAk/30119/qbXl5+cjKysLGRkZiImJgbOzs9pZqZJsn2dFR0dj0KBBRU67evUqMjIy0K1bN7X2nJwcNG/eHMCT/dSmTRu16SWp48cff0Tt2rXRtGlTAECzZs3g6uqKn376CYGBgaXeT++//z5GjRql1lajRo1SLYNevkePHqFKlSqIiopSey8DIH2FXKtWLcTExDx3OTY2NiVe59GjR5GcnAwXFxepLT8/H++99x6WLVtW4q9eX4RBh6gYlpaWSEtL02hPTU2VQkChZ0OEQqFAQUGBWtuzp/0fPXqE8ePHF3nK1sXFBZcvX35hjQYGT759fvrD6OmxQQBgampabP9Hjx4BAL7//nuND8ln3+xKa82aNbh//77a+gsKCnDu3DnMmzcPdnZ2sLKyUhvI+jwqlQp16tQp0bz9+vVT+3ue90FrYGAgLdfT0xPXrl3Du+++i/Xr1wN4so3mzZuHAQMGaPR9Ovw9j6720x9//KHxtyiVyhLVUJw1a9bg4sWLal8TFhQUYO3atQgMDETdunWhUChKvJ9sbW1LvJ+o4mjevDny8/ORnJxc7H8KjI2N4eHhobN1jhgxAr6+vmptfn5+GDFiBEaPHq2z9TDoEBWjfv362Lt3r0b76dOnUa9evTIvv0WLFrh06VKxHwoeHh7Iy8tDVFQUWrVqBQCIi4tTG9BqZ2cHAEhISJD+Z//svW2aNGmCH374Abm5uRqBzMHBAdWrV8e///6LYcOGFVmHp6cn1q9fj6ysLOmD/dkBqM+6d+8efvvtN2zevFltgGF+fj7at2+PvXv3okePHhgyZAjWr1+POXPmaIzTefToEUxMTLQap2NhYaH1lT4fffQRateujalTp6JFixZo0aIF4uLiit1Pnp6euHXrFhISEuDk5ARAc/s8vZ+qVasGoOj9FBERgXnz5mmso0GDBlAqlYiPj0fHjh2LrWPHjh1qbS/aT+fPn8epU6dw6NAhWFtbS+33799Hp06dEBsbCw8PD/j5+SE0NBSTJk3SCOypqakcp1NJPHr0CFevXpWeX79+HdHR0bC2tka9evUwbNgwjBw5EosXL0bz5s1x9+5dREREoEmTJlpdDfW89bm4uMDGxkbjDJCRkREcHR1Rv3597f/QZ+n0izAiGbl27ZowMTERwcHB4uzZsyI2NlYsXrxYGBoail27dknzBQQEiNdff12t7+TJk0XHjh2l50V973z27FlhamoqgoKCxJkzZ8Tly5fF9u3bRVBQkDRPjx49RPPmzcXx48fFqVOnRPv27YWpqanasl577TXh4+MjLl26JA4dOiRat26tNkYnJSVF2NjYiAEDBoiTJ0+Ky5cvi3Xr1onY2FghhBDff/+9MDU1FV9//bWIi4sT586dE2vXrhWLFy8WQgjx8OFDYWtrK4YPHy4uXrwo/vjjD1GnTp3njtFZunSpcHJyEgUFBRrTBg8eLAYOHCiEEOLevXvCw8ND1KxZU/zwww/i4sWL4vLly2LNmjWiTp060tiSjh07irFjx4qEhAS1x/3794vbfSXy7Hirp2vs3bu3EEKI3bt3C0NDQzF37lxx4cIFcenSJfHjjz+KmTNnCiGejEVq0KCB6Natm4iOjhZHjhwRXl5eamMScnJyhLOzsxg0aJC4fPmy2Llzp6hfv77aGJ24uDhhbGws3n33XXH27FkRExMjVqxYIe7evSuEEGLmzJnCxsZGhIeHi6tXr4qoqCixfPlyER4eLoQQ4ubNm8LY2FhMnz5dxMbGio0bNwpHR8fnjtGZPHmyaNOmTZHTWrduLaZPny6EeHIsODo6igYNGoitW7eKy5cvi0uXLomvv/5aeHh4SH1cXV3F/PnzNfZTWlpaqfYLlY/CsWLPPgICAoQQT16ns2fPFm5ubsLIyEg4OTmJN954Q5w7d65c1leU8hijw6BD9Bz//POP6Natm7CzsxMqlUq0adNGY0CdtkHn6eWbm5uLqlWriiZNmoiFCxdK0xMSEkTv3r2FUqkULi4uYt26dRrLunTpkvD29hampqaiWbNmYu/evWpBR4gnoap79+7CzMxMWFhYCB8fH3Ht2jVp+saNG0WzZs2EsbGxqFatmujQoYPa4NPIyEjRtGlTYWxsLJo1ayZ++eWX5wadxo0biwkTJhQ57aeffhLGxsbSB3hqaqr46KOPRN26dYWxsbFwcHAQvr6+Ytu2bVJQ6tixY5FvmH5+fkWuo6SKCzqRkZECgDhx4oQQ4knYadu2rTA1NRWWlpaidevWYvXq1dL8cXFxon379sLY2FjUq1dP7N69W2Pw5bFjx0Tjxo2FiYmJ8PHxEVu2bFELOkIIcejQIdG2bVuhVCqFlZWV8PPzk0JKQUGBWLZsmahfv74wMjISdnZ2ws/PTxw+fFjq//vvv4s6deoIpVIpfHx8xNq1a4sNOtnZ2cLGxkZ88cUXRW6bRYsWCXt7e2kQ+507d0RQUJA04LhGjRqiX79+aq8zV1fXIvfT+PHjX7AniMqPQohSjjQjIiIiqiR4Hx0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpKt/wPtJAdzRzKk1AAAAABJRU5ErkJggg==",
+ "image/svg+xml": [
+ "\n",
+ "\n",
+ "\n"
+ ],
"text/plain": [
- ""
+ ""
]
},
"metadata": {},
"output_type": "display_data"
- }
- ],
- "source": [
- "differences = [\n",
- " unreduced_ace - reduced_ace for unreduced_ace, reduced_ace in zip(unreduced_aces, reduced_aces)\n",
- "]\n",
- "sns.histplot(differences)\n",
- "plt.title(\"Differences between ACE from unreduced and reduced graph\")\n",
- "plt.xlabel(\"Unreduced ACE - Reduced ACE\")\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "296e27d5-5634-4c83-8772-8dab26376a00",
- "metadata": {},
- "source": [
- "We're in the numerical instability region since this graph is on the scale of 1e-13 - this seems to indicate that reducing the latent variables from the graph does not have an effect on the ACE."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7d5955ff-544b-4e8e-a965-54593b4e825d",
- "metadata": {},
- "source": [
- "# What's the effect of random seed?\n",
- "\n",
- "In the experiment above, a random seed of `seed=500` was generated which resulted in a \"true\" ATE and a \"calculated\" ATE. The conclusion was that because the calculated ATE was positive, that the treatment (X) had a negative effect on the outcome (Y). A few follow-up questions:\n",
- "\n",
- "1. What's the correspondence between these numbers? Should they be the same, or different, and by how much?\n",
- "2. What happens if we pick different random seeds?\n",
- "\n",
- "Below, I picked several different random seeds and calculated the differences, and got this chart. You can see that sometimes, the difference is positive, negative, or close to zero. This means we need a better way of interpreting these results, otherwise setting the random seed is effectively cherry picking."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "id": "a4db4ec5-808c-4fc6-937d-f6626cf38820",
- "metadata": {},
- "outputs": [
+ },
{
"data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGdCAYAAAAIbpn/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmT0lEQVR4nO3df1RU953/8dcgPxUYg8gPKyPEWkGzhtYfSJNtjaFSN8mJKydNmrBL1LXbLtIouxtLN9HqyZZsf0Qbixr3KLbbunZtG6O7W3Mi2ZDdDVhDmo02wMZWMqww6NjAIMKAMt8/epxvJ/6I4Ax3PvB8nHPPce4d3vNyTiqv3vncOzafz+cTAACAgSKsDgAAADBcFBkAAGAsigwAADAWRQYAABiLIgMAAIxFkQEAAMaiyAAAAGNRZAAAgLEirQ4QaoODg2pra1NCQoJsNpvVcQAAwE3w+Xzq7u7WlClTFBFx/fMuo77ItLW1KSMjw+oYAABgGFpbWzV16tTrHh/1RSYhIUHS79+IxMREi9MACCtutzR9euC+3/xGSk62Jg8AP4/Ho4yMDP/v8esZ9UXmysdJiYmJFBkAgbzeq/clJEj8WwGEjY9aFsJiXwAAYCyKDAAAMBZFBgAAGIsiAwAAjEWRAQAAxrK0yGRmZspms121lZaWSpL6+vpUWlqqSZMmKT4+XkVFRero6LAyMgAACCOWFpnjx4+rvb3dv73yyiuSpIceekiStG7dOh0+fFgHDhxQbW2t2tratHz5cisjAwCAMGLpfWQmT54c8PjZZ5/V9OnT9dnPflZdXV3avXu39u3bp8WLF0uSqqurlZOTo/r6ei1cuNCKyAAAIIyEzRqZ/v5+/ehHP9LKlStls9nU0NCggYEBFRQU+J+TnZ0th8Ohurq6687xer3yeDwBGwAAGJ3CpsgcPHhQnZ2devzxxyVJLpdL0dHRmjhxYsDzUlNT5XK5rjunsrJSdrvdv/E9SwAAjF5hU2R2796tpUuXasqUKbc0p6KiQl1dXf6ttbU1SAkBAEC4CYvvWnr//fd19OhR/fznP/fvS0tLU39/vzo7OwPOynR0dCgtLe26s2JiYhQTExPKuAAAIEyExRmZ6upqpaSk6L777vPvmzt3rqKiolRTU+Pf19zcLKfTqfz8fCtiAgCAMGP5GZnBwUFVV1erpKREkZH/P47dbteqVatUXl6upKQkJSYmqqysTPn5+VyxBIxiTqdTbrc7ZPOTk5PlcDhCNh/AyLK8yBw9elROp1MrV6686tiWLVsUERGhoqIieb1eFRYWavv27RakBDASnE6nsrNz1Nt7MWSvERc3Xk1NjZQZYJSwvMgsWbJEPp/vmsdiY2NVVVWlqqqqEU4FwAput1u9vReVt3KjEtMzgz7f096iY3s2ye12U2SAUcLyIgMAH5aYnqkkx0yrYwAwQFgs9gUAABgOigwAADAWRQYAABiLIgMAAIxFkQEAAMaiyAAAAGNRZAAAgLEoMgAAwFgUGQAAYCyKDAAAMBZFBgAAGIsiAwAAjEWRAQAAxqLIAAAAY1FkAACAsSgyAADAWBQZAABgLIoMAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjRVodAIBZnE6n3G53SGY3NjaGZC6A0YsiA+CmOZ1OZWfnqLf3YkhfZ8DbH9L5AEYPigyAm+Z2u9Xbe1F5KzcqMT0z6PPbT9Tp5KFdunTpUtBnAxidKDIAhiwxPVNJjplBn+tpbwn6TACjG4t9AQCAsSgyAADAWBQZAABgLIoMAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjUWQAAICxKDIAAMBYFBkAAGAsigwAADAWRQYAABjL8iJz5swZFRcXa9KkSYqLi9Mf/dEf6c033/Qf9/l82rBhg9LT0xUXF6eCggK99957FiYGAADhwtIi88EHH+iuu+5SVFSUfvGLX+jdd9/Vd7/7Xd12223+53zrW9/S888/r507d+rYsWOaMGGCCgsL1dfXZ2FyAAAQDiKtfPF/+Id/UEZGhqqrq/37srKy/H/2+XzaunWrnnrqKT344IOSpB/+8IdKTU3VwYMH9cgjj4x4ZgAAED4sLTKHDh1SYWGhHnroIdXW1upjH/uY/uqv/kqrV6+WJJ0+fVoul0sFBQX+n7Hb7crLy1NdXd01i4zX65XX6/U/9ng8of+LADBKY2OjJCnygw8050PH3nnnHV36g7PCQ5WcnCyHw3EL6QAMhaVF5re//a127Nih8vJyff3rX9fx48f11a9+VdHR0SopKZHL5ZIkpaamBvxcamqq/9iHVVZWatOmTSHPDsA8vV3nJdlUXFwsSUqWdO5Dz7m3oEDuW3iNuLjxampqpMwAI8TSIjM4OKh58+bpm9/8piTpk5/8pE6ePKmdO3eqpKRkWDMrKipUXl7uf+zxeJSRkRGUvADMNnCxW5JPuY+u1+SsbNl7PNLWJwKes2jt99Q1IXFY8z3tLTq2Z5PcbjdFBhghlhaZ9PR0zZo1K2BfTk6Ofvazn0mS0tLSJEkdHR1KT0/3P6ejo0O5ubnXnBkTE6OYmJjQBAYwKsSnOJTkmKmE7g+uOjZx6sc1LmH4Hy0BGFmWXrV01113qbm5OWDf//7v/2ratGmSfr/wNy0tTTU1Nf7jHo9Hx44dU35+/ohmBQAA4cfSMzLr1q3Tpz/9aX3zm9/UF77wBf3yl7/Url27tGvXLkmSzWbT2rVr9cwzz2jGjBnKysrS008/rSlTpmjZsmVWRgcAAGHA0iIzf/58vfjii6qoqNDmzZuVlZWlrVu36rHHHvM/58knn1RPT4++9KUvqbOzU3fffbeOHDmi2NhYC5MDAIBwYGmRkaT7779f999//3WP22w2bd68WZs3bx7BVAAAwASWf0UBAADAcFFkAACAsSz/aAkYi5xOp9zuW7nt2o1xd1kAYwVFBhhhTqdT2dk56u29GLLX4O6yAMYKigwwwtxut3p7Lypv5UYlpmcGfT53lwUwllBkAIskpmcqyTHT6hgAYDQW+wIAAGNRZAAAgLEoMgAAwFgUGQAAYCyKDAAAMBZFBgAAGIsiAwAAjEWRAQAAxqLIAAAAY1FkAACAsSgyAADAWBQZAABgLIoMAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjRVodAEBoNDY2GjETAG4FRQYYZXq7zkuyqbi4OGSvMeDtD9lsABgKigwwygxc7JbkU+6j6zU5Kzuos9tP1OnkoV26dOlSUOcCwHBRZIBRKj7FoSTHzKDO9LS3BHUeANwqFvsCAABjUWQAAICxKDIAAMBYFBkAAGAsigwAADAWRQYAABiLIgMAAIxFkQEAAMaiyAAAAGNRZAAAgLEoMgAAwFgUGQAAYCyKDAAAMJalReYb3/iGbDZbwJadne0/3tfXp9LSUk2aNEnx8fEqKipSR0eHhYkBAEA4sfyMzOzZs9Xe3u7f/uu//st/bN26dTp8+LAOHDig2tpatbW1afny5RamBQAA4STS8gCRkUpLS7tqf1dXl3bv3q19+/Zp8eLFkqTq6mrl5OSovr5eCxcuHOmoAAAgzFh+Rua9997TlClTdPvtt+uxxx6T0+mUJDU0NGhgYEAFBQX+52ZnZ8vhcKiuru6687xerzweT8AGAABGJ0uLTF5envbu3asjR45ox44dOn36tP74j/9Y3d3dcrlcio6O1sSJEwN+JjU1VS6X67ozKysrZbfb/VtGRkaI/xYAAMAqln60tHTpUv+f58yZo7y8PE2bNk3/8i//ori4uGHNrKioUHl5uf+xx+OhzAAAMEpZ/tHSH5o4caI+8YlP6NSpU0pLS1N/f786OzsDntPR0XHNNTVXxMTEKDExMWADAACjU1gVmQsXLug3v/mN0tPTNXfuXEVFRammpsZ/vLm5WU6nU/n5+RamBAAA4cLSj5b+5m/+Rg888ICmTZumtrY2bdy4UePGjdMXv/hF2e12rVq1SuXl5UpKSlJiYqLKysqUn5/PFUsAAECSxUXm//7v//TFL35R58+f1+TJk3X33Xervr5ekydPliRt2bJFERERKioqktfrVWFhobZv325lZAAAEEYsLTL79++/4fHY2FhVVVWpqqpqhBIBAACThNUaGQAAgKGgyAAAAGNRZAAAgLEoMgAAwFgUGQAAYCyKDAAAMBZFBgAAGIsiAwAAjEWRAQAAxqLIAAAAY1FkAACAsSgyAADAWBQZAABgLIoMAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjUWQAAICxKDIAAMBYFBkAAGAsigwAADAWRQYAABiLIgMAAIxFkQEAAMaiyAAAAGNRZAAAgLEoMgAAwFgUGQAAYCyKDAAAMBZFBgAAGIsiAwAAjEWRAQAAxoq0OgAAjDaNjY0hmZucnCyHwxGS2YCpKDIAECS9Xecl2VRcXByS+XFx49XU1EiZAf4ARQYAgmTgYrckn3IfXa/JWdlBne1pb9GxPZvkdrspMsAfoMgAQJDFpziU5JhpdQxgTGCxLwAAMBZFBgAAGIsiAwAAjBU2RebZZ5+VzWbT2rVr/fv6+vpUWlqqSZMmKT4+XkVFRero6LAuJAAACCthUWSOHz+uF154QXPmzAnYv27dOh0+fFgHDhxQbW2t2tratHz5cotSAgCAcGN5kblw4YIee+wx/eM//qNuu+02//6uri7t3r1bzz33nBYvXqy5c+equrpab7zxhurr6y1MDAAAwoXlRaa0tFT33XefCgoKAvY3NDRoYGAgYH92drYcDofq6uquO8/r9crj8QRsAABgdLL0PjL79+/XW2+9pePHj191zOVyKTo6WhMnTgzYn5qaKpfLdd2ZlZWV2rRpU7CjAgCAMDSsMzK33367zp8/f9X+zs5O3X777Tc1o7W1VU888YR+/OMfKzY2djgxrqmiokJdXV3+rbW1NWizAQBAeBlWkWlpadHly5ev2u/1enXmzJmbmtHQ0KCzZ8/qU5/6lCIjIxUZGana2lo9//zzioyMVGpqqvr7+9XZ2Rnwcx0dHUpLS7vu3JiYGCUmJgZsAABgdBrSR0uHDh3y//nll1+W3W73P758+bJqamqUmZl5U7PuvfdenThxImDfihUrlJ2drfXr1ysjI0NRUVGqqalRUVGRJKm5uVlOp1P5+flDiQ0AAEapIRWZZcuWSZJsNptKSkoCjkVFRSkzM1Pf/e53b2pWQkKC7rjjjoB9EyZM0KRJk/z7V61apfLyciUlJSkxMVFlZWXKz8/XwoULhxIbAACMUkMqMoODg5KkrKwsHT9+XMnJySEJdcWWLVsUERGhoqIieb1eFRYWavv27SF9TQAAYI5hXbV0+vTpYOeQJL322msBj2NjY1VVVaWqqqqQvB4AADDbsC+/rqmpUU1Njc6ePes/U3PFnj17bjkYAADARxlWkdm0aZM2b96sefPmKT09XTabLdi5AAAAPtKwiszOnTu1d+9e/dmf/Vmw8wAAANy0Yd1Hpr+/X5/+9KeDnQUAAGBIhlVk/uIv/kL79u0LdhYAAIAhGdZHS319fdq1a5eOHj2qOXPmKCoqKuD4c889F5RwAAAANzKsIvPOO+8oNzdXknTy5MmAYyz8BQAAI2VYReY//uM/gp0DAABgyIa1RgYAACAcDOuMzD333HPDj5BeffXVYQcCAAC4WcMqMlfWx1wxMDCgt99+WydPnrzqyyQBAABCZVhFZsuWLdfc/41vfEMXLly4pUAAAAA3K6hrZIqLi/meJQAAMGKCWmTq6uoUGxsbzJEAAADXNayPlpYvXx7w2Ofzqb29XW+++aaefvrpoAQDAAD4KMMqMna7PeBxRESEZs6cqc2bN2vJkiVBCQYAAPBRhlVkqqurg50DAABgyIZVZK5oaGhQY2OjJGn27Nn65Cc/GZRQAAAAN2NYRebs2bN65JFH9Nprr2nixImSpM7OTt1zzz3av3+/Jk+eHMyMwIhzOp1yu90hmX2l/AMAbt2wikxZWZm6u7v161//Wjk5OZKkd999VyUlJfrqV7+qf/7nfw5qSGAkOZ1OZWfnqLf3YkhfZ8DbH9L5ADAWDKvIHDlyREePHvWXGEmaNWuWqqqqWOwL47ndbvX2XlTeyo1KTM8M+vz2E3U6eWiXLl26FPTZADDWDKvIDA4OKioq6qr9UVFRGhwcvOVQQDhITM9UkmNm0Od62luCPhMAxqph3RBv8eLFeuKJJ9TW1ubfd+bMGa1bt0733ntv0MIBAADcyLCKzPe//315PB5lZmZq+vTpmj59urKysuTxeLRt27ZgZwQAALimYX20lJGRobfeektHjx5VU1OTJCknJ0cFBQVBDQcAAHAjQzoj8+qrr2rWrFnyeDyy2Wz63Oc+p7KyMpWVlWn+/PmaPXu2/vM//zNUWQEAAAIMqchs3bpVq1evVmJi4lXH7Ha7/vIv/1LPPfdc0MIBAADcyJCKzP/8z//o85///HWPL1myRA0NDbccCgAA4GYMqch0dHRc87LrKyIjI3Xu3LlbDgUAAHAzhlRkPvaxj+nkyZPXPf7OO+8oPT39lkMBAADcjCEVmT/5kz/R008/rb6+vquO9fb2auPGjbr//vuDFg4AAOBGhnT59VNPPaWf//zn+sQnPqE1a9Zo5szf3/W0qalJVVVVunz5sv7u7/4uJEEBAAA+bEhFJjU1VW+88Ya+8pWvqKKiQj6fT5Jks9lUWFioqqoqpaamhiQoAADAhw35hnjTpk3Tv//7v+uDDz7QqVOn5PP5NGPGDN12222hyAcAAHBdw7qzryTddtttmj9/fjCzAAAADMmwvmsJAAAgHFBkAACAsSgyAADAWMNeIwMAGHmNjY0hm52cnCyHwxGy+UAoUGQAwAC9Xecl2VRcXByy14iLG6+mpkbKDIxiaZHZsWOHduzYoZaWFknS7NmztWHDBi1dulSS1NfXp7/+67/W/v375fV6VVhYqO3bt3OvGgBjzsDFbkk+5T66XpOzsoM+39PeomN7NsntdlNkYBRLi8zUqVP17LPPasaMGfL5fPrBD36gBx98UL/61a80e/ZsrVu3Tv/2b/+mAwcOyG63a82aNVq+fLn++7//28rYAGCZ+BSHkhwzrY4BhA1Li8wDDzwQ8Pjv//7vtWPHDtXX12vq1KnavXu39u3bp8WLF0uSqqurlZOTo/r6ei1cuNCKyAAAIIyEzVVLly9f1v79+9XT06P8/Hw1NDRoYGBABQUF/udkZ2fL4XCorq7uunO8Xq88Hk/ABgAARifLi8yJEycUHx+vmJgYffnLX9aLL76oWbNmyeVyKTo6WhMnTgx4fmpqqlwu13XnVVZWym63+7eMjIwQ/w0AAIBVLC8yM2fO1Ntvv61jx47pK1/5ikpKSvTuu+8Oe15FRYW6urr8W2traxDTAgCAcGL55dfR0dH6+Mc/LkmaO3eujh8/ru9973t6+OGH1d/fr87OzoCzMh0dHUpLS7vuvJiYGMXExIQ6NgAACAOWn5H5sMHBQXm9Xs2dO1dRUVGqqanxH2tubpbT6VR+fr6FCQEAQLiw9IxMRUWFli5dKofDoe7ubu3bt0+vvfaaXn75Zdntdq1atUrl5eVKSkpSYmKiysrKlJ+fzxVLAABAksVF5uzZs/rzP/9ztbe3y263a86cOXr55Zf1uc99TpK0ZcsWRUREqKioKOCGeAAAAJLFRWb37t03PB4bG6uqqipVVVWNUCIAAGCSsFsjAwAAcLMoMgAAwFgUGQAAYCyKDAAAMBZFBgAAGIsiAwAAjEWRAQAAxqLIAAAAY1FkAACAsSgyAADAWBQZAABgLIoMAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjUWQAAICxKDIAAMBYFBkAAGAsigwAADAWRQYAABiLIgMAAIxFkQEAAMaiyAAAAGNRZAAAgLEoMgAAwFgUGQAAYCyKDAAAMBZFBgAAGIsiAwAAjEWRAQAAxqLIAAAAY1FkAACAsSKtDgAMh9PplNvtDsnsxsbGkMwFAAQfRQbGcTqdys7OUW/vxZC+zoC3P6TzAQC3jiID47jdbvX2XlTeyo1KTM8M+vz2E3U6eWiXLl26FPTZAIDgosjAWInpmUpyzAz6XE97S9BnAgBCg8W+AADAWBQZAABgLIoMAAAwlqVFprKyUvPnz1dCQoJSUlK0bNkyNTc3Bzynr69PpaWlmjRpkuLj41VUVKSOjg6LEgMAgHBiaZGpra1VaWmp6uvr9corr2hgYEBLlixRT0+P/znr1q3T4cOHdeDAAdXW1qqtrU3Lly+3MDUAAAgXll61dOTIkYDHe/fuVUpKihoaGvSZz3xGXV1d2r17t/bt26fFixdLkqqrq5WTk6P6+notXLjQitgAACBMhNUama6uLklSUlKSJKmhoUEDAwMqKCjwPyc7O1sOh0N1dXXXnOH1euXxeAI2AAAwOoVNkRkcHNTatWt111136Y477pAkuVwuRUdHa+LEiQHPTU1NlcvluuacyspK2e12/5aRkRHq6AAAwCJhU2RKS0t18uRJ7d+//5bmVFRUqKury7+1trYGKSEAAAg3YXFn3zVr1uhf//Vf9frrr2vq1Kn+/Wlpaerv71dnZ2fAWZmOjg6lpaVdc1ZMTIxiYmJCHRkAAIQBS8/I+Hw+rVmzRi+++KJeffVVZWVlBRyfO3euoqKiVFNT49/X3Nwsp9Op/Pz8kY4LAADCjKVnZEpLS7Vv3z699NJLSkhI8K97sdvtiouLk91u16pVq1ReXq6kpCQlJiaqrKxM+fn5XLEEAACsLTI7duyQJC1atChgf3V1tR5//HFJ0pYtWxQREaGioiJ5vV4VFhZq+/btI5wUAACEI0uLjM/n+8jnxMbGqqqqSlVVVSOQCAAAmCRsrloCAAAYKooMAAAwVlhcfo3Ryel0yu12B31uY2Nj0GcCAMxEkUFIOJ1OZWfnqLf3YsheY8DbH7LZAAAzUGQQEm63W729F5W3cqMS0zODOrv9RJ1OHtqlS5cuBXUuAMA8FBmEVGJ6ppIcM4M609PeEtR5AABzsdgXAAAYiyIDAACMRZEBAADGosgAAABjUWQAAICxKDIAAMBYFBkAAGAsigwAADAWRQYAABiLIgMAAIxFkQEAAMaiyAAAAGPxpZEAAL/GxsaQzU5OTpbD4QjZfIxNFBkAgHq7zkuyqbi4OGSvERc3Xk1NjZQZBBVFBgCggYvdknzKfXS9JmdlB32+p71Fx/ZsktvtpsggqCgyAAC/+BSHkhwzrY4B3DQW+wIAAGNRZAAAgLEoMgAAwFgUGQAAYCyKDAAAMBZFBgAAGIsiAwAAjEWRAQAAxqLIAAAAY1FkAACAsSgyAADAWBQZAABgLIoMAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjUWQAAICxLC0yr7/+uh544AFNmTJFNptNBw8eDDju8/m0YcMGpaenKy4uTgUFBXrvvfesCQsAAMKOpUWmp6dHd955p6qqqq55/Fvf+paef/557dy5U8eOHdOECRNUWFiovr6+EU4KAADCUaSVL7506VItXbr0msd8Pp+2bt2qp556Sg8++KAk6Yc//KFSU1N18OBBPfLIIyMZFQAAhCFLi8yNnD59Wi6XSwUFBf59drtdeXl5qquru26R8Xq98nq9/scejyfkWQEAN6exsTEkc5OTk+VwOEIyG+EtbIuMy+WSJKWmpgbsT01N9R+7lsrKSm3atCmk2QAAQ9PbdV6STcXFxSGZHxc3Xk1NjZSZMShsi8xwVVRUqLy83P/Y4/EoIyPDwkQAgIGL3ZJ8yn10vSZnZQd1tqe9Rcf2bJLb7abIjEFhW2TS0tIkSR0dHUpPT/fv7+joUG5u7nV/LiYmRjExMaGOBwAYhvgUh5IcM62OgVEkbO8jk5WVpbS0NNXU1Pj3eTweHTt2TPn5+RYmAwAA4cLSMzIXLlzQqVOn/I9Pnz6tt99+W0lJSXI4HFq7dq2eeeYZzZgxQ1lZWXr66ac1ZcoULVu2zLrQAAAgbFhaZN58803dc889/sdX1raUlJRo7969evLJJ9XT06MvfelL6uzs1N13360jR44oNjbWqsgAACCMWFpkFi1aJJ/Pd93jNptNmzdv1ubNm0cwFQAAMEXYrpEBAAD4KBQZAABgLIoMAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjUWQAAICxwvZLIwEAGIrGxsaQzU5OTuabtcMURQYAYLTervOSbCouLg7Za8TFjVdTUyNlJgxRZAAARhu42C3Jp9xH12tyVnbQ53vaW3Rszya53W6KTBiiyAAARoX4FIeSHDOtjoERxmJfAABgLIoMAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjcfn1GOZ0OuV2u0MyO5R32AQA4AqKzBjldDqVnZ2j3t6LIX2dAW9/SOcDAMY2iswY5Xa71dt7UXkrNyoxPTPo89tP1OnkoV26dOlS0GcDAHAFRWaMS0zPDMmdMD3tLUGfCQDAh7HYFwAAGIsiAwAAjEWRAQAAxqLIAAAAY1FkAACAsSgyAADAWFx+DQCAxUJ5p3VJSk5OlsPhCNl8K1FkAACw0EjcaT0ubryamhpHZZmhyAAAYKFQ32nd096iY3s2ye12U2QAAEBohOpO66Mdi30BAICxOCNzC0K9OMvr9SomJiYksxsbG0MyFwAQnkL1777VC4kpMsM0EouzZLNJPl/o5ksa8PaHdD4AwFq9Xecl2VRcXByS+VYvJKbIDFOoF2e1n6jTyUO7lPvoek3Oyg7Z/EuXLgV9NgAgfAxc7JbkC8nvk3BYSEyRuUWhWpzlaW+RJMWnOEI6HwAwNoTq94nVWOwLAACMxRkZAABuQqgWy3Lxxa2hyAAAcAOhXix7BRdfDI8RRaaqqkrf/va35XK5dOedd2rbtm1asGCB1bEAAGNAKBfLSlx8cavCvsj85Cc/UXl5uXbu3Km8vDxt3bpVhYWFam5uVkpKitXxAABjBBdfhKewX+z73HPPafXq1VqxYoVmzZqlnTt3avz48dqzZ4/V0QAAgMXC+oxMf3+/GhoaVFFR4d8XERGhgoIC1dXVXfNnvF6vvF6v/3FXV5ckyePxBDXbhQsXJEm/e79Zl7y9QZ0tSZ729yVJXWfeU1SkjfkjNJv51s22Yn5fT7c+/C/Dud+clGdCQlDmB9Noe+9Nmc38j5jtckr6/e/EYP+evTLP91E3hvWFsTNnzvgk+d54442A/X/7t3/rW7BgwTV/ZuPGjT5JbGxsbGxsbKNga21tvWFXCOszMsNRUVGh8vJy/+PBwUH97ne/06RJk2SzBb/phiOPx6OMjAy1trYqMTHR6jijCu9t6PDehg7vbejw3oaOz+dTd3e3pkyZcsPnhXWRSU5O1rhx49TR0RGwv6OjQ2lpadf8mZiYmKu+aHHixImhihjWEhMT+R9WiPDehg7vbejw3oYO721o2O32j3xOWC/2jY6O1ty5c1VTU+PfNzg4qJqaGuXn51uYDAAAhIOwPiMjSeXl5SopKdG8efO0YMECbd26VT09PVqxYoXV0QAAgMXCvsg8/PDDOnfunDZs2CCXy6Xc3FwdOXJEqampVkcLWzExMdq4ceNVH7Hh1vHehg7vbejw3oYO7631bD7fR13XBAAAEJ7Ceo0MAADAjVBkAACAsSgyAADAWBQZAABgLIrMGOH1epWbmyubzaa3337b6jjGa2lp0apVq5SVlaW4uDhNnz5dGzduVH9/v9XRjFVVVaXMzEzFxsYqLy9Pv/zlL62OZLzKykrNnz9fCQkJSklJ0bJly9Tc3Gx1rFHp2Weflc1m09q1a62OMuZQZMaIJ5988iNv84yb19TUpMHBQb3wwgv69a9/rS1btmjnzp36+te/bnU0I/3kJz9ReXm5Nm7cqLfeekt33nmnCgsLdfbsWaujGa22tlalpaWqr6/XK6+8ooGBAS1ZskQ9PT1WRxtVjh8/rhdeeEFz5syxOsqYxOXXY8AvfvELlZeX62c/+5lmz56tX/3qV8rNzbU61qjz7W9/Wzt27NBvf/tbq6MYJy8vT/Pnz9f3v/99Sb+/g3dGRobKysr0ta99zeJ0o8e5c+eUkpKi2tpafeYzn7E6zqhw4cIFfepTn9L27dv1zDPPKDc3V1u3brU61pjCGZlRrqOjQ6tXr9Y//dM/afz48VbHGdW6urqUlJRkdQzj9Pf3q6GhQQUFBf59ERERKigoUF1dnYXJRp+uri5J4r/TICotLdV9990X8N8vRlbY39kXw+fz+fT444/ry1/+subNm6eWlharI41ap06d0rZt2/Sd73zH6ijGcbvdunz58lV3605NTVVTU5NFqUafwcFBrV27VnfddZfuuOMOq+OMCvv379dbb72l48ePWx1lTOOMjIG+9rWvyWaz3XBramrStm3b1N3drYqKCqsjG+Nm39s/dObMGX3+85/XQw89pNWrV1uUHLix0tJSnTx5Uvv377c6yqjQ2tqqJ554Qj/+8Y8VGxtrdZwxjTUyBjp37pzOnz9/w+fcfvvt+sIXvqDDhw/LZrP591++fFnjxo3TY489ph/84Aehjmqcm31vo6OjJUltbW1atGiRFi5cqL179yoigv9vMFT9/f0aP368fvrTn2rZsmX+/SUlJers7NRLL71kXbhRYs2aNXrppZf0+uuvKysry+o4o8LBgwf1p3/6pxo3bpx/3+XLl2Wz2RQRESGv1xtwDKFDkRnFnE6nPB6P/3FbW5sKCwv105/+VHl5eZo6daqF6cx35swZ3XPPPZo7d65+9KMf8Y/WLcjLy9OCBQu0bds2Sb//GMThcGjNmjUs9r0FPp9PZWVlevHFF/Xaa69pxowZVkcaNbq7u/X+++8H7FuxYoWys7O1fv16Pr4bQayRGcUcDkfA4/j4eEnS9OnTKTG36MyZM1q0aJGmTZum73znOzp37pz/WFpamoXJzFReXq6SkhLNmzdPCxYs0NatW9XT06MVK1ZYHc1opaWl2rdvn1566SUlJCTI5XJJkux2u+Li4ixOZ7aEhISrysqECRM0adIkSswIo8gAw/DKK6/o1KlTOnXq1FWlkJOcQ/fwww/r3Llz2rBhg1wul3Jzc3XkyJGrFgBjaHbs2CFJWrRoUcD+6upqPf744yMfCAgBPloCAADGYmUiAAAwFkUGAAAYiyIDAACMRZEBAADGosgAAABjUWQAAICxKDIAAMBYFBkAAGAsigwAADAWRQYAABiLIgMAAIxFkQEAAMb6f52NkphEh/JtAAAAAElFTkSuQmCC",
+ "text/markdown": [
+ "**Caveat**: Eliater does not yet have an automated explanation of what the results of this analysis mean.\n"
+ ],
"text/plain": [
- ""
+ ""
]
},
"metadata": {},
@@ -1826,14 +6052,19 @@
}
],
"source": [
- "differences = [\n",
- " background_ate - unreduced_ate\n",
- " for background_ate, unreduced_ate in zip(background_ates, unreduced_aces)\n",
- "]\n",
+ "eliater.step_5_notebook_synthetic(\n",
+ " graph=graph,\n",
+ " reduced_graph=reduced_graph,\n",
+ " example=example,\n",
+ " treatment=treatment,\n",
+ " outcome=outcome,\n",
+ " seed=SEED,\n",
+ ")\n",
"\n",
- "sns.histplot(differences, label=\"Generated ACEs\")\n",
- "plt.axvline(ATE_value - true_ate, linewidth=3, color=\"red\")\n",
- "plt.show()"
+ "# Generate credible interval - use pyro or some other probprog lang\n",
+ "# if the credible interval doesn't cross 0, then you can give i.e. 90% confidence that the effect is positive\n",
+ "# sample 10000 times\n",
+ "# difference is frequentist vs bayesian test"
]
}
],
@@ -1853,7 +6084,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.6"
+ "version": "3.11.9"
}
},
"nbformat": 4,
diff --git a/setup.cfg b/setup.cfg
index 89b98ca..2f08a71 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -49,7 +49,7 @@ keywords =
[options]
install_requires =
- y0>=0.2.7
+ y0>=0.2.8
scipy
numpy
ananke-causal>=0.5.0
@@ -128,12 +128,16 @@ strictness = short
#########################
[flake8]
ignore =
- S301 # pickle
- S403 # pickle
+ # pickle
+ S301
+ # pickle
+ S403
S404
S603
- W503 # Line break before binary operator (flake8 is wrong)
- E203 # whitespace before ':'
+ # Line break before binary operator (flake8 is wrong)
+ W503
+ # whitespace before ':'
+ E203
exclude =
.tox,
.git,
diff --git a/src/eliater/__init__.py b/src/eliater/__init__.py
index 1cace1c..63848c4 100644
--- a/src/eliater/__init__.py
+++ b/src/eliater/__init__.py
@@ -4,7 +4,19 @@
from .api import workflow
from .discover_latent_nodes import remove_nuisance_variables
-from .network_validation import add_ci_undirected_edges, plot_ci_size_dependence
+from .network_validation import (
+ add_ci_undirected_edges,
+ discretize_binary,
+ plot_ci_size_dependence,
+ plot_treatment_and_outcome,
+)
+from .notebook_utils import (
+ step_1_notebook,
+ step_2_notebook,
+ step_3_notebook,
+ step_5_notebook_real,
+ step_5_notebook_synthetic,
+)
from .version import get_version
__all__ = [
@@ -12,6 +24,13 @@
"remove_nuisance_variables",
"add_ci_undirected_edges",
"plot_ci_size_dependence",
+ "plot_treatment_and_outcome",
+ "discretize_binary",
+ "step_1_notebook",
+ "step_2_notebook",
+ "step_3_notebook",
+ "step_5_notebook_real",
+ "step_5_notebook_synthetic",
]
diff --git a/src/eliater/discover_latent_nodes.py b/src/eliater/discover_latent_nodes.py
index 5cad59f..ec21760 100644
--- a/src/eliater/discover_latent_nodes.py
+++ b/src/eliater/discover_latent_nodes.py
@@ -141,17 +141,16 @@
$Z_4$ is removed as its children are a subset of $Z_1$'s children.
"""
-import itertools
-from typing import Iterable, Optional, Set, Union
+import warnings
+from typing import Set, Union
-import networkx as nx
-
-from y0.algorithm.simplify_latent import simplify_latent_dag
+from y0.algorithm.simplify_latent import evans_simplify
from y0.dsl import Variable
-from y0.graph import DEFAULT_TAG, NxMixedGraph
+from y0.graph import NxMixedGraph, _ensure_set, get_nodes_in_directed_paths
__all__ = [
"remove_nuisance_variables",
+ "find_nuisance_variables",
]
@@ -159,90 +158,23 @@ def remove_nuisance_variables(
graph: NxMixedGraph,
treatments: Union[Variable, Set[Variable]],
outcomes: Union[Variable, Set[Variable]],
- tag: Optional[str] = None,
) -> NxMixedGraph:
"""Find all nuisance variables and remove them based on Evans' simplification rules.
:param graph: an NxMixedGraph
:param treatments: a list of treatments
:param outcomes: a list of outcomes
- :param tag: The tag for which variables are latent
:return: the new graph after simplification
"""
- rv = NxMixedGraph(
- directed=graph.directed.copy(),
- undirected=graph.undirected.copy(),
- )
- lv_dag = mark_nuisance_variables_as_latent(
- graph=rv, treatments=treatments, outcomes=outcomes, tag=tag
- )
- simplified_latent_dag = simplify_latent_dag(lv_dag, tag=tag)
- return NxMixedGraph.from_latent_variable_dag(simplified_latent_dag.graph, tag=tag)
-
-
-def mark_nuisance_variables_as_latent(
- graph: NxMixedGraph,
- treatments: Union[Variable, Set[Variable]],
- outcomes: Union[Variable, Set[Variable]],
- tag: Optional[str] = None,
-) -> nx.DiGraph:
- """Find all the nuisance variables and mark them as latent.
-
- Mark nuisance variables as latent by first identifying them, then creating a new graph where these
- nodes are marked as latent. Nuisance variables are the descendants of nodes in all proper causal paths
- that are not ancestors of the outcome variables nodes. A proper causal path is a directed path from
- treatments to the outcome. Nuisance variables should not be included in the estimation of the causal
- effect as they increase the variance.
-
- :param graph: an NxMixedGraph
- :param treatments: a list of treatments
- :param outcomes: a list of outcomes
- :param tag: The tag for which variables are latent
- :return: the modified graph after simplification, in place
- """
- if tag is None:
- tag = DEFAULT_TAG
nuisance_variables = find_nuisance_variables(graph, treatments=treatments, outcomes=outcomes)
- lv_dag = NxMixedGraph.to_latent_variable_dag(graph, tag=tag)
- # Set nuisance variables as latent
- for node, data in lv_dag.nodes(data=True):
- if Variable(node) in nuisance_variables:
- data[tag] = True
- return lv_dag
-
-
-def find_all_nodes_in_causal_paths(
- graph: NxMixedGraph,
- treatments: Union[Variable, Set[Variable]],
- outcomes: Union[Variable, Set[Variable]],
-) -> Set[Variable]:
- """Find all the nodes in proper causal paths from treatments to outcomes.
-
- A proper causal path is a directed path from treatments to the outcome.
-
- :param graph: an NxMixedGraph
- :param treatments: a list of treatments
- :param outcomes: a list of outcomes
- :return: the nodes on all causal paths from treatments to outcomes.
- """
- if isinstance(treatments, Variable):
- treatments = {treatments}
- if isinstance(outcomes, Variable):
- outcomes = {outcomes}
-
- return {
- node
- for treatment, outcome in itertools.product(treatments, outcomes)
- for causal_path in nx.all_simple_paths(graph.directed, treatment, outcome)
- for node in causal_path
- }
+ return evans_simplify(graph, latents=nuisance_variables)
def find_nuisance_variables(
graph: NxMixedGraph,
treatments: Union[Variable, Set[Variable]],
outcomes: Union[Variable, Set[Variable]],
-) -> Iterable[Variable]:
+) -> Set[Variable]:
"""Find the nuisance variables in the graph.
Nuisance variables are the descendants of nodes in all proper causal paths that are
@@ -255,25 +187,26 @@ def find_nuisance_variables(
:param outcomes: a list of outcomes
:returns: The nuisance variables.
"""
- if isinstance(treatments, Variable):
- treatments = {treatments}
- if isinstance(outcomes, Variable):
- outcomes = {outcomes}
-
- # Find the nodes on all causal paths
- nodes_on_causal_paths = find_all_nodes_in_causal_paths(
- graph=graph, treatments=treatments, outcomes=outcomes
+ treatments = _ensure_set(treatments)
+ outcomes = _ensure_set(outcomes)
+ intermediaries = get_nodes_in_directed_paths(graph, treatments, outcomes)
+ return (
+ graph.descendants_inclusive(intermediaries)
+ - graph.ancestors_inclusive(outcomes)
+ - treatments
+ - outcomes
)
- # Find the descendants of interest
- descendants_of_nodes_on_causal_paths = graph.descendants_inclusive(nodes_on_causal_paths)
-
- # Find the ancestors of outcome variables
- ancestors_of_outcomes = graph.ancestors_inclusive(outcomes)
- descendants_not_ancestors = descendants_of_nodes_on_causal_paths.difference(
- ancestors_of_outcomes
+def find_all_nodes_in_causal_paths(
+ graph: NxMixedGraph,
+ treatments: Union[Variable, Set[Variable]],
+ outcomes: Union[Variable, Set[Variable]],
+) -> Set[Variable]:
+ """Find all the nodes in proper causal paths from treatments to outcomes."""
+ warnings.warn(
+ "This has been replaced with an efficient implementation in y0",
+ DeprecationWarning,
+ stacklevel=1,
)
-
- nuisance_variables = descendants_not_ancestors.difference(treatments.union(outcomes))
- return nuisance_variables
+ return get_nodes_in_directed_paths(graph, treatments, outcomes)
diff --git a/src/eliater/examples/frontdoor_backdoor_discrete/example1.py b/src/eliater/examples/frontdoor_backdoor_discrete/example1.py
index 8496aee..3324066 100644
--- a/src/eliater/examples/frontdoor_backdoor_discrete/example1.py
+++ b/src/eliater/examples/frontdoor_backdoor_discrete/example1.py
@@ -66,7 +66,7 @@ def generate(
*,
seed: int | None = None,
) -> pd.DataFrame:
- """Generate discrete testing data for the multiple_mediators_with_multiple_confounders_nuisances_discrete case study.
+ """Generate discrete test data for the multiple_mediators_with_multiple_confounders_nuisances_discrete case study.
:param num_samples: The number of samples to generate. Try 1000.
:param treatments: An optional dictionary of the values to fix each variable to.
diff --git a/src/eliater/examples/t_cell_signaling_pathway.py b/src/eliater/examples/t_cell_signaling_pathway.py
index 24f506d..f7ebdce 100644
--- a/src/eliater/examples/t_cell_signaling_pathway.py
+++ b/src/eliater/examples/t_cell_signaling_pathway.py
@@ -13,7 +13,7 @@
# does not want to read the reference. Spoon feed the important information
# 2. Is there associated data to go with this graph? Commit it into the examples folder
-
+from eliater.data import load_sachs_df
from y0.algorithm.identify import Query
from y0.examples import Example
from y0.graph import NxMixedGraph
@@ -40,6 +40,7 @@
reference="K. Sachs, O. Perez, D. Pe’er, D. A. Lauffenburger, and G. P. Nolan. Causal protein-signaling"
"networks derived from multiparameter single-cell data. Science, 308(5721): 523–529, 2005.",
graph=graph,
+ data=load_sachs_df(),
description="This is an example of a protein signaling network of the T cell signaling pathway"
"It models the molecular mechanisms and regulatory processes of human cells involved"
"in T cell activation, proliferation, and function. The observational data consisted of quantitative"
diff --git a/src/eliater/network_validation.py b/src/eliater/network_validation.py
index ceaab2c..88195b3 100644
--- a/src/eliater/network_validation.py
+++ b/src/eliater/network_validation.py
@@ -158,6 +158,7 @@
"""
import time
+import warnings
from typing import Optional
import matplotlib.pyplot as plt
@@ -165,15 +166,23 @@
import pandas as pd
import seaborn as sns
from numpy import mean, quantile
+from sklearn.preprocessing import KBinsDiscretizer
from tabulate import tabulate
from tqdm.auto import trange
-from y0.algorithm.conditional_independencies import get_conditional_independencies
+import y0.algorithm.conditional_independencies
from y0.algorithm.falsification import get_graph_falsifications
from y0.graph import NxMixedGraph
-from y0.struct import CITest, _ensure_method, get_conditional_independence_tests
+from y0.struct import (
+ DEFAULT_SIGNIFICANCE,
+ CITest,
+ _ensure_method,
+ get_conditional_independence_tests,
+)
__all__ = [
+ "discretize_binary",
+ "plot_treatment_and_outcome",
"add_ci_undirected_edges",
"print_graph_falsifications",
"p_value_of_bootstrap_data",
@@ -182,7 +191,25 @@
]
TESTS = get_conditional_independence_tests()
-DEFAULT_SIGNIFICANCE = 0.01
+
+
+def plot_treatment_and_outcome(data, treatment, outcome, figsize=(8, 2.5)) -> None:
+ """Plot the treatment and outcome histograms."""
+ fig, (lax, rax) = plt.subplots(1, 2, figsize=figsize)
+ sns.histplot(data=data, x=treatment.name, ax=lax)
+ lax.axvline(data[treatment.name].mean(), color="red")
+ lax.set_title("Treatment")
+
+ sns.histplot(data=data, x=outcome.name, ax=rax)
+ rax.axvline(data[outcome.name].mean(), color="red")
+ rax.set_ylabel("")
+ rax.set_title("Outcome")
+
+
+def discretize_binary(data: pd.DataFrame) -> pd.DataFrame:
+ """Discretize continuous data into binary data using K-Bins Discretization."""
+ kbins = KBinsDiscretizer(n_bins=2, encode="ordinal", strategy="uniform")
+ return pd.DataFrame(kbins.fit_transform(data), columns=data.columns)
def add_ci_undirected_edges(
@@ -204,18 +231,15 @@ def add_ci_undirected_edges(
the tested variables. If none, defaults to 0.05.
:returns: A copy of the input graph potentially with new undirected edges added
"""
- rv = NxMixedGraph(
- directed=graph.directed.copy(),
- undirected=graph.undirected.copy(),
+ warnings.warn(
+ "This method has been replaced by a refactored implementation in "
+ "y0.algorithm.conditional_independencies.add_ci_undirected_edges",
+ DeprecationWarning,
+ stacklevel=1,
+ )
+ return y0.algorithm.conditional_independencies.add_ci_undirected_edges(
+ graph=graph, data=data, method=method, significance_level=significance_level
)
- if significance_level is None:
- significance_level = DEFAULT_SIGNIFICANCE
- for judgement in get_conditional_independencies(rv):
- if not judgement.test(
- data, boolean=True, method=method, significance_level=significance_level
- ):
- rv.add_undirected_edge(judgement.left, judgement.right)
- return rv
def print_graph_falsifications(
diff --git a/src/eliater/notebook_utils.py b/src/eliater/notebook_utils.py
new file mode 100644
index 0000000..3622c6a
--- /dev/null
+++ b/src/eliater/notebook_utils.py
@@ -0,0 +1,677 @@
+"""High-level Eliater workflow for use in Jupyter notebooks."""
+
+import time
+from operator import attrgetter
+from textwrap import dedent
+from typing import Optional
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import seaborn as sns
+from scipy.stats import ttest_1samp
+from tqdm.auto import tqdm, trange
+
+from eliater.discover_latent_nodes import find_nuisance_variables, remove_nuisance_variables
+from eliater.network_validation import discretize_binary
+from eliater.regression import estimate_query_by_linear_regression, get_adjustment_set
+from y0.algorithm.estimation import estimate_ace
+from y0.algorithm.falsification import get_graph_falsifications
+from y0.algorithm.identify import identify_outcomes
+from y0.dsl import Variable
+from y0.examples import Example
+from y0.graph import NxMixedGraph
+from y0.struct import DEFAULT_SIGNIFICANCE, CITest, _ensure_method
+
+
+def display_markdown(s: str) -> None:
+ """Display a markdown string in Jupyter notebook."""
+ import IPython.display
+
+ IPython.display.display(IPython.display.Markdown(dedent(s)))
+
+
+def display_df(df: pd.DataFrame) -> None:
+ """Display a Pandas dataframe in Jupyter notebook."""
+ import IPython.display
+
+ html = df.reset_index(drop=True).to_html(index=False)
+ IPython.display.display(IPython.display.HTML(html))
+
+
+def step_1_notebook(
+ graph: NxMixedGraph,
+ data: pd.DataFrame,
+ *,
+ binarize: bool = False,
+ method: Optional[CITest] = None,
+ max_given: Optional[int] = 5,
+ significance_level: Optional[float] = None,
+ show_all: bool = False,
+ acceptable_percentage: float = 0.3,
+ show_progress: bool = False,
+):
+ """Print the summary of conditional independency test results.
+
+ Prints the summary to the console, which includes the total number of conditional independence tests,
+ the number and percentage of failed tests, and statistical information about each test such as p-values,
+ and test results.
+
+ :param graph: an NxMixedGraph
+ :param data: observational data corresponding to the graph
+ :param binarize: Should the data be discretized into a two-class problem?
+ :param method: the conditional independency test to use. If None, defaults to ``pearson`` for continuous data
+ and ``chi-square`` for discrete data.
+ :param max_given: The maximum set size in the power set of the vertices minus the d-separable pairs
+ :param significance_level: The statistical tests employ this value for
+ comparison with the p-value of the test to determine the independence of
+ the tested variables. If none, defaults to 0.01.
+ :param show_all: If true, shows all p-values from falsification, if false, only shows significant ones
+ :param acceptable_percentage: The percentage of tests that need to fail to output an interpretation
+ that additional edges should be added. Should be between 0 and 1.
+ :param show_progress: If true, shows a progress bar for calculating d-separations
+ """
+ display_markdown("## Step 1: Checking the ADMG Structure")
+ if significance_level is None:
+ significance_level = DEFAULT_SIGNIFICANCE
+ if binarize:
+ data = discretize_binary(data)
+ display_markdown(
+ "On this try, we're going to discretize the data using K-Bins discretization with K as 2."
+ " Here are the first few rows of the transformed dataframe after doing that:"
+ )
+ display_df(data.head())
+
+ start_time = time.time()
+ method = _ensure_method(method, data)
+ evidence_df = get_graph_falsifications(
+ graph=graph,
+ df=data,
+ method=method,
+ significance_level=significance_level,
+ max_given=max_given,
+ verbose=show_progress,
+ sep=";",
+ ).evidence
+ end_time = time.time() - start_time
+ time_text = f"Finished in {end_time:.2f} seconds."
+ n_total = len(evidence_df)
+ n_failed = evidence_df["p_adj_significant"].sum()
+ percent_failed = n_failed / n_total
+ if n_failed == 0:
+ display_markdown(
+ f"All {n_total} d-separations implied by the ADMG's structure are consistent with the data, meaning "
+ f"that none of the data-driven conditional independency tests' null hypotheses with the {method} test "
+ f"were rejected at p<{significance_level}. {time_text}\n"
+ )
+ elif percent_failed < acceptable_percentage:
+ display_markdown( # noqa:T201
+ f"Of the {n_total} d-separations implied by the ADMG's structure, only {n_failed} "
+ f"({percent_failed:.2%}) rejected the null hypothesis for the {method} test at p<{significance_level}."
+ f"\n\nSince this is less than {acceptable_percentage:.0%}, Eliater considers this minor and leaves the "
+ f"ADMG unmodified. {time_text}\n"
+ )
+ else:
+ display_markdown(
+ f"Of the {n_total} d-separations implied by the ADMG's structure, {n_failed} ({percent_failed:.2%}) "
+ f"rejected the null hypothesis with the {method} test at p<{significance_level}.\n\nSince this is more "
+ f"than {acceptable_percentage:.0%}, Eliater considers this a major inconsistency and therefore suggests "
+ f"adding appropriate bidirected edges using the eliater.add_ci_undirected_edges() function. {time_text}\n"
+ )
+ if show_all:
+ dd = evidence_df
+ else:
+ dd = evidence_df[evidence_df["p_adj_significant"]]
+
+ display_markdown(dd.reset_index(drop=True).to_markdown(index=False))
+
+
+def step_2_notebook(*, graph: NxMixedGraph, treatment: Variable, outcome: Variable):
+ """Check query identifiability."""
+ tlatex = treatment.to_latex()
+ olatex = outcome.to_latex()
+ introduction = dedent(
+ f"""
+ ## Step 2: Check Query Identifiability
+
+ The causal query of interest is the average treatment effect of ${tlatex}$ on ${olatex}$, defined as:
+ $\\mathbb{{E}}[{olatex} \\mid do({tlatex}=1)] - \\mathbb{{E}}[{olatex} \\mid do({tlatex}=0)]$.
+ """
+ )
+
+ estimand = identify_outcomes(graph=graph, treatments=treatment, outcomes=outcome)
+ if estimand is None:
+ analysis = dedent(
+ """\
+ The query was not identifiable, so we can not proceed to Step 3.
+ """
+ )
+ else:
+ analysis = dedent(
+ f"""\
+
+ Running the ID algorithm defined by [Identification of joint interventional distributions in recursive
+ semi-Markovian causal models](https://dl.acm.org/doi/10.5555/1597348.1597382) (Shpitser and Pearl, 2006)
+ and implemented in the $Y_0$ Causal Reasoning Engine gives the following estimand:
+
+ ${estimand.to_latex()}$
+
+ Because the query is identifiable, we can proceed to Step 3.
+ """
+ )
+
+ display_markdown(introduction + "\n" + analysis)
+
+
+def step_3_notebook(
+ *, graph: NxMixedGraph, treatment: Variable, outcome: Variable
+) -> Optional[NxMixedGraph]:
+ """Identify nuisance variables and simplify the ADMG."""
+ display_markdown("## Step 3/4: Identify Nuisance Variables and Simplify the ADMG")
+ nv = find_nuisance_variables(graph, treatments=treatment, outcomes=outcome)
+ if not nv:
+ display_markdown(
+ """\
+ No variables were identified as nuisance variables.
+
+ Nevertheless, the algorithm proposed in [Graphs for margins of Bayesian
+ networks](https://arxiv.org/abs/1408.1809) (Evans, 2016) and implemented in
+ the $Y_0$ Causal Reasoning Engine is applied to the ADMG to attempt to
+ simplify the graph by reasoning over its bidirected edges (if they exist).
+ """
+ )
+ else:
+ nv_text = ", ".join(f"${x.to_latex()}$" for x in sorted(nv, key=attrgetter("name")))
+
+ display_markdown(
+ f"""\
+ The following {len(nv)} variables were identified as _nuisance_ variables,
+ meaning that they appear as descendants of nodes appearing in paths between
+ the treatment and outcome, but are not themselves ancestors of the outcome variable:
+
+ {nv_text}
+
+ These variables are marked as "latent", then
+ the algorithm proposed in [Graphs for margins of Bayesian
+ networks](https://arxiv.org/abs/1408.1809) (Evans, 2016) and implemented in
+ the $Y_0$ Causal Reasoning Engine is applied to the ADMG to
+ simplify the graph. This minimally removes the latent variables and makes
+ further simplifications if the latent variables are connected by bidirected
+ edges to other nodes.
+ """
+ )
+
+ new_graph = remove_nuisance_variables(graph, treatments=treatment, outcomes=outcome)
+ if new_graph == graph:
+ display_markdown("The simplification did not modify the graph.")
+ return None
+ new_graph.draw()
+ return new_graph
+
+
+def step_5_notebook_real(
+ *,
+ graph: NxMixedGraph,
+ example: Example,
+ treatment: Variable,
+ outcome: Variable,
+ n_subsamples: int = 500,
+ subsample_size: int = 1_000,
+ eps: float = 1e-10,
+ threshold: float = 0.01,
+):
+ """Calculate the average causal effect and give context on how to interpret it for real data."""
+ subsamples = [
+ example.data.sample(subsample_size)
+ for _ in trange(n_subsamples, desc="Subsampling", leave=False)
+ ]
+ adjustment_set, _ = get_adjustment_set(graph=graph, treatments=treatment, outcome=outcome)
+
+ actual_ananke_ace = estimate_ace(
+ graph, treatments=treatment, outcomes=outcome, data=example.data
+ )
+ actual_linreg_ace = estimate_query_by_linear_regression(
+ graph,
+ treatments=treatment,
+ outcome=outcome,
+ data=example.data,
+ query_type="ate",
+ _adjustment_set=adjustment_set,
+ )
+
+ (
+ ananke_ace_reference,
+ ananke_ace_reference_var,
+ ananke_ace_significance_p_value,
+ linreg_ace_reference,
+ linreg_ace_reference_var,
+ linreg_ace_significance_p_value,
+ linreg_ev_reference,
+ linreg_ev_reference_var,
+ ) = _calc_helper(
+ graph=graph,
+ subsamples=subsamples,
+ treatment=treatment,
+ outcome=outcome,
+ adjustment_set=adjustment_set,
+ )
+ _plot_helper(
+ ananke_ace_reference,
+ ananke_ace_reference_var,
+ ananke_ace_significance_p_value,
+ linreg_ace_reference,
+ linreg_ace_reference_var,
+ linreg_ace_significance_p_value,
+ reference=actual_ananke_ace,
+ reference_right=actual_linreg_ace,
+ )
+
+ plt.tight_layout()
+ plt.show()
+
+
+def step_5_notebook_synthetic(
+ *,
+ graph: NxMixedGraph,
+ reduced_graph: Optional[NxMixedGraph],
+ example: Example,
+ treatment: Variable,
+ outcome: Variable,
+ seed: int = 42,
+ samples: int = 10_000,
+ n_subsamples: int = 500,
+ subsample_size: int = 1_000,
+ eps: float = 1e-10,
+ threshold: float = 0.01,
+):
+ """Calculate the average causal effect and give context on how to interpret it for synthetic data."""
+ tlatex = treatment.to_latex()
+ olatex = outcome.to_latex()
+
+ data_obs = example.generate_data(samples, seed=seed)
+ data_1 = example.generate_data(samples, {treatment: 1.0}, seed=seed)
+ data_0 = example.generate_data(samples, {treatment: 0.0}, seed=seed)
+
+ ates = []
+ for _ in range(n_subsamples):
+ idx = np.random.permutation(samples)[:subsample_size]
+ data_1_mean = data_1.loc[idx][outcome.name].mean()
+ data_0_mean = data_0.loc[idx][outcome.name].mean()
+ diff = data_1_mean - data_0_mean
+ ates.append(diff)
+
+ ate = np.mean(ates)
+ ate_var = np.var(ates)
+
+ display_markdown(
+ f"""\
+ ## Step 5: Estimate the Query
+
+ ### Calculating the True Average Treatment Effect (ATE)
+
+ We first generated synthetic observational data. Now, we generate two interventional datasets:
+ one where we set ${treatment.to_latex()}$ to $0.0$ and one where we set ${treatment.to_latex()}$ to $1.0$.
+ We can then calculate the "true" average treatment effect (ATE) as the difference of the means
+ for the outcome variable ${outcome.to_latex()}$ in each. The ATE is formulated as:
+
+ $ATE = \\mathbb{{E}}[{olatex} \\mid do({tlatex} = 1)] - \\mathbb{{E}}[{olatex} \\mid do({tlatex} = 0)]$
+
+ After generating {samples:,} samples for each distribution, we took {n_subsamples:,} subsamples of size
+ of size {subsample_size:,} and calculated the
+ ATE for each. The variance comes to {ate_var:.1e}, which shows that the ATE is very stable with respect
+ to random generation. We therefore calculate the _true_ ATE as the average value from these samplings,
+ which comes to {ate:.1e}.
+
+ The ATE can be interpreted in the following way:
+
+ 1. If the ATE is positive, it suggests that the treatment ${tlatex}$ has a negative effect on the outcome ${olatex}$
+ 2. If the ATE is negative, it suggests that the treatment ${tlatex}$ has a positive effect on the outcome ${olatex}$
+
+ ### Estimating the Average Treatment Effect (ATE)
+
+ In practice, we are often unable to get the appropriate interventional data, and therefore want to estimate
+ the average treatment effect (ATE) from observational data. Because we're using synthetic data, we generate
+ {samples:,} samples, then took {n_subsamples:,} subsamples of size {subsample_size:,} through which we calculated
+ the following:
+
+ 1. The ATE, using the y0/ananke implementation
+ 2. The ATE, using the Eliater linear regression implementation
+ """
+ )
+
+ subsamples = [
+ data_obs.sample(subsample_size)
+ for _ in trange(n_subsamples, leave=False, desc="Subsampling", unit="sample")
+ ]
+
+ reference_adjustment_set, _ = get_adjustment_set(
+ graph=graph, treatments=treatment, outcome=outcome
+ )
+ (
+ ananke_ace_reference,
+ ananke_ace_reference_var,
+ ananke_ace_significance_p_value,
+ linreg_ace_reference,
+ linreg_ace_reference_var,
+ linreg_ace_significance_p_value,
+ linreg_ev_reference,
+ ev_reference_var,
+ ) = _calc_helper(
+ graph=graph,
+ subsamples=subsamples,
+ treatment=treatment,
+ outcome=outcome,
+ adjustment_set=reference_adjustment_set,
+ )
+ _, ananke_ace_correctness_p_value = ttest_1samp(
+ ananke_ace_reference, ate, alternative="two-sided"
+ )
+ _, linreg_ace_correctness_p_value = ttest_1samp(
+ linreg_ace_reference, ate, alternative="two-sided"
+ )
+
+ if reduced_graph is not None:
+ reduced_adjustment_set, _ = get_adjustment_set(
+ graph=reduced_graph, treatments=treatment, outcome=outcome
+ )
+ (
+ ananke_ace_reduced,
+ ananke_ace_reduced_var,
+ _ananke_ace_significance_p_value,
+ linreg_ace_reduced,
+ linreg_ace_reduced_var,
+ _linreg_ace_significance_p_value,
+ linreg_ev_reduced,
+ linreg_ev_reduced_var,
+ ) = _calc_helper(
+ graph=reduced_graph,
+ subsamples=subsamples,
+ treatment=treatment,
+ outcome=outcome,
+ adjustment_set=reduced_adjustment_set,
+ )
+
+ ananke_ace_diffs = [a - b for a, b in zip(ananke_ace_reference, ananke_ace_reduced)]
+ linreg_ace_diffs = [a - b for a, b in zip(linreg_ace_reference, linreg_ace_reduced)]
+ ev_diffs = [a - b for a, b in zip(linreg_ev_reference, linreg_ev_reduced)]
+
+ fig, axes = plt.subplots(2, 3, figsize=(14, 6.5))
+
+ sns.histplot(ananke_ace_reference, ax=axes[0][0])
+ axes[0][0].set_title(
+ f"ATEs on Original ADMG\nVariance: {ananke_ace_reference_var:.1e}, "
+ f"$p={ananke_ace_significance_p_value:.2e}$"
+ )
+ axes[0][0].set_xlabel("ATE from y0.algorithm.estimation.estimate_ace")
+ axes[0][0].axvline(ate, color="red")
+ sns.histplot(ananke_ace_reduced, ax=axes[0][1])
+ axes[0][1].set_title(f"ATEs on Reduced ADMG\nVariance: {ananke_ace_reduced_var:.1e}")
+ axes[0][1].set_xlabel("ATE from y0.algorithm.estimation.estimate_ace")
+ axes[0][1].set_ylabel("")
+ axes[0][1].axvline(ate, color="red")
+ sns.histplot(ananke_ace_diffs, ax=axes[0][2])
+ axes[0][2].set_xlabel("Reduced ADMG - Original ADMG")
+ axes[0][2].set_ylabel("")
+ axes[0][2].set_title(_diff_subtitle(ananke_ace_diffs, eps))
+
+ sns.histplot(linreg_ace_reference, ax=axes[1][0])
+ axes[1][0].set_title(
+ f"ATEs on Original ADMG\nVariance: {linreg_ace_reference_var:.1e}, "
+ f"$p={linreg_ace_significance_p_value:.2e}$"
+ )
+ axes[1][0].set_xlabel("ATE from eliater.estimate_query")
+ axes[1][0].axvline(ate, color="red")
+ sns.histplot(linreg_ace_reduced, ax=axes[1][1])
+ axes[1][1].set_title(f"ATEs on Reduced ADMG\nVariance: {linreg_ace_reduced_var:.1e}")
+ axes[1][1].set_xlabel("ATE from eliater.estimate_query")
+ axes[1][1].set_ylabel("")
+ axes[1][1].axvline(ate, color="red")
+ sns.histplot(linreg_ace_diffs, ax=axes[1][2])
+ axes[1][2].set_xlabel("Reduced ADMG - Original ADMG")
+ axes[1][2].set_ylabel("")
+ axes[1][2].set_title(_diff_subtitle(linreg_ace_diffs, eps))
+
+ else:
+ _plot_helper(
+ ananke_ace_reference,
+ ananke_ace_reference_var,
+ ananke_ace_significance_p_value,
+ linreg_ace_reference,
+ linreg_ace_reference_var,
+ linreg_ace_significance_p_value,
+ reference=ate,
+ )
+
+ plt.tight_layout()
+ plt.show()
+
+ display_markdown(
+ f"""\
+ Notes on this plot:
+
+ 1. We show the _true_ ATE as a red vertical line
+ 2. We show both this process done with the original ADMG and the reduced ADMG. This shows that the reduction
+ on the ADMG does not affect estimation. However, reduction is still valuable for simplifying visual exploration.
+
+ #### Interpreting $Y_0$/Ananke Estimation of ACE
+
+ {_interpret_nonzero_test(
+ p_value=ananke_ace_significance_p_value,
+ threshold=threshold,
+ treatment=treatment,
+ outcome=outcome,
+ distribution=ananke_ace_reference,
+ label="ananke/y0 estimation of the ACE",
+ )}
+
+ {_interpret_same_ate_test(
+ p_value=ananke_ace_correctness_p_value,
+ threshold=threshold,
+ reference=ate,
+ label="ananke/y0 estimation of the ACE",
+ )}
+
+ #### Interpreting Linear Regression Estimation of ACE
+
+ {_interpret_nonzero_test(
+ p_value=linreg_ace_significance_p_value,
+ threshold=threshold,
+ treatment=treatment,
+ outcome=outcome,
+ distribution=linreg_ace_reference,
+ label="Eliater linear regression estimation of the ACE",
+ )}
+
+ {_interpret_same_ate_test(
+ p_value=linreg_ace_correctness_p_value,
+ threshold=threshold,
+ reference=ate,
+ label="Eliater linear regression estimation of the ACE",
+ )}
+ """
+ )
+
+ display_markdown(
+ f"""\
+ ### Estimating the Expected Value
+
+ We now estimate the query in the form of the expected value:
+
+ $\\mathbb{{E}}[{olatex} \\mid do({tlatex} = 0)]$
+ """
+ )
+
+ if reduced_graph is not None:
+ fig, axes = plt.subplots(1, 3, figsize=(14, 3))
+
+ sns.histplot(linreg_ev_reference, ax=axes[0])
+ axes[0].set_title(
+ f"$E[{olatex} \\mid do({tlatex} = 0)]$ on Original ADMG\nVariance: {ev_reference_var:.1e}"
+ )
+ axes[0].set_xlabel(f"$E[{olatex} \\mid do({tlatex} = 0)]$ from eliater.estimate_query")
+ sns.histplot(linreg_ev_reduced, ax=axes[1])
+ axes[1].set_title(
+ f"$E[{olatex} \\mid do({tlatex} = 0)]$ on Reduced ADMG\nVariance: {linreg_ev_reduced_var:.1e}"
+ )
+ axes[1].set_xlabel(f"$E[{olatex} \\mid do({tlatex} = 0)]$ from eliater.estimate_query")
+ axes[1].set_ylabel("")
+ sns.histplot(ev_diffs, ax=axes[2])
+ axes[2].set_xlabel("Reduced ADMG - Original ADMG")
+ axes[2].set_ylabel("")
+ axes[2].set_title(_diff_subtitle(ev_diffs, eps))
+ plt.tight_layout()
+ else:
+ fig, axis = plt.subplots(1, 1, figsize=(5, 3))
+
+ sns.histplot(linreg_ev_reference, ax=axis)
+ axis.set_title(
+ f"$E[{olatex} \\mid do({tlatex} = 0)]$ on Original ADMG\nVariance: {ev_reference_var:.1e}"
+ )
+ axis.set_xlabel(f"$E[{olatex} \\mid do({tlatex} = 0)]$ from eliater.estimate_query")
+ # plt.tight_layout()
+
+ plt.show()
+
+ display_markdown(
+ """\
+ **Caveat**: Eliater does not yet have an automated explanation of what the results of this analysis mean.
+ """
+ )
+
+
+def _interpret_nonzero_test(p_value, threshold, treatment, outcome, distribution, label):
+ if p_value < threshold:
+ # note that the interpretation is opposite of the sign
+ direction = "negative" if np.mean(distribution) > 0 else "positive"
+ return dedent(
+ f"""\
+ The p-value for testing if {label} is non-zero is {p_value:.2e}, which is below
+ the the significance threshold of {threshold}. Therefore, we reject the
+ null hypothesis of the 1 Sample T-test and conclude that the distribution is significantly
+ different from zero. This means that the treatment ${treatment.to_latex()}$ has
+ *a {direction} effect* on the outcome ${outcome.to_latex()}$.
+ """
+ ).replace("\n", " ")
+ else:
+ return dedent(
+ f"""\
+ The p-value for testing if {label} is non-zero is is {p_value:.2e}, \
+ which is not below the significance threshold of {threshold}. Therefore, we do not reject the \
+ null hypothesis of the 1 Sample T-test and conclude that the distribution is not significantly \
+ different from zero. This means that the treatment ${treatment.to_latex()}$ has *no significant effect* \
+ on the outcome ${outcome.to_latex()}$.
+ """
+ ).replace("\n", " ")
+
+
+def _plot_helper(
+ ananke_ace_reference,
+ ananke_ace_reference_var,
+ ananke_ace_significance_p_value,
+ linreg_ace_reference,
+ linreg_ace_reference_var,
+ linreg_ace_significance_p_value,
+ *,
+ reference,
+ reference_right=None,
+):
+ if reference_right is None:
+ reference_right = reference
+ fig, axes = plt.subplots(1, 2, figsize=(8, 3))
+
+ sns.histplot(ananke_ace_reference, ax=axes[0])
+ axes[0].set_title(
+ f"ATEs on Original ADMG\nVariance: {ananke_ace_reference_var:.1e}, $p={ananke_ace_significance_p_value:.2e}$"
+ )
+ axes[0].set_xlabel("ATE from y0.algorithm.estimation.estimate_ace")
+ axes[0].axvline(reference, color="red")
+ sns.histplot(ananke_ace_reference, ax=axes[0])
+
+ sns.histplot(linreg_ace_reference, ax=axes[1])
+ axes[1].set_title(
+ f"ATEs on Original ADMG\nVariance: {linreg_ace_reference_var:.1e}, $p={linreg_ace_significance_p_value:.2e}$"
+ )
+ axes[1].set_xlabel("ATE from eliater.estimate_query")
+ axes[1].axvline(reference_right, color="red")
+ sns.histplot(linreg_ace_reference, ax=axes[1])
+
+
+def _calc_helper(*, graph, subsamples, treatment, outcome, adjustment_set):
+ ananke_ace_reference = []
+ linreg_ace_reference = []
+ linreg_ev_reference = []
+ for subsample in tqdm(subsamples, desc="Estimating", leave=False):
+ ananke_ace_reference.append(
+ estimate_ace(graph, treatments=treatment, outcomes=outcome, data=subsample)
+ )
+ linreg_ace_reference.append(
+ estimate_query_by_linear_regression(
+ graph,
+ treatments=treatment,
+ outcome=outcome,
+ data=subsample,
+ query_type="ate",
+ _adjustment_set=adjustment_set,
+ )
+ )
+ linreg_ev_reference.append(
+ estimate_query_by_linear_regression(
+ graph,
+ data=subsample,
+ treatments=treatment,
+ outcome=outcome,
+ interventions={treatment: 0},
+ query_type="expected_value",
+ _adjustment_set=adjustment_set,
+ )
+ )
+
+ # Check that the p-values are significantly different from zero to say
+ # if the ATE is significant (then after you can use the sign)
+ _, ananke_ace_significance_p_value = ttest_1samp(
+ ananke_ace_reference, 0, alternative="two-sided"
+ )
+ _, linreg_ace_significance_p_value = ttest_1samp(
+ linreg_ace_reference, 0, alternative="two-sided"
+ )
+
+ return (
+ ananke_ace_reference,
+ np.var(ananke_ace_reference),
+ ananke_ace_significance_p_value,
+ linreg_ace_reference,
+ np.var(linreg_ace_reference),
+ linreg_ace_significance_p_value,
+ linreg_ev_reference,
+ np.var(linreg_ev_reference),
+ )
+
+
+def _interpret_same_ate_test(p_value, threshold, label, reference):
+ if p_value < threshold:
+ return dedent(
+ f"""\
+ The p-value for testing if {label} is different from the reference ATE
+ is {p_value:.2e}, which is below the the significance threshold of {threshold}.
+ Therefore, we reject the null hypothesis of the 1-Sample T-test and conclude that
+ the distribution is significantly different from the ground truth calculated
+ by synthetic generation of interventional data (ATE={reference:.2e}).
+ """
+ ).replace("\n", " ")
+ else:
+ return dedent(
+ f"""\
+ The p-value for testing if {label} is different from the reference ATE
+ is {p_value:.2e}, which is not below the the significance threshold of {threshold}.
+ Therefore, we do not reject the null hypothesis of the 1-Sample T-test and conclude that
+ the distribution is not significantly different from the ground truth calculated
+ by synthetic generation of interventional data (ATE={reference:.2e}).
+ """
+ ).replace("\n", " ")
+
+
+def _diff_subtitle(diffs, eps):
+ if all(diff < eps for diff in diffs):
+ return f"Pairwise Differences\nAll $< {eps:.1e}$ (i.e., artifacts of floating pt. math)"
+ else:
+ return "Pairwise Differences\nSome are significant"
diff --git a/src/eliater/regression.py b/src/eliater/regression.py
index ea69769..f4d3224 100644
--- a/src/eliater/regression.py
+++ b/src/eliater/regression.py
@@ -227,6 +227,7 @@
"""
import statistics
+import warnings
from operator import attrgetter
from typing import Dict, Literal, NamedTuple, Tuple
@@ -242,6 +243,7 @@
__all__ = [
# High-level functions
"estimate_query_by_linear_regression",
+ "estimate_query", # deprecated
"estimate_ate",
"estimate_probabilities",
"summary_statistics",
@@ -329,6 +331,7 @@ def fit_regression(
data: pd.DataFrame,
treatments: Variable | set[Variable],
outcome: Variable,
+ _adjustment_set=None,
) -> RegressionResult:
"""Fit a regression model to the adjustment set over the treatments and a given outcome.
@@ -346,7 +349,10 @@ def fit_regression(
treatments = _ensure_set(treatments)
if len(treatments) > 1:
raise MultipleTreatmentsNotImplementedError
- adjustment_set, _ = get_adjustment_set(graph=graph, treatments=treatments, outcome=outcome)
+ if _adjustment_set:
+ adjustment_set = _adjustment_set
+ else:
+ adjustment_set, _ = get_adjustment_set(graph=graph, treatments=treatments, outcome=outcome)
variable_set = adjustment_set.union(treatments).difference({outcome})
variables = sorted(variable_set, key=attrgetter("name"))
model = LinearRegression()
@@ -362,6 +368,7 @@ def estimate_query_by_linear_regression(
*,
query_type: Literal["ate", "expected_value", "probability"] = "ate",
interventions: Dict[Variable, float] | None = None,
+ _adjustment_set=None,
) -> float | list[float]:
"""Estimate treatment effects using Linear Regression.
@@ -385,6 +392,7 @@ def estimate_query_by_linear_regression(
data=data,
treatments=treatments,
outcome=outcome,
+ _adjustment_set=_adjustment_set,
)
elif query_type in {"expected_value", "probability"}:
@@ -405,11 +413,18 @@ def estimate_query_by_linear_regression(
raise TypeError(f"Unknown query type {query_type}")
+def estimate_query(*args, **kwargs):
+ """Fit a regression model to the adjustment set over the treatments and a given outcome."""
+ warnings.warn("This function was renamed", DeprecationWarning, stacklevel=1)
+ return estimate_query_by_linear_regression(*args, **kwargs)
+
+
def estimate_ate(
graph: NxMixedGraph,
data: pd.DataFrame,
treatments: Variable | set[Variable],
outcome: Variable,
+ _adjustment_set=None,
) -> float:
"""Estimate the average treatment effect (ATE) using Linear Regression.
@@ -426,7 +441,9 @@ def estimate_ate(
treatments = _ensure_set(treatments)
if len(treatments) > 1:
raise MultipleTreatmentsNotImplementedError
- coefficients, _intercept = fit_regression(graph, data, treatments=treatments, outcome=outcome)
+ coefficients, _intercept = fit_regression(
+ graph, data, treatments=treatments, outcome=outcome, _adjustment_set=_adjustment_set
+ )
return coefficients[list(treatments)[0]]
diff --git a/tox.ini b/tox.ini
index 2d61758..8d5b54a 100644
--- a/tox.ini
+++ b/tox.ini
@@ -88,7 +88,7 @@ description = Check that the MANIFEST.in is written properly and give feedback o
skip_install = true
deps =
darglint
- flake8<5.0.0
+ flake8
flake8-black
flake8-bandit
flake8-bugbear
@@ -111,7 +111,9 @@ commands = pyroma --min=10 .
description = Run the pyroma tool to check the package friendliness of the project.
[testenv:mypy]
-deps = mypy
+deps =
+ mypy
+ types-tabulate
skip_install = true
commands = mypy --install-types --non-interactive --ignore-missing-imports src/
description = Run the mypy tool to check static typing on the project.