forked from pivotal-cf/docs-pcf-install
-
Notifications
You must be signed in to change notification settings - Fork 1
/
mysql_bootstrap_ert.html.md.erb
253 lines (197 loc) · 12.2 KB
/
mysql_bootstrap_ert.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
---
title: Recovering MySQL from Elastic Runtime Downtime
owner: RelEng
---
<strong><%= modified_date %></strong>
This topic describes the manual procedure for recovering a terminated Elastic Runtime cluster. In the event that all of the MySQL virtual machines (VMs) are terminated, the cluster will not automatically restart. For example, this process is necessary when all Elastic Runtime VMs are removed on AWS. Follow this procedure to recreate the lost VMs before completing the bootstrapping process to recover the cluster.
## <a id='mysql'></a>Prepare MySQL VMs for Bootstrapping ##
1. Run `bosh deployments` on the Ops Manager Director.
<pre class="terminal">
$ bosh deployments
Acting as user 'director' on 'p-bosh-30c19bdd43c55c627d70'
+-------------------------+-------------------------------+----------------------------------------------+--------------+
| Name | Release(s) | Stemcell(s) | Cloud Config |
+-------------------------+-------------------------------+----------------------------------------------+--------------+
| cf-e82cbf44613594d8a155 | cf-autoscaling/28 | bosh-aws-xen-hvm-ubuntu-trusty-go_agent/3140 | none |
| | cf-mysql/23 | | |
| | cf/225 | | |
| | diego/0.1441.0 | | |
| | etcd/18 | | |
| | garden-linux/0.327.0 | | |
| | notifications-ui/10 | | |
| | notifications/19 | | |
| | push-apps-manager-release/397 | | |
+-------------------------+-------------------------------+----------------------------------------------+--------------+
</pre>
1. Download the manifest.
<pre class="terminal">
$ bosh download manifest cf-e82cbf44613594d8a155 /tmp/cf.yml
Acting as user 'director' on deployment 'cf-e82cbf44613594d8a155' on 'p-bosh-30c19bdd43c55c627d70'
Deployment manifest saved to `/tmp/cf.yml'
</pre>
1. Set BOSH to use the deployment manifest you downloaded.
<pre class="terminal">
$ bosh deployment /tmp/cf.yml
</pre>
1. Run `bosh vms` to check for placeholder VMs.
<pre class="terminal">
$ bosh vms
Acting as user 'director' on 'p-bosh-30c19bdd43c55c627d70'
Deployment `cf-e82cbf44613594d8a155'
Director task 33
Task 33 done
+-------------------------+--------------------+------------------+-------+
| Job/index | State | Resource Pool | IPs |
+-------------------------+--------------------+------------------+-------+
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
</pre>
<p class="note"><strong>Note</strong>: If you do not see these three <code>unknown/unknown</code> items on your list, you might see the <code>mysql-partition</code> VMs, which indicates that your VMs were not destroyed. If that is the case, disregard the rest of the instructions in this section, Prepare MySQL VMs for Bootstrapping, and follow the instructions in the <a href="#bootstrapping">Bootstrap</a> section below.</p>
1. Run the BOSH Cloud Check interactive command `bosh cck` to delete the placeholder VMs. If given the option, select **Delete VM reference**.
<pre class="terminal">
$ bosh cck
Acting as user 'director' on deployment 'cf-e82cbf44613594d8a155' on 'p-bosh-30c19bdd43c55c627d70'
Performing cloud check...
Director task 34
Started scanning 22 vms
Started scanning 22 vms > Checking VM states. Done (00:00:10)
Started scanning 22 vms > 19 OK, 0 unresponsive, 3 missing, 0 unbound, 0 out of sync. Done (00:00:00)
Done scanning 22 vms (00:00:10)
Started scanning 10 persistent disks
Started scanning 10 persistent disks > Looking for inactive disks. Done (00:00:02)
Started scanning 10 persistent disks > 10 OK, 0 missing, 0 inactive, 0 mount-info mismatch. Done (00:00:00)
Done scanning 10 persistent disks (00:00:02)
Task 34 done
Started 2015-11-26 01:42:42 UTC
Finished 2015-11-26 01:42:54 UTC
Duration 00:00:12
Scan is complete, checking if any problems found.
Found 3 problems
Problem 1 of 3: VM with cloud ID <span>`</span>i-afe2801f' missing.
1. Skip for now
2. Recreate VM
3. Delete VM reference
Please choose a resolution [1 - 3]: 3
Problem 2 of 3: VM with cloud ID `i-36741a86' missing.
1. Skip for now
2. Recreate VM
3. Delete VM reference
Please choose a resolution [1 - 3]: 3
Problem 3 of 3: VM with cloud ID <span>`</span>i-ce751b7e' missing.
1. Skip for now
2. Recreate VM
3. Delete VM reference
Please choose a resolution [1 - 3]: 3
Below is the list of resolutions you've provided
Please make sure everything is fine and confirm your changes
1. VM with cloud ID `i-afe2801' missing.
Delete VM reference
2. VM with cloud ID `i-36741a86' missing.
Delete VM reference
3. VM with cloud ID `i-ce751b7e' missing.
Delete VM reference
Apply resolutions? (type 'yes' to continue): yes
Applying resolutions...
Director task 35
Started applying problem resolutions
Started applying problem resolutions > missing<span>_</span>vm 11: Delete VM reference. Done (00:00:00)
Started applying problem resolutions > missing<span>_</span>vm 27: Delete VM reference. Done (00:00:00)
Started applying problem resolutions > missing<span>_</span>vm 26: Delete VM reference. Done (00:00:00)
Done applying problem resolutions (00:00:00)
Task 35 done
Started 2015-11-26 01:47:08 UTC
Finished 2015-11-26 01:47:08 UTC
Duration 00:00:00
Cloudcheck is finished
</pre>
1. Run `bosh edit deployment` to launch a `vi` editor and modify the deployment.
<pre class="terminal">
$ bosh edit deployment
</pre>
1. Search for the jobs section: `jobs`.
1. Search for the mysql-partition: `name: mysql-partition`.
1. Search for the update section: `update`.
1. Update `max_in_flight` to `3`.
1. Add a line: `canaries: 0` below the `max_in_flight` line.
1. Set `update.serial` to `false`.
1. Run `bosh deploy` to apply these changes.
<p class="note"><strong>Note</strong>: Review the changes for accuracy. Only type `yes` if the changes listed are correct.</p>
<pre class="terminal">
Jobs
mysql-partition-f8abb6ac43ccc9fe16d7
update
± max_in_flight:
- 1
+ 3
+ canaries: 0<br>
Properties
No changes<br>
Please review all changes carefully<br>
Deploying
\---------
Are you sure you want to deploy? (type 'yes' to continue):
</pre>
After you type `yes`, BOSH attempts to re-create the lost VMs:
<pre class="terminal">
Done creating bound missing vms > mysql-partition-f8abb6ac43ccc9fe16d7/0 (00:01:54)
Done creating bound missing vms > mysql-partition-f8abb6ac43ccc9fe16d7/2 (00:01:54)
Done creating bound missing vms > mysql-partition-f8abb6ac43ccc9fe16d7/1 (00:02:04)
</pre>
Then BOSH fails to update the lost VMs. You can ignore the error message.
<pre class="terminal">
Failed updating job mysql-partition-f8abb6ac43ccc9fe16d7 > mysql-partition-f8abb6ac43ccc9fe16d7/1: \`mysql-partition-f8abb6ac43ccc9fe16d7/1' is not running after update (00:05:57)
Failed updating job mysql-partition-f8abb6ac43ccc9fe16d7 > mysql-partition-f8abb6ac43ccc9fe16d7/0: \`mysql-partition-f8abb6ac43ccc9fe16d7/0' is not running after update (00:05:59)
Failed updating job mysql-partition-f8abb6ac43ccc9fe16d7 > mysql-partition-f8abb6ac43ccc9fe16d7/2: \`mysql-partition-f8abb6ac43ccc9fe16d7/2' is not running after update (00:06:01)
Failed updating job mysql-partition-f8abb6ac43ccc9fe16d7 (00:06:01)
Error 400007: \`mysql-partition-f8abb6ac43ccc9fe16d7/1' is not running after update
</pre>
1. Run `bosh vms` a second time to validate that you now have the VMs all in the failing state.
<pre class="terminal">
$ bosh vms
Acting as user 'director' on 'p-bosh-30c19bdd43c55c627d70'
Deployment `cf-e82cbf44613594d8a155'
Director task 33
Task 33 done
+------------------------------------------+-----------+-----------------------------------------+------------+
| Job/index | State | Resource Pool | IPs |
+------------------------------------------+-----------+-----------------------------------------+------------+
| mysql-partition-f8abb6ac43ccc9fe16d7/0 | failing | mysql-partition-f8abb6ac43ccc9fe16d7 | 10.0.16.12 |
| mysql-partition-f8abb6ac43ccc9fe16d7/1 | failing | mysql-partition-f8abb6ac43ccc9fe16d7 | 10.0.16.60 |
| mysql-partition-f8abb6ac43ccc9fe16d7/2 | failing | mysql-partition-f8abb6ac43ccc9fe16d7 | 10.0.16.61 |
</pre>
Next, complete the bootstrapping process in the following section.
## <a id='bootstrapping'></a>Bootstrap ##
<p class='note'><strong>Note</strong>: Bootstrapping requires you to run commands from the <a href="./../customizing/index.html">Ops Manager Director</a>. Follow the instructions to use the <a href="./../customizing/trouble-advanced.html#prepare">BOSH CLI</a> for command-line access.</p>
Bootstrapping identifies the primary node to synchronize all of the nodes in a cluster. You must run all of the steps in this section to ensure successful future deployments and accurate reporting of the status of your jobs.
1. Follow the [Bootstrap](https://docs.pivotal.io/p-mysql/bootstrapping.html) process to restart your MySQL cluster.
1. Run `bosh vms` to verify that your MySQL cluster is functioning as expected:
<pre class="terminal">
$ bosh vms
Acting as user 'director' on 'p-bosh-30c19bdd43c55c627d70'
Deployment `cf-e82cbf44613594d8a155'
Director task 33
Task 33 done
+------------------------------------------+-----------+-----------------------------------------+------------+
| Job/index | State | Resource Pool | IPs |
+------------------------------------------+-----------+-----------------------------------------+------------+
| mysql-partition-f8abb6ac43ccc9fe16d7/0 | running | mysql-partition-f8abb6ac43ccc9fe16d7 | 10.0.16.12 |
| mysql-partition-f8abb6ac43ccc9fe16d7/1 | running | mysql-partition-f8abb6ac43ccc9fe16d7 | 10.0.16.60 |
| mysql-partition-f8abb6ac43ccc9fe16d7/2 | running | mysql-partition-f8abb6ac43ccc9fe16d7 | 10.0.16.61 |
</pre>
1. Run `bosh edit deployment` to launch a `vi` editor and reset the deployment manifest.
<pre class="terminal">
$ bosh edit deployment
</pre>
1. Set `update.canaries` to `1`.
1. Set `update.max_in_flight` to `1`.
1. Set `update.serial` to `true`.
1. Run `bosh deploy` to apply these changes.
## <a id='common'></a>Resolve Common Issues ##
- You may experience an error for too many key identities loaded in your authentication agent:
<pre class="terminal">
\> Received disconnect from 10.0.1.19: 2: Too many authentication failures for bosh_64898ue98
</pre>
If so, you can clear all of them using `ssh-add -D`.
- If you have an issue with not being able to `monit start`, it may be due to a bug in `mariadb_ctrl` that does not collect a defunct process. In that case, call Pivotal support for help. You can also see the [Known Issues](../../../p-mysql/known-issues.html) topic for more information.
</pre>