[TUTORIAL] RESCHEDULE MDADM scrubbing process out of wdPoST

[s0nik42]

DISCLAIMIER : DON'T DO THAT DURING YOUR SCRUBBING PERIOD if you don't want all your scrubbing to start again.

Context

by default the first sunday of each month mdadm start scrubbing all arrays. This process is creating massive IO, slowing down the miner and could lead to missing the wdPoST.

How is works (on Ubuntu)

mdadm scrubbing consists in 2 jobs triggered by systemd :

You can see these jobs running this command :

 sudo systemctl list-timers
NEXT                         LEFT                LAST                         PASSED        UNIT                                    ACTIVATES
Tue 2021-09-07 11:06:13 CEST 13h left            Mon 2021-09-06 13:54:31 CEST 8h ago        mdcheck_continue.timer                  mdcheck_continue.service
Sat 2021-10-02 11:45:13 CEST 3 weeks 4 days left Sat 2021-09-04 11:06:41 CEST 2 days ago    mdcheck_start.timer                     mdcheck_start.service

1. The first time trigger "mdcheck_start.timer" contains the following conditions :

```
[Timer]
OnCalendar=Sun *-*-1..7 1:00:00
RandomizedDelaySec=24h
```

Meaning : If it's 1AM and we are a Sunday that is between the 1st to the 7th day of the month then let's wait for a Random time picked up between 0 and 24H before starting the job mdcheck_start.service.

mdmonitor.service contains :

Environment=MDADM_CHECK_DURATION='"6 hours"'
ExecStart=/usr/share/mdadm/mdcheck --duration $MDADM_CHECK_DURATION

Meaning : Lets execute mdcheck for 6Hours then stop the scrubbing if not finished. After these 6Hours, mdcheck will stop, record the position where it stopped and wait another job to resume the scrubbing the day after for 6more hours.

2. The second time trigger "mdcheck_continue.timer" contains the following conditions :

[Timer]
OnCalendar=daily
RandomizedDelaySec=12h

Meaning : everyday at midnight lets wait for a Random time picked up between 0 and 12H before starting the job mdcheck_continue.service.

Environment= MDADM_CHECK_DURATION='"6 hours"'
ExecStart=/usr/share/mdadm/mdcheck --continue --duration $MDADM_CHECK_DURATION

Meaning : if will run mdcheck in a "continue" mode for 6Hours then stop and what for the day after to resume and so on until the array have bin completely scrubbed

How make it suitable for your miner

Consideration :

1. Look at your wdPoST time
2. Scrubbing is 6 hours a day
3. Scrubbing start at a random time for a good reason its never a good idea to start heavy workload at a fix time. Try to keep that logic. 

How to modify :

1. All the files are located here : /lib/systemd/system/ (in /etc you only have symlinks)

2. adjust OnCalendar and RandomizedDelaySec of mdcheck_start.timer and mdcheck_continue.timer to match your needs.

3. OnCalendar strings can be safely tested with the folowing command line :

systemd-analyze calendar examples :

systemd-analyze calendar "Sat *-*-1..7 9:00:00"
  Original form: Sat *-*-1..7 9:00:00
Normalized form: Sat *-*-01..07 09:00:00
    Next elapse: Sat 2021-10-02 09:00:00 CEST
       (in UTC): Sat 2021-10-02 07:00:00 UTC
       From now: 3 weeks 4 days left

4. Once all your modification have been made, you need to run the following command for all the settings to apply.

systemctl daemon-reload

5. You can verifiy your modifications have been taken into account with :

sudo systemctl list-timers

[s0nik42]

@jennijuju

[stuberman]

What is the scope of this? Does it apply to ZFS pool scrubbing?

[s0nik42]

mdadm only, zfs does'nt rely on it