HDD deduplication does nothing?

[daemongloom]

Is there anything special required for HDD deduplication? Custom DS3622xs+, 16GB of RAM, DSM 7.2.1-69057 Update 3.
Storage pool with 2x2TB SATA SSD was deduplicated successfully and it works fine for a few months already. But a pool of 2x14TB HDDs does not get any data deduplicated (automatic does nothing, manual shows deduplication progress bar, ram usage increases - but it still does not deduplicate anything). That disk contains multiple copies of photos/videos in some directories, so it definitely should have some data for deduplication.

Script result:

Synology_enable_Deduplication v1.2.15
DS3622xs+ DSM 7.2.1-69057-3
StorageManager 1.0.0-0017
Running from: /volume2/homes/admin/syno_enable_dedupe.sh
NAS has 16 GB of memory.
Btrfs Data Deduplication already enabled.
Dedupe config menu for HDDs and 2.5" SSDs already enabled.
Finished

[007revad]

Deduplication works on my DS1821+ on a single HDD. I didn't try it on a pool with more drives.

I was going to ask if you have tried the deduplication analyzer to see if it says there are duplicate files, but I remembered the deduplication analyzer option only shows if you have never run deduplication on the volume.

What does the following return:

cat /var/log/synobtrfsdedupe.log | grep -A 33 "Init new data dedupe workflow" | cut -d":" -f1-2,5-8

[007revad]

Do you know if there's actually any duplicate data on those volumes?

Total data scanned is the amount of data it scanned. It seems to only scan data that is new or changed since the time dedupe was run.
Total release is the amount saved.

I suspect that previous dedupe runs found and deduped data, but you're not seeing the free space change because you have snapshots enabled. Deduplication does not deduplicate data stored in snapshots. So the free space should be reclaimed when the current snapshots get deleted by your snapshot schedule.

I have a small single HDD with very little on it, and no snapshots. If I copy a 2GB file from my PC to that drive, and then copy that same 2GB file from my PC to a different folder on that drive and run deduplication on that HDD's volume I get:

HDD volume (volume1):

2024-01-10T16:10:02+11:00 SENNA synobtrfsdedupe[15387]: ----- Init new data dedupe workflow: /volume1 -----
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Total data scanned : 4.87 GB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Hole : 0.00 B
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Small extent : 1.26 MB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Cloned extent : 6.26 MB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Rewrote extent : 0.00 B
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Not found error count : 0
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Dedupe :
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Extent-same requested : 2.06 GB (21)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Success : 2.06 GB (21)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Zero : 0.00 B (0)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Failed by diff data : 0.00 B (0)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Failed by toxic : 0.00 B (0)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Failed by src not fount : 0.00 B (0)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Failed by dst not fount : 0.00 B (0)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Failed by other error : 0.00 B (0)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Too small : 0.00 B (0)
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Success ranges :
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: 128K - 256K : 408.00 KB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: 256K - 512K : 0.00 B
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: 512K - 1M : 744.00 KB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: 1M - 4M : 2.20 MB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: 4M - 16M : 0.00 B
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: 16M - 64M : 54.06 MB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: 64M - : 2.00 GB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Release by extent-same : 2.00 GB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Release by reclaim : 54.06 MB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Total release : 2.05 GB
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Sample hit : 21
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Load bucket : 6
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Cache hit : 15
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Record rotated : 0
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: Rotated fp size : 0.00 B
2024-01-10T16:10:53+11:00 SENNA synobtrfsdedupe[15387]: ----- End new data dedupe workflow: /volume1 -----

[daemongloom]

I've copied 30GB file from Synology to my PC and copied it back 3 times.
Free space was reduced from 2.69TB to 2.6TB.
I've started manual deduplication on this volume, free space was temporarily reduced to 2.51TB.
After deduplication process it went to 2.6TB, so these 3 copies are still there as separate files.
I'm not using any snapshots (LUN or volume) as far as I know. Unless Hyper Backup client uses it without any indication for user. I've installed "Snapshot Replication" to check it and it doesn't show any existing snapshots.

Log for that deduplication run:

2024-01-10T12:06: ----- Init new data dedupe workflow: /volume2 -----
2024-01-10T12:47: Total data scanned : 213.21 GB
2024-01-10T12:47: Hole : 560.00 KB
2024-01-10T12:47: Small extent : 40.11 MB
2024-01-10T12:47: Cloned extent : 213.14 GB
2024-01-10T12:47: Rewrote extent : 0.00 B
2024-01-10T12:47: Not found error count : 0
2024-01-10T12:47: Dedupe :
2024-01-10T12:47: Extent-same requested : 214.39 GB (10545)
2024-01-10T12:47: Success : 212.89 GB (10060)
2024-01-10T12:47: Zero : 0.00 B (0)
2024-01-10T12:47: Failed by diff data : 332.00 KB (1)
2024-01-10T12:47: Failed by toxic : 145.47 MB (491)
2024-01-10T12:47: Failed by src not fount : 0.00 B (0)
2024-01-10T12:47: Failed by dst not fount : 0.00 B (0)
2024-01-10T12:47: Failed by other error : 0.00 B (0)
2024-01-10T12:47: Too small : 44.45 MB (763)
2024-01-10T12:47: Success ranges :
2024-01-10T12:47: 128K - 256K : 2.74 MB
2024-01-10T12:47: 256K - 512K : 8.84 MB
2024-01-10T12:47: 512K - 1M : 36.09 MB
2024-01-10T12:47: 1M - 4M : 318.71 MB
2024-01-10T12:47: 4M - 16M : 8.20 GB
2024-01-10T12:47: 16M - 64M : 143.58 GB
2024-01-10T12:47: 64M - : 60.75 GB
2024-01-10T12:47: Release by extent-same : 344.00 KB
2024-01-10T12:47: Release by reclaim : 0.00 B
2024-01-10T12:47: Total release : 344.00 KB
2024-01-10T12:47: Sample hit : 11329
2024-01-10T12:47: Load bucket : 86
2024-01-10T12:47: Cache hit : 14286
2024-01-10T12:47: Record rotated : 0
2024-01-10T12:47: Rotated fp size : 0.00 B
2024-01-10T12:47: ----- End new data dedupe workflow: /volume2 -----

[daemongloom]

I've tried once more.
Free space on disk:
2.6TB (before another three copies)
2.5TB (after additional three copies)
2.42TB (while deduplicating)
2.5TB (after deduplication).

I've deleted all additional 6 copies from two runs and free space is back to 2.69TB (exact same size as it was before all additional copies).

Log for second run:

2024-01-10T14:19: ----- Init new data dedupe workflow: /volume2 -----
2024-01-10T14:41: Total data scanned : 91.29 GB
2024-01-10T14:41: Hole : 0.00 B
2024-01-10T14:41: Small extent : 4.75 MB
2024-01-10T14:41: Cloned extent : 91.28 GB
2024-01-10T14:41: Rewrote extent : 0.00 B
2024-01-10T14:41: Not found error count : 0
2024-01-10T14:41: Dedupe :
2024-01-10T14:41: Extent-same requested : 91.29 GB (1807)
2024-01-10T14:41: Success : 91.29 GB (1807)
2024-01-10T14:41: Zero : 0.00 B (0)
2024-01-10T14:41: Failed by diff data : 0.00 B (0)
2024-01-10T14:41: Failed by toxic : 2.27 MB (3)
2024-01-10T14:41: Failed by src not fount : 0.00 B (0)
2024-01-10T14:41: Failed by dst not fount : 0.00 B (0)
2024-01-10T14:41: Failed by other error : 0.00 B (0)
2024-01-10T14:41: Too small : 776.00 KB (42)
2024-01-10T14:41: Success ranges :
2024-01-10T14:41: 128K - 256K : 1.12 MB
2024-01-10T14:41: 256K - 512K : 7.45 MB
2024-01-10T14:41: 512K - 1M : 43.66 MB
2024-01-10T14:41: 1M - 4M : 292.44 MB
2024-01-10T14:41: 4M - 16M : 2.33 GB
2024-01-10T14:41: 16M - 64M : 21.76 GB
2024-01-10T14:41: 64M - : 66.86 GB
2024-01-10T14:41: Release by extent-same : 0.00 B
2024-01-10T14:41: Release by reclaim : 0.00 B
2024-01-10T14:41: Total release : 0.00 B
2024-01-10T14:41: Sample hit : 1866
2024-01-10T14:41: Load bucket : 42
2024-01-10T14:41: Cache hit : 3123
2024-01-10T14:41: Record rotated : 0
2024-01-10T14:41: Rotated fp size : 0.00 B
2024-01-10T14:41: ----- End new data dedupe workflow: /volume2 -----

[daemongloom]

I've finally found a solution. You were right, it was because of snapshots that were created by Synology Drive. They aren't shown anywhere but they do exist and that causes deduplication to fail. After disabling history in Synology Drive free space has increased greatly. Tests with copies of that file and manual deduplication are successful now.

[MaximumFish]

Just posting to say that this thread has been helpful for troubleshooting my own HDD dedupe and helping to confirm that it's at least working as expected. Just surprised and a little disappointed that it's only managed to save ~35GB from 8.5TB worth of data!

Edit: I note that the Storage Manager has a "Deduplication Savings" section when looking at deduped SSDs but not for HDDs. Is that normal (seeing as HDDs aren't officially supported)? Or should I be rebooting or something 🤔

[ThomasGoering]

Edit: I note that the Storage Manager has a "Deduplication Savings" section when looking at deduped SSDs but not for HDDs. Is that normal (seeing as HDDs aren't officially supported)? Or should I be rebooting or something 🤔

#67

[MaximumFish]

Cheers, I'll skip the reboot then! Thanks for your work on this BTW.