forked from hachi/MogileFS-Server
-
Notifications
You must be signed in to change notification settings - Fork 99
/
CHANGES
1064 lines (653 loc) · 42.9 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2018-01-18: Release version 2.73
* fsck: avoid infinite wait on dead devices (Eric Wong <e@80x24.org>)
* client: always disable watch_read after a command (Eric Wong <e@80x24.org>)
* client: use single write for admin commands (Eric Wong <e@80x24.org>)
* tracker: client fairness, backpressure, and expiry (Eric Wong <e@80x24.org>)
* client connection should always be nonblocking (Eric Wong <e@80x24.org>)
* ConnectionPool: avoid undefined behavior for hash iteration (Eric Wong <e@80x24.org>)
* replicate: avoid buffered IO on reads (Eric Wong <e@80x24.org>)
* enable DB upgrade for host readonly state (Eric Wong <e@80x24.org>)
* replicate: reduce backoff for too_happy FIDs (Eric Wong <e@80x24.org>)
* fsck: this avoid redundant fsck log entries (Eric Wong <normalperson@yhbt.net>)
* fsck: do not log FOND if note_on_device croaks (Eric Wong <normalperson@yhbt.net>)
* reaper: detect resurrection of "dead" devices (Eric Wong <normalperson@yhbt.net>)
2014-12-15: Release version 2.72
* Work with DBD::SQLite's latest lock errors (dormando <dormando@rydia.net>)
* remove update_host_property (Eric Wong <e@80x24.org>)
* remove users of unreachable_fids table (Eric Wong <e@80x24.org>)
* monitor: batch MySQL device table updates (Eric Wong <normalperson@yhbt.net>)
* monitor: defer DB updates until all HTTP requests are done (Eric Wong <normalperson@yhbt.net>)
* connection/poolable: defer expiry of timed out connections (Eric Wong <e@80x24.org>)
* connection/poolable: disable watch_write before retrying write (Eric Wong <normalperson@yhbt.net>)
* connection/poolable: do not write before event_write (Eric Wong <normalperson@yhbt.net>)
* add conn_pool_size configuration option (Eric Wong <normalperson@yhbt.net>)
* enable TCP keepalives for iostat watcher sockets (Eric Wong <normalperson@yhbt.net>)
* host: add "readonly" state to override device "alive" state (Eric Wong <normalperson@yhbt.net>)
* add LICENSE file to distro (dormando <dormando@rydia.net>)
2013-08-18: Release version 2.70
* This release features a very large rewrite to the Monitor worker to run
checks in parallel. There are no DB schema changes.
* replicate: use persistent connection from pool if possible (Eric Wong <normalperson@yhbt.net>)
* replicate: enforce expected Content-Length in http_copy (Eric Wong <normalperson@yhbt.net>)
* create_open: parallelize directory vivification (Eric Wong <normalperson@yhbt.net>)
* device: reuse HTTP connections for MKCOL (Eric Wong <normalperson@yhbt.net>)
* delete worker uses persistent HTTP connections (Eric Wong <normalperson@yhbt.net>)
* httpfile: use HTTP connection pool for DELETE (Eric Wong <normalperson@yhbt.net>)
* httpfile: use Net::HTTP::NB, remove LWP::UserAgent (Eric Wong <normalperson@yhbt.net>)
* fsck: parallelize size checks for any given FID (Eric Wong <normalperson@yhbt.net>)
* monitor: refactor/rewrite to use new async API (Eric Wong <normalperson@yhbt.net>)
2013-08-07: Release version 2.68
* optimize monitor worker for large installs (Eric Wong <normalperson@yhbt.net>)
* allow startup without job_master (and dependent workers) (Eric Wong <normalperson@yhbt.net>)
* store: do not disconnect for max_handles while locked (Eric Wong <normalperson@yhbt.net>)
* store: do not auto-reconnect while holding a lock (Eric Wong <normalperson@yhbt.net>)
* add naive MultipleDevice replication policy (Eric Wong <normalperson@yhbt.net>)
* httpfile: log mogstored I/O errors when checksumming (Eric Wong <normalperson@yhbt.net>)
* ProcManager: log socketpair errors correctly (Eric Wong <normalperson@yhbt.net>)
* fix "drain" handling used by MultipleHosts replpolicy (Eric Wong <normalperson@yhbt.net>)
* httpfile: correct timeouts for sidechannel digest (Eric Wong <normalperson@yhbt.net>)
* httpfile: correct FILE_MISSING check in digest_mgmt (Eric Wong <normalperson@yhbt.net>)
* mogstored: avoid bareword on IO::AIO w/o fadvise (Eric Wong <normalperson@yhbt.net>)
* ProcManager: only log times_out_of_qworkers for new queries (Eric Wong <normalperson@yhbt.net>)
* Don't emit warnings if we're lacking the space free of a device. If we don't find space on devices with known space free, try the unknowns. (Dave Lambley <davel@state51.co.uk>)
* list_keys: escape in Store, allow [%\\] as prefix (Eric Wong <normalperson@yhbt.net>)
* list_keys: consistent ESCAPE usage across DB types (Eric Wong <normalperson@yhbt.net>)
2013-02-02: Release version 2.67
* Serialize tempfile reaping (dormando <dormando@rydia.net>)
* reaper: ensure worker can be stopped via "!want" (Eric Wong <normalperson@yhbt.net>)
* domain removal also removes its default class (Eric Wong <normalperson@yhbt.net>)
* store: wrap create_class in a transaction to avoid races (Eric Wong <normalperson@yhbt.net>)
* mogstored: fix kqueue usage with daemonization (Eric Wong <normalperson@yhbt.net>)
* Filter the devices before we do an expensive sort. (Dave Lambley <davel@state51.co.uk>)
* httpfile: avoid killing worker on down sidechannel (Eric Wong <normalperson@yhbt.net>)
* move checksum and tempfile delete to delete worker (Eric Wong <normalperson@yhbt.net>)
* sqlite: use immediate transactions to prevent busy errors (Eric Wong <normalperson@yhbt.net>)
* disable Nagle's algorithm for accepted clients (Eric Wong <normalperson@yhbt.net>)
* ProcManager: favor using recently-used queryworkers (Eric Wong <normalperson@yhbt.net>)
* Do both sorts in one method, to save on shared initialisation. (Dave Lambley <davel@state51.co.uk>)
* Pull out device sorting into it's own method for overriding. (Dave Lambley <davel@state51.co.uk>)
* Reseed the random number generator after forking. (Dave Lambley <davel@state51.co.uk>)
* support nginx server type in mogstored command line options (Daniel Frett <daniel.frett@ccci.org>)
(also Gernot Vormayr <notti@gbcc.at>, others)
2013-01-06: Release version 2.66
* add a hook to cmd_updateclass (Daniel Frett <daniel.frett@ccci.org>)
* support updating the class to the default class which has an id of 0 (Daniel Frett <daniel.frett@ccci.org>)
* reaper: validate DB connection before reaping (Eric Wong <normalperson@yhbt.net>)
Fixes occasional crash in reaper process.
* improve handling of classes which change hash algorithm (Eric Wong <normalperson@yhbt.net>)
* fsck: skip non-existent FIDs properly (Eric Wong <normalperson@yhbt.net>)
* fsck: use replicate lock when fixing FID (Eric Wong <normalperson@yhbt.net>)
* query: allow "0" key on all commands which take keys (Eric Wong <normalperson@yhbt.net>)
* prevent reqid mismatches (and queryworker death) (Eric Wong <normalperson@yhbt.net>)
Fixes crash case with specific error types.
* fix use_dest_devs for rebalance (Pyry Hakulinen <pyry@automattic.com>)
Fixes "use_dest_devs" argument during rebalance.
* Fix "skip_devcount" during rebalance (Pyry Hakulinen <pyry@automattic.com>)
Now actually skips updating devcount column during rebalance.
2012-08-13: Release version 2.65
* Postgres advisory lock instead of table-based lock (Robin H. Johnson <robbat2@gentoo.org>)
Now requires minimum Postgres version of pg8.4.
* reaper: switch to Danga::Socket for scheduling (Eric Wong <normalperson@yhbt.net>)
* reaper: add queue_size_for_reaper server setting (Eric Wong <normalperson@yhbt.net>)
* reaper: add "queue_rate_for_reaper" server setting (Eric Wong <normalperson@yhbt.net>)
defaults to 1000, same as previously.
* reaper: global lock around DB interaction (Eric Wong <normalperson@yhbt.net>)
prevents reapers clobbering each other, causing a reduction in DB writes.
* tests: add basic test for reaper (Eric Wong <normalperson@yhbt.net>)
* fix tests when /etc/mogilefs/mogstored.conf exists (Eric Wong <normalperson@yhbt.net>)
* iostat: increase flexibility of iostat parser (Eric Wong <normalperson@yhbt.net>)
* iostat: allow MOG_IOSTAT_CMD env override (Eric Wong <normalperson@yhbt.net>)
* When a mogstored is down, die with a more informative message. (Dave Lambley <davel@state51.co.uk>)
2012-06-21: Release version 2.64
* Delete memcache data when we replicate fids (Pyry Hakulinen <pyry@automattic.com>)
* implement "case_sensitive_list_keys" server setting (Eric Wong <normalperson@yhbt.net>)
* get_paths: deprioritize devs in "drain" state (Eric Wong <normalperson@yhbt.net>)
* make marking a host down cause devices to act as down (Eric Wong <normalperson@yhbt.net>)
* monitor skips hosts marked dead or down (Eric Wong <normalperson@yhbt.net>)
2012-05-29: Release version 2.63
* Critical bugfix for Postgres users introduced by 2.61. New file uploads
would fail. (noticed by robin H. Johnson, fixed by Eric Wong)
2012-05-19: Release version 2.62
* Critical bugfix for a compilation error (dormando, reported by Robin)
* Fix for upgrading a Postgres install for checksums (Robin * H. Johnson <robbat2@gentoo.org>)
2012-05-18: Release version 2.61
* fix issue #57 by Pyry and Eric (dormando <dormando@rydia.net>)
(mogadm host status sometimes allowed typos)
* avoid unnecessary devcount update in create_close (Eric Wong <normalperson@yhbt.net>)
* size arg check from tomas doran (dormando <dormando@rydia.net>)
(pass in a "size" argument (in bytes) to create_open and mogilefs will
ensure there's enough free space for it)
* sqlite: implement locking via tables (Eric Wong <normalperson@yhbt.net>)
* worker/query: Add optional callid parameter (Gernot Vormayr <notti@bgcc.at>)
(allows command pipelining)
* delete: prevent orphan files from replicator race (Eric Wong <normalperson@yhbt.net>)
* fsck: prevent running over 100% completion (Eric Wong <normalperson@yhbt.net>)
* fsck: cleanup and reduce unnecessary devcount updates (Eric Wong <normalperson@yhbt.net>)
* fsck: update devcount, forget devs on unfixable FIDs (Eric Wong <normalperson@yhbt.net>)
* fsck: log bad count correctly instead of policy violation (Eric Wong <normalperson@yhbt.net>)
* tests: add tests for fsck functionality (Eric Wong <normalperson@yhbt.net>)
* monitor: only broadcast reject_bad_md5 on change (Eric Wong <normalperson@yhbt.net>)
* worker: delete_domain returns has_classes error (Eric Wong <normalperson@yhbt.net>)
* log: enable autoflush for stdout logging (Eric Wong <normalperson@yhbt.net>)
2012-03-30: Release version 2.60
* Fix fsck status when running for the first time (dormando <dormando@rydia.net>)
* Checksum support (Major update!) (Eric Wong <normalperson@yhbt.net>)
See doc/checksums.txt for an overview of how the new checksum system
works. Also keep an eye on the wiki (http://www.mogilefs.org) for more
complete documentation in the coming weeks.
2012-02-29: Release version 2.59
* don't make SQLite error out on lock calls (dormando <dormando@rydia.net>)
* Postgres wasn't honoring a no-wait timeout (dormando <dormando@rydia.net>)
2012-02-28: Release version 2.58
* reduce UPDATE's to device table (dormando <dormando@rydia.net>)
* replicate: retry on short writes to destination (Eric Wong <normalperson@yhbt.net>)
* enable TCP keepalives for accepted sockets (Eric Wong <normalperson@yhbt.net>)
* Populate memcache even when noverify is missing (Pyry Hakulinen <pyry@automattic.com>)
* Remove FID from memcache when it is deleted (Pyry Hakulinen <pyry@automattic.com>)
* Unset memc when memcache servers have been removed (Pyry Hakulinen <pyry@automattic.com>)
* Filter out illegal memcache servers from memcache_servers (Pyry Hakulinen <pyry@automattic.com>)
* Add skip_mkcol setting for servers which do not need it (Pyry Hakulinen <pyry@automattic.com>)
* skip_devcount variable that disables devcount updates (Pyry Hakulinen <pyry@automattic.com>)
* Back off if master DB is down (dormando <dormando@rydia.net>)
* use cached setting for slave list (dormando <dormando@rydia.net>)
* fix "fsck never stops" issue (dormando <dormando@rydia.net>)
2012-01-31: Release version 2.57
* Silence a warning about undefined value introduced by the slave database ordering work (hachi@kuiki.net)
* Need a hook to be able to check the slave list during startup (hachi@kuiki.net)
2012-01-27: Release version 2.56
* Add database connect timeouts (dormando <dormando@rydia.net>)
* fix potential hang in JobMaster added in 2.55 (dormando <dormando@rydia.net>)
* fix "set_weight" command (Eric Wong <normalperson@yhbt.net>)
* sqlite: consider SQLITE_BUSY and SQLITE_LOCKED deadlocks (Eric Wong <normalperson@yhbt.net>)
* kill a little more dead fsck/Checker code (dormando <dormando@rydia.net>)
* Destroy mogdeps (dormando <dormando@rydia.net>)
* improve slave handling code (dormando <dormando@rydia.net>)
* don't spew errors if the master goes down (dormando <dormando@rydia.net>)
* provide an async server setting cache via monitor worker (dormando <dormando@rydia.net>)
* fix return value of commands that clear_cache (Eric Wong <normalperson@yhbt.net>)
* make reaper wait less time in test mode (dormando <dormando@rydia.net>)
* queryworker: reimplement "clear_caches" command (Eric Wong <normalperson@yhbt.net>)
* worker: use timeout with read_from_parent() instead of sleep (Eric Wong <normalperson@yhbt.net>)
* Fixed issue where too many devices being active could cause the monitor job
to be timed out by the watchdog. (Tomas Doran)
2011-11-14: Release version 2.55
* fixed sources in replication are now a suggestion (dormando <dormando@rydia.net>)
* More stats to help troubleshoot rebalance (dormando <dormando@rydia.net>)
* return 1 from condthrow if all is well (dormando <dormando@rydia.net>)
* schedule fsck replications for the future. (dormando <dormando@rydia.net>)
* stop enforcing a source id on new file upload (dormando <dormando@rydia.net>)
* restore new file upload replication priority (dormando <dormando@rydia.net>)
* Prevent queue related MySQL query hangs (dormando <dormando@rydia.net>)
2011-10-29: Release version 2.54
* Issue 36: Further insulate from mysql restarts resetting the fid counter (dormando <dormando@rydia.net>)
* Allow default class to be overridden again. (dormando <dormando@rydia.net>)
* Fix crash in old delete worker code (Daniel Frett <daniel.frett@ccci.org>)
* Fix mb_free() counting empty device as missing (git.user <git.user@gmail.com>)
* sqlite: fix mogdbsetup failure on new setups (Eric Wong <normalperson@yhbt.net>)
* fid size workaround for lighttpd 1.4.x (dormando <dormando@rydia.net>)
* Fix various issues in the new mogilefs server query worker. (Tomas Doran <bobtfish@bobtfish.net>)
* Allow chunked PUT's of unlimited size (dormando <dormando@rydia.net>)
* Fix listing FIDs.
* Fix fsck fixing a file from logging a message which appears to be
two commands (due to spurious \n). Also fix debug function to
not be able to log messages like this.
2011-08-12: Release version 2.53
* Fix host status not updating for trackers (important)
2011-07-30: Release version 2.52
* Plugins (such as FilePaths) might want to be run in info/debug commands (Michael Baehr <mbaehr@plos.org>)
* schema version 14; support devices > 16TB (dormando <dormando@rydia.net>)
* Make FSCK worker handle dead devices (dormando <dormando@rydia.net>)
* Throw out old HTTPFile->size code and use LWP (dormando <dormando@rydia.net>)
* Remove forced loading of DBD::mysql (dormando <dormando@rydia.net>)
* Don't test write on readonly devices (Akzhan Abdulin <akzhan.abdulin@gmail.com>)
2011-06-27: Release version 2.51
* Add minor backcompat for ZoneLocal and friends (dormando <dormando@rydia.net>)
* Only monitor a device if we should monitor it. (dormando <dormando@rydia.net>)
2011-06-15: Release version 2.50
* Completely rewire meta object caches. Device/Host/Class/Domain caches are
now asynchronously loaded via the Monitor worker. This should drastically
reduce the number of small queries issued against the master database.
(dormando <dormando@rydia.net>)
* Don't run workers until Monitor runs once (dormando <dormando@rydia.net>)
* s/TYPE/ENGINE/ for innodb initialization (dormando <dormando@rydia.net>)
* Create mogile database explicitly with UTF-8 encoding. (Robin H. Johnson <robbat2@gentoo.org>)
* Actually use the MOGTEST_* variables for Postgres testing... (Robin H. Johnson <robbat2@gentoo.org>)
* Add support for config file includes for Mogstored under lighttpd (Jason Mills <jmmills@cpan.org>)
* Make postgres use new delete worker code (Robin H. Johnson <robbat2@gentoo.org>)
* New test env variables: MOGTEST_DBROOT{USER,PASS} for easier test usage. (Robin H. Johnson <robbat2@gentoo.org>)
* Document test variables. (Robin H. Johnson <robbat2@gentoo.org>)
* Support inclusion of custom lighttpd config. (Jason Mills <jmmills@cpan.org>)
2011-01-13: Release version 2.46
* Fix job_master crash under postgres (Robin H. Johnson <robbat2@gentoo.org>)
* Revert "only dbh->ping if not used in over a minute" (dormando <dormando@rydia.net>)
*IMPORTANT FIX* if you're using 2.45, you should upgrade to this.
* Example application. (dormando <dormando@rydia.net>)
* missing $fidid for source_down error message (dormando <dormando@rydia.net>)
2011-01-08: Release version 2.45
* file_debug command (dormando <dormando@rydia.net>)
* add "file_info" command for fetching metadata. (dormando <dormando@rydia.net>)
* don't allow clients to upload to the wrong dev (dormando <dormando@rydia.net>)
* validate database handles in the right places (dormando <dormando@rydia.net>)
* only dbh->ping if not used in over a minute (dormando <dormando@rydia.net>)
* make list_fids work with gaps (dormando <dormando@rydia.net>)
* Postgres enqueue_fids_* cleanup (Robin H. Johnson <robbat2@gentoo.org>)
* Fixes MogileFS::Store::retry_on_deadlock to honor non-deadlock exceptions (Jason Mills <jmmills@cpan.org>)
* Implements tests for MogileFS::Store::retry_on_deadlock (Jason Mills <jmmills@cpan.org>)
* Implements tests for MogileFS::Store::ignore_replace (Jason Mills <jmmills@cpan.org>)
* job_master was never revalidating its dbh handle. (dormando <dormando@rydia.net>)
* add a couple useful stats (dormando <dormando@rydia.net>)
* Correct iostat command on darwin (Tomas Doran <bobtfish@bobtfish.net>)
* Fix test db creation with SQLite (Tomas Doran <bobtfish@bobtfish.net>)
* Fix SQLite so tracker at least starts (Tomas Doran <bobtfish@bobtfish.net>)
* additional debian packaging fixes (kad <kad@blegh.net>)
* debian packaging updates. (Andre Bohr <andrepascha@googlemail.com>)
* Documentation updates. (Pyry Hakulinen <pyh@iki.fi>)
* HTTPFile returns -1 if the file is missing from the device. Content-Length: 0 is valid and it should return BLEN not MISS. (Pyry Hakulinen <pyh@iki.fi>)
* some further editing of the info file. (dormando <dormando@rydia.net>)
* Changed repository metadata (Tomas Doran <bobtfish@bobtfish.net>)
2010-11-25: Release version 2.44
* Fix issue with rebalance not filtering source devs from the
destination list.
* Make global rebalance limits work (Martijn Lina)
2010-10-10: Release version 2.43
* Make over-replication fixable again. (sigh)
* Fix old bug in MultipleHosts() which would not let you drop from
mindev > 1 down to 1.
2010-10-08: Release version 2.42
* Make FSCK run again (Andre Pascha)
* Fix Postgres schema upgrade issue (hopefully)
2010-10-06: Release version 2.41
* One line change that to fix uprading to 2.40
2010-09-28: Release version 2.40
* Optimize worker sleep patterns and worker/parent communications.
* New rebalance/drain code. See the docs/wiki for how to use.
***NOTE*** Old drain/rebalance code is disabled. Setting a device
into drain mode no longer does what you think it would.
http://code.google.com/p/mogilefs/wiki/Rebalance
2010-08-13: Release version 2.37
* Make HTTP DELETE's work reliably
* Fix obscure crashes related to "unknown devices"
* make it slightly harder to crash trackers with bad slave DB settings
(Tomas Doran)
* make tracker -> storage node connection timeout configurable.
was hardcoded to 2 seconds. (Tomas Doran)
* update_class command (Tomas Doran)
2010-04-16: Release version 2.36
* Bump required version of perlbal for mogstored.
* Critical bug fix for creating files.
If you have many dead devices, poor filter ordering could cause
MogileFS to not consider any alive devices for new files.
* Actually default to BIGINT under MySQL...
Bug would always initialize new setups to INT.
2010-04-02: Release version 2.35
* Nuke ancient mogilefs 1.1 replication code.
no_unreachable_tracking and old_repl_compat variables are gone.
(dormando)
* Support modifying class replpolicy strings (requires new
mogadm/MogileFS::Client) (dormando)
* Print fidid and devid on errors when fsck fails to fix something
(t0m)
* Allow list_keys to work with a prefix of '0' (dormando)
* Print timestamp when logging to STDOUT (t0m)
* Sort by free space and weight fewer devices to avoid CPU churn
(dormando)
* Fix deadlock-related crash bugs in db queue fetch routines.
* Fix dependencies in MogileFS::Server (t0m)
* Fix bug in list_keys command that prevented searches with a
'0' prefix. (Ask Bjørn Hansen)
2009-12-05: Release version 2.34
* Fix bug in new copycount reduction code. If it was more than off by
one it wouldn't reduce the count properly.
* Improve accept loop. A reasonably busy tracker could start dropping
new client connections too easily.
2009-11-24: Release version 2.33
* 'mogadm fsck status' will now be more accurate.
* Support non-GNU df (paul armstrong)
* Remove old !replication command. It's in `mogadm stats` and tends
to kill the parent process.
* Force refreshing internal monitoring status every 5 minutes. Help
work around a potential bug where state changes are missed.
* Use new delete queue for tempfile culling
* Calculate free space from available space, not total space
(todd lipcon)
* FSCK code has new tunables to control how aggressive it runs:
queue_size_for_fsck (how many fids to keep in file_to_queue table)
queue_rate_for_fsck (how many fids to put into the table per second)
the generic internal queue is now tunable:
internal_queue_limit (how many fids to pull out of a queue table at
once. old default was 500).
All are tunable by 'mogadm settings set'
(dormando)
* Replication will now attempt to handle over replication situations.
FSCK will catch over replication as a policy violation, and
replicate will attempt to fix it. (dormando)
(rarely it might need multiple fscks)
* Since FIDSizes is no longer used, remove it and remove gearman
dependency. (dormando)
* FSCK no longer rabidly fills the queue table. (dormando)
* Make storage code resilient to rare deadlocks (dormando)
* Add 'max_handles' config option to restart a DB connection if there
are too many handles to it (victori).
2009-10-18: Release version 2.32
* No changes... MFS is hard to release due to mogdeps :/
2009-10-18: Release version 2.31
* Fix create_close race condition. (pipegeek)
* Fix bug in reporting of available disk space.
* Many minor bug/documentation fixes.
* MySQL and Postgres installs now use BIGINT's for fid/length by
default. (dormando, robbat2)
* Accept 405 responses to MKCOL from the WebDAV server since it often
just means that the directory already exists. (Ask Bjørn Hansen)
* Fix showing domain and class in mogadm output when there's only the
default class in the domain (Mark Imbriaco)
* mogstored: pidfile support
* New FSCK work and error for BCNT for bad devcount values (they would
not otherwise be caught unless the triggered another fix).
2009-01-12: Release version 2.30
* fsck will now fix broken 'devcount' cols.
* schema version bump to 12, and new delete worker code to use the new
queue system. delete should be faster and more scalable.
* add server setting queue_rate_for_fsck to allow rate limiting how
fast the new distributed fsck creates jobs. the original default of
10k could utterly defeat wimpy databases.
* Support 0 byte files.
* Fix a bug where rebalance would accidentally delete copies of a fid.
* Create new internal queue management system, with new job
'JobMaster'. This greatly reduces database load and increases
throughput for workers using the new system.
* Add 'no_unreachable_tracking' config. Disables more of the old
replication code. Use this if you have no 1.x trackers.
* Add hook for alternate replication sources. Obscure, but can be
nice.
* Add replicate_order_final_choices hook, so multidc replication may
pull from the local network if one copy already exists.
* Fix subtle bugs in process management.
* Allow overriding of default classes.
2008-08-18: Release version 2.20
* One more time, and hopefully never again. No changes. META.yml
fixes.
2008-08-18: Release version 2.19
* No changes except for META.yml fixes.
_should_ be able to upload releases without owning half of CPAN.
2008-08-16: Release version 2.18
* Add repl_use_get_port option (Brian Lynch)
* Remove tempfile data and rows on create_close (Chaos Wang)
* Don't delete files accidentally during tempfile table cleanup
* Specify username to run the mogstored daemon (Jonathan Share)
* Many spelling errors corrected.
* PostGres locking routines did not include the hostname for the lock
release.
* RFC 2518 says we "should" use a trailing slash when calling
MKCOL. Some servers (nginx) appears to require it. (Spotted by
Timu Eren).
* Add server option 'rebalance_ignore_missing' in mogilefsd config
that allows the rebalance to ignore files that are already missing
when it tries to delete them.
* Fix MultipleHosts replication policy to no longer return desperate
devices after no longer being desperate, instead returning a state
which will queue for retries in the future till it can be truly
happy. This prevents files from being replicated to all writable
devs when there are two few writable hosts to satisfy the policy.
* Add --chmod-mountpoints option to mogautomount, in an attempt to
prevent mogstored machines from accidentally writing to their under-
lying filesystem.
* New optional argument 'startpos' to fsck_reset.
* Use correct per-Store port, defaulting to UNIX sockets on localhost.
* add --dbport option to mogdbsetup (Ask Bjoern Hansen)
* Fix state_change() bug that could make file_to_delete_later
grow unnecessarily (Radu Greab)
* Add Mogstored::HTTPServer::None to allow using mogstored
with unmanaged backend DAV servers.
* Add experimental cmd_edit_file. Client requires DAV MOVE and
partial PUT on the store to use it.
* make mogstored and mogilefsd use the now-bundled dependency
Perl modules as a last resort, unless MOGILE_NO_BUILTIN_DEPS
environment variable is true. this makes it easier for people
to use mogilefs, without dependency hell. distros/packagers
are expected to ignore (or scoff at) this "feature". but
easier for people working from tarball/svn releases. bundled
dependencies will be updated over time, as needed.
* Bring the SQLite store up to date
* Increase the timeouts in the test suite
* In the test suite, also search for mogadm in /usr/local/bin/ and
/usr/local/sbin (spotted by Dean Wilson)
* SCHEMA VERSION 10: 'replpolicy' column on 'class' table.
it's safe to --ignore-schema-version and run this mogilefsd
against an older schema. new column enables per-class
pluggable replication policies. it was always
abstract, but not easy to plugin your own alternatives.
see doc/pluggable-replication-policies.txt
* add start of a MogileFS::ReplicationPolicy::Union implementation
* fix crash in queryworker's create_open command, when a device
has its directory made on a mogile storage node (the host of
which is being monitored by the 'monitor' process), but the
device isn't registered in the devices table yet.
2007-07-03: Release version 2.17
* get_paths command now takes a pathcount option, defaulting to
2, instead of having a hardcoded constant of 2. This allows
requesting more than 2 paths to a file.
* make fsck log over-replicated files as policy violations. it
won't delete over-replicated files, though, just log
* Document how to test MogileFS under doc/testing.txt. You really
should read this if you want to use MogileFS to store your data.
* Add a safety check to ensure that Postgresql-8.2 or newer is used.
* Update mogdbsetup usage, and ensure it chooses the correct admin
username automatically.
* Document the --type option to mogdbsetup.
* Introduce the PostGres Store implementation. The locking system might
still have issues, beware of stale locks! Initial prototype by Radu
Greab <rg@yx.ro>, and developed heavily by Robin H. Johnson
<robbat2@gentoo.org>.
* start of support to make mogstored wrap Apache + web_dav
* Remove duplicated [fsck] in log output of Fsck worker.
* Handle HiRes gettimeofday correctly to get useful results. Arrays are
important. This gives us correct timing output.
* Handle input to log() correctly. Based on the implementation of
Sys::Syslog::syslog(), produces no ugly warnings, and makes sure that
printf stuff works nicely.
* The unix_timestamp function belongs to the Store, not other classes.
Need to add test cases for process_deletes and stats.
* Using NULL during an INSERT for the AUTO_INCREMENT column does not
work very well outside of MySQL. Pass the correct number of arguments
to the INSERT instead.
* Clean up all raw usages of UNIX_TIMESTAMP() function, in preparation
for database independence. If you need the unix timestamp, include the output
of $self->unix_timestamp directly into your query.
* Alter $0 so that the scripts show up as '$SCRIPTNAME' in the process
listing instead of '/usr/bin/perl $SCRIPTNAME'. This makes init.d
scripts easier to manage.
* make mogstored's lighttpd config use mod_status for "/", so
mogadm doesn't think it's dead when it previously saw a 404
(Justin Huff <jjhuff@mspin.net>)
* remove Linux::AIO support, now that Perlbal has removed it as well,
and because Linux::AIO support (when Perlbal then chooses
IO::AIO) was hanging mogstored on shutdown. so also add a
mogstored shutdown test (after a full daemonization of it)
* make Mgd::validate_dbh() re-check/ping slave dbhs as well.
(problem mentioned by Steven Shou <steven.shou@verizon.net>)
* support drive identifiers above sdz (Arjan <info@bouncetek.com>)
* make mogstored's iostat child process properly shutdown,
even when iostat binary isn't installed. (previously
it was stuck in a loop every 10 seconds, hoping for
iostat to be installed, but never respecting parent
process' death)
* Change delete worker to use new DeviceStates interface.
* Fix get_paths to honor new device state of 'drain' via new
DeviceStates interface.
* use significantly less CPU when re-broadcasting around
iostat device utilization% to child processes. (future:
don't rebroadcast at all and use shared memory)
* slightly faster fsck (do less syscalls)
* efficiency: don't broadcast dup state information to children
(track in parent what children know already)
* if using Gearman::Server >= 1.09, respawn the fidsizes
worker if it dies.
2007-05-09: Release version 2.16
* BUG FIX from 2.15: partially written but never closed
tempfiles weren't being cleaned up properly.
* start to work better when compiled as a stand-alone
PAR all-in-one script/archive
2007-05-07: Release version 2.15 ("fsck/drain/rebalance")
* minor bug fixes and enhancements for MultipleHosts replication
policy. bunch of corner cases now checked with new tests.
now easy to write more replication policy tests in future.
* replication policies can now return "desperate" requests,
signalling that a replication reassessment should be
enqueued for the future, to see if things could be improved.
(that part's not currently implemented, but the real feature
and motivation is that the rebalancer now won't delete a
DevFID if it results in a desperate move, only an ideal move. )
* replication policies now can optionally return a new return
value of the (new) type MogileFS::ReplicationRequest, which
has pretty accessor names, can suggest multiple places,
can indicate non-ideal emergency replication decisions.
old plugins' return values will be transparently upgraded
to the equivalent new return value objects.
* adding new device state: "drain". it's a hybrid of "dead"
(in that files are migrated off it) and "readonly", it that
it still serves traffic... it just doesn't get new files.
this also introduces the new object-oriented DeviceState class,
and device_state($name) utility function to get the DeviceState
singleton by name
* internal code cleanup. notably, kill the old & nasty legacy
'find_deviceid' function which was ridiculously long and hairy.
the two callers are now more readable with sorts/greps/etc.
* make mogstored's devN/usage writing process (DiskUsage) be less
racy with the mogilefsd monitoring code... don't open file for
write... open read/write, then in one write system call, write
the entire file, with newline padding at end to cover old data,
then truncate it if necessary. should remove harmless (but scary)
error messages previously reported by the mogilefsd monitor
about zero-length usage files.
* new protocol commands to list/set (certain) server settings,
with value sanity checking (see MogileFS::Config for which
are settable, and with what values). needed for "enable_rebalance".
was partially enabled before for slave settings. also needed
for memcached support before, which was never possible to
set with mogadm, only with db tweaking.
* make mogilefsd fsck use new mogstored fid_sizes command, to
do bulk stats. speeds up fscks a ton.
* be robust against system clocks that go backwards between
gettimeofday calls:
http://lists.danga.com/pipermail/mogilefs/2007-April/000897.html
* Put gearman server in mogstored process, add worker
'mogstored-fidsizes' which runs as subprocess of mogstored. Add
side-channel command 'fid_sizes' which allows us to quickly enumerate
and get sizes for files across entire devices on a storage node.
* remove all code like $state eq "readonly", $state=~ /^dead|down$/
and instead convert it into specific questions on policy/traits
of given state, like $dev->should_put_new_files_on, or
$dev->should_drain_files_off. see MogileFS::DeviceState,
objects of which are accessed via $dev->dstate, or new
MogileFS::Util device_state($name) wrapper.
* start of rebalance support. (where replication workers, in their
idle time, can rearrange files to even out disk space and/or IO
activity on storage nodes... policy isn't hard-coded, and is
in fact currently random)
* lighttpd support in both mogilefsd and mogstored. passes test
suite with environment MOGSTORED_SERVER_TYPE=lighttpd set now.
* abstract out the HTTP server support in mogstored, so
mogstored isn't just a perlbal wrapper, but an anything
wrapper. (in particular, lighttpd and apache) mogstored still
exists for all its other misc admin/monitoring functions,
but can then manage/configure apache/lighttpd child process(es).
so far they're just stubbed out.
* split mogstored into separate files per class, rather than one
large script.
2007-04-24: Release version 2.10
* bug from earlier refactoring: changing some 'next' to 'return'
in MogileFS::Device's create_directory method. (thanks to
Justin Huff <jjhuff@mspin.net>)
* make sure mogdbsetup doesn't upgrade an existing table (with
potentially lot of data) from MyISAM to InnoDB. just warn.
only new tables should become InnoDB on creation. also,
detect InnoDB even on old MySQL 4.0.x, not just 4.1/5.0.
* make fsck faster by reducing number of db queries (both
reads and writes), though more unrelated fastness remains to
be done.
* make fsck_status faster (keep summary stats of evcode counts,
rather than doing a full GROUP BY query all the time)
2007-04-20: Release version 2.09
* filesystem checker (Fsck worker). off by default, unless
it's enabled via protocol. Control/monitor it with mogadm
in MogileFS-Utils 2.08, which needs MogileFS::Client 1.06.
* replication speed-up on configurations with only 1 host.
(replication would randomly have no suggestions before, when
there was only 1 target host, so replication would be rescheduled
until the future, when random returned a different device on
that host)
2007-04-17: Release version 2.08
* (no runtime-visible changes from 2.07)
* mogilefsd: remove incomplete "Checker" worker, replace with
incomplete "Fsck" worker.
* test cleanup/robustness improvements. properly skips testing when
testing deps not found. but searches harder for them now, too.
tests run under "make disttest".
* packaging cleanups, released using ShipIt.
2007-03/04:
* mogilefsd: Split Plugin::MetaData and Plugin::FilePaths into their own distributions.
* mogilefsd: Fix mogdbsetup to not ask empty questions.
* mogilefsd: command line/config file support for listening on specific IPs, also
support for listening on multiple IPs/ports
2007-03-19:
* mogstored: use XS HTTPHeaders by default, unless disabled with environment variable
* mogilefsd: memcache support for get_paths (see doc/memcache-support.txt)
2007-03-16:
* mogstored: make child processes be exec'ed (as as their own scripts, not inheriting
Perlbal, Linux::AIO/IO::AIO, etc baggage). so they don't own listening
sockets, pipes to child workers, signal handlers, etc. just a lot
cleaner now, and SIGTERM, SIGINT properly shut everything down now,
both in foreground and as daemon
2007-03-15:
* mogstored: this changelog is totally neglected. :/ see svn log.
* mogstored: make perlbal's "shutdown graceful" work, by not doing
sidechannel accepting ourselves (which never hung up),
but subclass Perlbal's TCPListener class, which does it
all correctly.
2006-12/2007-01: (2.00_05)
* mogilefsd: everything converted to OO API internally. old internal APIs removed.