forked from flueke/mvme
-
Notifications
You must be signed in to change notification settings - Fork 0
/
TODO
1319 lines (1045 loc) · 60.5 KB
/
TODO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
==================================================
TODO list
==================================================
- figure out the vme -> analysis module index mapping stuff. why is it needed?
the analysis does see the vmeconfig in beginRun() so it could/should create the
index mappings itself. Don't forget IStreamModuleConsumers, they also need the
correct indices.
- when creating an new event config from file let the existing logic figure out
variable values for sys_irq and and mesy_mcst. This way the existing setup won't
be broken as easily. More generally do not import system and mesytec variables,
only user defined ones.
- plot widgets:
* move index selector to the very left of the toolbar
* replace resolution slider with a drop down box like in the plot grid
- When generating histograms for an output pipe the source operator could be
examined to determine the resolution for the histogram.
- Analysis errors and highlighting
* Histo sinks without an allocated histo should have an error highlight
(something is wrong with the inputs).
* Operators having a zero output size should be highlighted too. Operator should
have at least one non-zero sized output.
* Expr Cond (maybe a general issue): setting the "Parent Event" to unspecified
does not mark the operator as having an issue (need two VME events to reproduce
as otherwise the only existing event is used as the parent).
* ExprCond: having an expression compile error is not visible in the analysis ui
(operator is not highlighted in red)
* doc: describe listfile formats and ZMQ transport
* Multi plot window
- Support RateMonitors
serving as a plot gallery
- react to sink modifications after beginRun() was called: have to update the
histo list as otherwise we might display state histograms kept alive by our copy
* move module templates to JSON format
* add GSI VFTX module template (JSON format if possible, otherwise the old template format)
* Load module templates from JSON, then convert existing module templates to
JSON format.
* root_client: avoid creating duplicate branch names for trees
* Maybe: remove the VME controller auto (re)connect feature. instead try to connect when needed.
* Packet loss and error counters for the MVLC command side.
* Document ListFilterExtractor
* Update the EventServer and Clients to handle multi-output data sources (MultiHit Extractor).
* Update the DAQControl and listfile GUI to better support non-file listfiles,
e.g. ZMQ. Its currently only designed to handle file based listfiles.
* VME Script execution in the timeout/error case can hang for a long while.
Instead the execution should be aborted earlier, upong encountering the first
error.
* DAQ init sequence: on error try to link back or highlight the VME script that
caused the error.
* Configurable colors for the Trigger/IO connection arrows (Nik).
* FIXME: crateIndex handling is unclear. Should it be specified in
make_readout_parser()? This works for the current readout code as it's
purely handled on the software side and doesn't care about the CtrlId
set for the MVLC.
On the other hand the listfile_generation code and the merged multicrate
listfiles do need the crateIndex present in the data stream. It's already
used in mvlc_stream_worker.cc to check for UnixTimetick events coming from
the main crate. Also the same code ignores the crateIndex completely! Weird
state right now!
* when stopping a run do wait for the analysis to finish. right now the code
only waits for the readout/replay side to finish.
* add an assert macro to get rid of the unused var warnings when compiling in
release mode
#ifdef NDEBUG
#define ASSERT(x) do { (void)sizeof(x);} while (0)
#else
#include <assert.h>
#define ASSERT(x) assert(x)
#endif
* Add undo/redo buttons to the vme_script editor.
* Errors from the mvlc analysis callback chain could be reported in a better way:
[2021-10-19 15:04:38.684] [readout_parser] [warning] end parsing ETH buffer 1, size=1048320 bytes, exception=vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
[2021-10-19 15:04:38.689] [readout_parser] [warning] exception from parse_eth_packet(): vector::_M_range_check: __n (which is 0) >= this->size() (which is 0), skipping packet
=> No idea where the occured. Horrible to debug. std::function does not make it easier.
Wanpeng
-------
In addition to the reversion issue of the VME settings, I'd like to comment more on the MVME software (v1.4.6), in particular for some desired online features:
* Rate monitor is very useful. But right now it either displays the full range or can be zoomed in for a fixed range. It'll be more useful if one can make it visible for the last hour or some designated period of time while keep incrementing.
* Right now, 1-d histograms can only be displayed one by one. Is it possible to overlap two or three plots in the same window for comparison? That would be a great feature to have.
* A dynamic count rate ratio would be good for online monitoring. Right now, I am using three steps to construct it (prev, diff, and division). It'd be great if you can implement one operation like (rate1 - rate1_previous_value) / (rate2 - rate2_previous_value) for a simple definition.
* a standalone config editor.
* analysis operator event affinity:
See file 1_mdpp16.analysis:
- amplitudes are extracted but the histo is never filled.
- editing the histo shows that the event context is set to 'unspecified'
- changing it to the correct event immediately fixes the problem.
How to fix? Auto assign to the parent module? Also before being able to
explicitly specify the event affinity there must have been some way this was
stored or determined.
* wrong state after creating a new workspace:
- in the old workspace open a listfile
- select File -> New Workspace
- create a directory and accept via 'Ok'
=> The start button still shows "Start Replay", the listfile from the old workspace
is still open.
* Workspace switching under windows: sometimes mvme.log cannot be moved to
last_mvme.log. mvme still has an open handle to the log file at this point
(renaming using the explorer also does not work).
* Do not allow (or warn about) relative writes in the Readout Loop -> Cycle Start/End scripts.
* The run script button of the VMEScriptEditor inside the Trigger IO GUI does
not work. The runScript() signal is not connected to anything.
* Crash when opening workspace directory which does not contain a mvmeworkspace.ini
* Test opening the same workspace directory that's currently open again.
mvme.log and last_mvme.log seem to cause trouble in that case.
* Rate Monitor:
- mouse hover sample index is often negative
- stats display needs updating for Sample based x-scale
- avg. Rate in the stats display is incorrectly 0 most of the times. why?
- it's so messy :(
* DSO gui:
- histogramming: could switch between osciolloscope and histogram view.
draw rising and falling edge times in different colors.
- When modifying the trigIO the trace select tree gets collapsed :(
- Sim: 36ns difference between simulated and sampled routed trace. why? subtract it?
* trigger_io: when changing a vme event from periodic to irq the trigger_io
timer -> stack_start connection is not disabled. Could find out which timer
was used to trigger the event and remove the connection from the timer to the
stackstart unit.
* When a ListfitlerExtractor changes size an open RateMonitorWidget is not
properly updated. The legend will be updated but the number of elements is
not updated. Something is not correct here.
* Difference operator asserts when arrays have different sizes and when one
array has size 1 (different assertion).
* add checks for mesytec modules analysing the eventcounter/timestamp values
This should happen after multi event processing and should be fast. So just
picking the last word of each events module data should be enough.
Handle overflow properly!
* analysis cross-process copy/paste is broken
* make named indices inline renamable (maybe, these are weird anyways)
* doc: Add network tuning and data export section
* document MVLC jumbo frame support in mvme manual
* finally write mesytec-mvlc library docs
* [analysis]
* Module data Prefix/Suffix is missing from:
- a2 system
- analysis ui
- EventServer
- Rename (internally these are called extractors):
* MultiwordFilter -> Word Pattern Filter
* Listfilter -> Word Order Filter
- Add new extractor or make the multiword one even more complicated:
Start filter with optional size bits <- yields the number of words the extraction filter should be applied to
Extraction filter <- extract (address, value) pairs
Optional End filter; required if no size bits are given. <- Stop processing once this matches.
- Conditions: core should be functional, work on the UIs again
- listfilter add is buggy: the last filter is selected and will be edited if
not manually pressing the 'add' button
* vme modules: allow saving a concrete module to a vme template.
Allow instantiating modules from user vme templates. Important: handle the
analysis correctly: if module type name stays the same the current code will
load the correct analysis default filters.
* analysis: allow more than just "default filters". the new system should be
able to handle loading arbitrary analysis files as default filters. Matching
is done via the module type names.
* non-existent workspace directories mentioned in the mvme.ini file are
silently created. This is annoying when renaming a workspace and starting mvme again.
Instead the dialog to select a workspace directory should pop up.
* analysis: inline rename of 2d histos does not work because of text updates to
the node. These updates should probably be suppressed while inline editing is
active.
* analysis: disabling of operators would be cool
* Filter: mdpp16 samples: kanalwort, dann werte
-> so eine art subheader
-> subheader suchen, addresse extrahieren und merken
danach weitere daten mit weniger adressbits oder positionsbasierter offset zur "base address" und
natuerlich datenbits extrahieren
wenn wieder ein subheader kommt (via match erkannt) dann base address anpassen und damit weitermachen
* Mehr siehe Zettel!
* Things missing to support other vendors modules:
- CAEN v1190: Need to check a register bit to know when the chip controlling the CERN
TDCs is ready. This could be done in software or on the MVLC via a special command.
* multi_event_splitter: count unused words (per event and module) if possible.
make this data available as a counter.
* add functionality for checking if the scripts in the vme tree can be parsed
successfully. Right now parse errors are only detected on DAQ startup.
* write unit test for CrateConfig import/export
* log the DAQ run duration when the DAQ is stopped
* Analysis processing skip flag:
Decide directly after the data filters if this events output should be
further processed and passed to the EventServer or if it should be
skipped/filtered out.
The decision should be based on the current(?) values and previous data.
* MDPP-16_SCP "variable rate" setup:
threshold 0
gain >= 2000
WindowOfInterest -100ns
-> pulser data should appear to be uniformely or randomly distributed
* apparently there's a crash when compiling an expression operator script while
a replay is running. Robert ran into this while editing an expression and
lost all of his edits... Cannot reproduce
* event_server: test with the mvlc. measure performance
* mvme_root_client: runId is incorrect for live DAQ runs. it seems correct for replays.
* test m2r. get rid of duplicate files. clean up the env file. write some docs
* improve mesytec diagnostics
* qthelp: most images are links but they do show the binary file contents
instead of the image. Note that not all images behave this way. Some are not
clickable, e.g the very first 'mvme overview' image in the introduction.
* heisencrash when deleting vme scripts? might have been due to changing
compile flags. not sure.
* MVLC release work
--------------------------------------------------
- use mvlc trigger io docs for the standalone mvlc doc
- lib design: (+ are things we want and that are mostly clear, ? are unclear things)
+ direct mvlc com, direct vme com, stack upload, stack exec, irq setup,
+ readout_worker, queued communication, lossless or lossfull queueing strategies
+ blocking consumer channel (listfile) and non-blocking consumer channel (queue)
+ stack -> readout parsing, readout reassembly
+ stats, logging, python wrapper
? trigger_io setup, automatic periodic stack setup
- multicreate: merging of the data stream. different strategies, one true way?
analysis_ui: no small toolbar stats display with mvlc (rate, replayRate, etc)
* do add in per pipe and per packet channel stats displays for the mvlc!
* mvlc parser debugging
* change the analysis default filters generation to allow including custom
calibration operators and other things in the default_filters.analysis for
each module. The use case is to calibrate the module_timestamp to
microseconds by default and to allow custom calibrations for some of the
mdpp32_qdc time data.
Think about how this should be implemented. Some partial analysis or analyis
object library or something.
-> object rewrite code is already in place. When creating default filters the
source module is known, thus the object graph can be rewritten to connect to
that module. A problem is still the naming of things. Since the introduction
of directories for the default calibrations and histograms the indvidual
operator names can be pretty generic but the directory names should contain
the module name they're connected to. Use placeholders in the template names?
%m to be replaced by the actual module name?
* think about moving the business logic in vme_config_ui into factory functions
for new EventConfig and ModuleConfig instances. Also factor out things like
getting the next available mcst address.
-> done for EventConfig (vme_config_util.h)
* trigger_io gui [maybe]
- show static connections when hovering over units.
- highlight possible connections when hovering a pin
- allow making connections by dragging from pin to pin or by clicking two
pins.
- maybe allow inline editing of names.
- ellide long names. maybe add logic to make the block wider when longer
names are used or increase the width in general.
http://www.sympnp.org/proceedings/57/G29.pdf
* load and save button in the trigger ui editor are hidden now.
implement the functionality!
what should happen if the script is not a trigger io script? meaning the
correct meta block doesn't exist in it?
-> display error message, leave existing script untouched
* generate text file of input and output names used in the setup. this could be
printed and used at experiment time to figure out what's what.
* when modifying the mvlc trigger io via the gui the vme config is not marked
as modified. I'm not sure why because the underlying script does get
modified. Maybe i'm ignoring the modification on purpose?
* daqcontrol listfile filename is not updated when using the vmusb
* Event Server component only mentioned in the daq_control section of the docs.
This needs to have its own section once it performs better.
* Docs: info about basic controls. right-click context menus, shift and control
click selections, double clicks :D
* Help overlay for the analysis. Big fat arrows indicating data flow and arrows
pointing at histograms.
* LUT editor minimize might not be correct. Test this more. Mostly I confuse
myself when creating functions.
* add readout side buffer sniffing similar to what's done on the analysis side.
* after finishing a run/replay with pause enabled the next run will always
start paused even after having toggled pause/resume
* analysis pause does not work when replaying from mvlc(_usb) listfile
* when creating a new counter difference rate monitor the x-axis label says
'not set'. also the y-axis label has no effect. toggling the top-right
'combined' checkbox fixes this.
MVLC mac address for 0007: 04:85:46:d2:00:07
Mon 09 Dec 2019 04:21:00 PM CET Performance Tests with the MVLC
---------------------------------------------------------------
* Linux, USB3, MTDC, Pulser 7 (event len 26-30), 100 max_xfer -> 127 MB/s
TODO: vary max_xfer for the above test
* Linux, USB2 -> 38 MB/s (USB2 limit)
* Linux, ETH, MTDC, Pulser 7 (event len 26-30), 10 max_xfer -> TODO
* make all name input dialogs only accept valid C identifiers
* MVLC issues
======================
* the controller stays 'connected' even tought the command or data pipe
operations lead to connection error. for example when turning off the
mvlc_usb the error poller will get FT_DEVICE_NOT_CONNECTED but the
MVLC_VMEController will not be set to disconnected.
* After a boot the firmware and hardware registers yield 0. Only after starting
and stopping a DAQ those registers are populated. Try to narrow this down.
-> Test this again with the latest MVLC firmware
* Trigger I/O Editor:
- Add ability to reserve units. Used for Timer and StackStart units when
periodic events are present.
- Add ability to generate a script containing only selected/modified units
* NSIS installer: make sure there's no "release" component when packaging the
release branch. This is just because I noticed a 'dev' component in the dev
builds.
* NSIS Installer: make the zadig installer less prominent and annoying.
-> Add a page asking if you're using the VMUSB. If true go to zadig page,
else skip it.
* failing host lookups block the gui. espcially noticeable when loading a
replay and starting it. the gui will feel unresponsive.
* 'QuaZipFile::close(): file isn't open' warnings when recording with the MVLC_ETH
Configurable Button Grid for VME Scripts
------------------------------------------
Buttons in a grid. Can do single column and single row layouts.
Other layout options? No. Just allow having less buttons than (rows * cols).
Also do not immediately discard data when resizing the grid. Instead use it in
case the number of buttons increases again.
Edit mode:
Set Number of rows
Set Number of columns
assign objects or actions to buttons
Set checked/unchecked action/object
Set checked/unchecked icon
set button label
Override icon
Feature: Checkable buttons where one script is executed on check and one on un-check.
Have different labels for the states.
-> Implement this externally from the grid. By passing the bool 'checked' around.
Note: passing the checkstate to the outside means that QSignalMapper
cannot be directly used. Instead two mappings, one for the checked
and one for the unchecked case can be used.
Feature: Repeat mode
Set repeat interval.
Add extra checkboxes next to each button. If checked the action will
be executed periodically.
Checkable buttons will toggle between states as if clicked manually.
Note: there's no way to have different intervals for the checked and
unchecked states.
* Analysis:
- Connecting to arrays of size 1 is not great: the index[0] is always used
when generating names for objects
- Play around with connecting to things. Some cases are worse than they where
in previous mvme versions.
-> Difference: first input connects to an array of size 1. Now the 2nd
input only accepts values instead of full arrays even if those have size 1.
- Check that histo under/overflow is working with active subranges.
- Figure out how the entrycount for histos works right now. It does not get
reset when the histo is reset, only when a new run starts.
* mvme_root_client: on new run detect if the incoming data is compatible in
case things are already loaded. use streaminfo, maybe generate code in memory
and compare to on disk versions
* fix the EventServer "disconnect on listfile" load issue
-> Managing Qt thread affinity and parent/child things are the real problems
here. Maybe a non-qt rewrite of the server component would be a good idea?
* how to get run completion info when interrupting a replay in mvme?
buffers_processed can be equal to buffers_read. The issue is that not all of
the listfile was read. Maybe use a SystemEvent::End
* void MVMEContext::prepareStart() starting mvme stream worker
Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt.
You must not let any exception whatsoever propagate through Qt code.
If that is not possible, in Qt 5 you must at least reimplement
QCoreApplication::notify() and catch all exceptions there.
terminate called after throwing an instance of 'vme_script::ParseError'
Thread 6 "analysis" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe37fe700 (LWP 18067)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff55c3535 in __GI_abort () at abort.c:79
#2 0x00007ffff5ae24f5 in __gnu_cxx::__verbose_terminate_handler() () from /home/florian/src/mvme2/3rdparty/ftdi-d3xx-linux-x86_64/libftd3xx.so
#3 0x00007ffff5adfc56 in __cxxabiv1::__terminate(void (*)()) () from /home/florian/src/mvme2/3rdparty/ftdi-d3xx-linux-x86_64/libftd3xx.so
#4 0x00007ffff5adfc83 in std::terminate() () from /home/florian/src/mvme2/3rdparty/ftdi-d3xx-linux-x86_64/libftd3xx.so
#5 0x00007ffff6180dd3 in qTerminate() () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#6 0x00007ffff6183182 in ?? () from /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#7 0x00007ffff7451fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007ffff569a4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
* when writing a zip listfile change the names of the analysis and message log
files to contain the run id. this also needs changes to the loading code when
searching both of these files.
* cleanup threaded listfile writing code. Also come up with a way to handle write
errors.
* analysis refactoring:
- Get rid of most of data_filter.h in ~/src. Implement what's needed in terms
of the a2 data filter. I think what's needed is something to contain the
original strings and forward everything to the a2 implementation of the
filter.
- Get rid of Parameter::value but add a name field instead. The high level
parameter vector pipes are not needed to hold data and valid/invalid state
anymore. Instead a feature for naming individual array elements should be
added. The name could be stored directly in the parameter.
- Get rid of runtime data from the higher level Analysis layer. Refactor
Analysis to only contain config data and output size calculations where
needed. Maybe move runtime information and the a2 adapter into a separate
state object. This way Analysis would not be responsible for config
storage, connection logic, building and running.
- Rethink the connection logic, who owns things and when they are destroyed.
Also redesign how threading and the worker startup logic is done. Try to
use a copy of the analysis inside the worker. Make ui access to the runtime
data safe.
* Protect access to the globals DAQStats object with a mutex.
* histogram data is wrong after zooming and applying a subrange
* check out 'pigz' by Mark Adler for how to compress data in parallel.
* Maybe: slow mode for replay so that users can see what's going on.
* Nicer format for analysis rates (top side): less decimals, monospace font
* When clearing histos the entry counts in the analysis tree do not reflect the
new counts (bottom row of analysis ui).
* revive the event_server client code. treewriter_client2 needs some work.
Then document the whole thing and the idea behind it.
* fix the windows warning about winsock2.h before windows.h include order warning somehow
* MVLC_ReadoutWorker: If no data was read within some interval that fact should be logged.
* readout code and timetick sections in the listfile:
if multiple timeticks are generated by
TimetickGenerator::generateElapsedSeconds() in one of the readout workers
and then written to the listfile using via
m_listfileHelper->writeTimetickSection() then the text contents of the
timetick sections in the listfile will all contain the formatted datetime
value of the point in time where writeTimetickSection() is called. This means
multiple timeticks will have the same formatted time string instead of each
being different by rougly 1s.
When does timetick gen generate more than one second?
-> If the readout loop blocks for >= 2s. Otherwise if the time difference is
>= 1s only one tick will be generated and the remainder is stored for the
next tick.
This is not a problem anymore with the MVLC because 64-bit unix timestamp
values are stored in the contents of the timetick system event.
* use TicketMutex in histograms
* Design and implement an interface to make VMEScript extendable. This way
controller specific functions do not need to be implemented directly in the
VMEScript implementation but could be added dynamically when switching VME
controllers.
Also and probably more important is that libmvme_core would not have to
depend on VMUSB anymore.
* Find a better way to access global actions than the method used in
VMEConfigTreeWidget::setupActions(). Maybe use a global action factory thingy
returning QAction "singletons".
* remote control: issuing a start followed by a stop can crash the application.
There are missing checks if the system is transitioning. Set a flag
immediately before calling startDAQ and unset it either on error or when the
start succeeded. While the flag is set do not allow calls to startDAQ/stopDAQ
but return "Starting"/"Stopping", etc
* JSON RPC Remote Control and/or DAQ Start/Stop handling is still buggy!
Quickly issuing startDAQ/stopDAQ commands (including multiple starts/stops in
sequence) can end up in a broken state and eventually lead to a crash:
DAQ: Idle, Analysis State: Running is shown in the gui
Crash in the json rpc code but I believe the heap is corrupted before that.
In MVMEContext::startDAQReadout(): introduce a definitive DAQ/System state
inside MVMEContext that is modified from within the GUI thread only and
that reflects what the state the system is in.
Set this flag to "Starting" right before issuing invokeMethod calls on the
workers. This way multiple calls to startDAQ in quick succession would not
cause an issue as the 2nd call would immediately see the "Starting" state.
Also in stopDAQ check that the current state is "Running" only then issue
the stop request to the worker threads.
* when trying to open a workspace but the directory does not actually contain a
workspace (ini missing is the only hard criteria I think) then ask the user
if she wants to create a workspace instead of just creating stuff.
* during a replay extract timestamps from timetick text and pass this on to
data consumers.
think about storing a 64bit seconds-since-epoch value instead to avoid having
to parse string data.
On the other hand I do like the string data.
In any case timezone information is not stored right now but I think
durations are more interesting to the user than absolute points in time.
data transfer client lib
- no libmvme dependency
- depends on headers shipped with mvme at compile time
- header only for now
root data transfer client
- uses data transfer client lib
high level overview:
- connect to server
- receive run description
- generate code for exported objects, Experiment, Events, Modules, data members inside the modules.
- at this point the base class interfaces are known to the compiler, but not the
concrete, newly generated specific classes
- the generated code is built and loaded using TSystem::Load()
- the specific Experiment subclass can be instantiated via
auto exp = reinterpret_cast<TheExperimentClassName *>(
ProcessLineSync("new TheExperimentClassName()"));
-> We now have a working Experiment instance that can generate its event trees
and that can be queried for events and modules.
- open output file now and generate the event trees using the virtual exp->MakeTrees()
During the run switch on the incoming event number and fill the corresponding
event and module data structures. Figure out how to safely fill the arrays
without crashing in case of size mismatches, etc.
Then call fill on the event tree to commit the data to ROOT.
1) At this point more code could be loaded, e.g. from a user lib.
What's needed: begin and end hooks and hooks for specific events.
Also info on whether this is a replay or an online run.
NOTE: Now there are multiple ways to replay data: from within mvme doing the
steps outlined above and from an existing output ROOT tree.
/* State of the analysis UI and future plans
*
* Finding nodes for objects
* Can use QTreeWidgetItem::treeWidget() to get the containing tree. Right now
* trees are recreated when updating (repopulate) and even with smarter updates
* trees and nodes will get added and removed so the mapping has to be updated
* constantly.
* Can easily visit nodes given a container of analysis objects and create
* chains of node handling objects. These objects could be configured by giving
* them different sets of trees and modes. Also the responsibility chains could
* be modified on the fly.
*
*/
--------------------------------------------------
BUGS
--------------------------------------------------
* runid handling: currently a chicken egg problem:
the readout worker thread generates a new run filename and tries to open that file.
errors can happen and thus the run cannot proceed properly. also the runid
information cannot be sent to e.g. clients connected to the data server.
Fix: move output file creation, opening and error handling from the readout worker
to the main thread and pass the open file handle to the readout worker.
* When having a single h1d bin filled the calculated and displayed fwhm of the
gauss is 1, but I was told it should be 0. Figure this out and fix it.
* Recheck/Test ConditionFilter
* Analysis single step button can hit an assertion when clicked in quick
succession
* blt count reads do not work properly. This needs fixing for both controllers.
Being able to run those commands by uploading a command list and executing
that directly would be very helpful for debugging this.
- VMUSB: data is offset by 16 bits and this looks mangled. The parser code errors out.
- SIS: figure out what's happening. It looks like the number read from the
register is put into the data stream (verify).
* analyis processing trees: After renaming an object resort the tree it is contained in.
This will conflict with the planned manual sorting of DataSources. Do not auto sort
the data extraction tree.
* Add a refresh button to the ui. (same a repop right now, just with a nicer icon :)
* SIS: when not connected and doing an Init Module mvme hangs in recvfrom()
* Editing a H2D name via 'F2' is not really possible while a run is active. The
nodes text will be updated periodically which interferes with editing.
* When changing VME event trigger conditions the readout cycle end script can
become empty. How? What's happening? Is this true?
* Add BlockReadFlags to VMEController::blockRead() and implement support for
the controller side wordswap in the sis library.
--------------------------------------------------
Testing setup for ESS
--------------------------------------------------
* Use RPC interface to start/stop the DAQ
* Enable Jumbo Frames!
* Do this repeatedly with different amounts of on/off time.
* Check the results of the RPC calls.
* Poll stats peridocally and make sure they are valid.
* Try with listfile writing on and off.
* Cycle 100 times or more
According to Anton things get unstable with Jumbos enabled (he cannot enable
the pulser which looks like register writes are failing). Also mvme crashes
sometimes when starting a new DAQ run.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WISHLIST
--------------------------------------------------
* unify runinfo into a single variantmap
* Think about file I/O for the expression operator
* Add new data sources yielding:
- the current timetick value (1s granularity, paramvector of size 1)
- the current event #
* Doc: add ListFilterExtractor to Data Sources
* DAQ Start and then immediately pressing "Run script" in the debug window
causes a crash.
* Debug CBUS functionality: MDPP-16 doesn't get a CBUS result after leaving DAQ
mode. It's needs a write to the reset register to make it work again. Why?
* if loading a config file fails when opening a filename from the
mvmeworkspace.ini file show an explicit error message about the filename in
the ini possibly being wrong.
* void Histo1DWidgetPrivate::calibApply()
void Histo1DWidgetPrivate::calibResetToFilter()
Change them to make use of the improved analysis rebuild mechanism
* JSON RPC: add info about SIS hostname/IP and firmware revision and VMUSB usb
port and firmware.
* Try to get rid of the module assignment dialog. Instead use the "unassigned"
category for data sources, the usual "input is not conneceted -> draw in red"
way of communicating issues with the operator to the user and drag/drop to
fix module assignments.
Problems: events may not match up at all. Right now there's no way to display
stuff that can't be assigned to a particular VME event. Also cross-event
drag/drop is not possible in any way right now.
* workspace and vme controller settings and such:
Roberts use case is: open listfile, try to locally reproduce problems a
customer has. This means he will in most/all cases have to change the sis
address or even the controller type to match his local setup. This is
annoying. Even more of a problem is that after making changes he has to
change the SIS back to the customer settings if he wants to send them the
listfile.
Instead when loading a listfile keep the local controller settings and/or use
a "dummy" controller.
My use case: see exactly what the customer is doing: hostname or ip-address,
jumbo frames, additional controller settings. I sometimes really need this
info and do not want it to be hidden behind some magic.
* New Feature: add pause/resume markers and/or absolute timestamps to the listfile.
This would allow for example splitting the run on pause/resume if the user
changed settings in-between.
* RPC: limit max request size to something reasonable
* RPC: fork jcon-cpp on github and push local changes there
* Improve the loadAnalysisConfig() and similar callchains. Right now they end
up in gui_read_json_file() which displays a message box with a generic "could
not read from" error but nothing specific.
Return value can be tested by checking for isNull() on the returned document.
* Multievent: figure out how and if the stream processor correctly handles
modules that are disabled in the vme config.
* FIXME: why does mutli event for the whole event get disabled if one of
the modules is disabled? This does seem unnecessary.
* in openWorkspace(): figure out if the special handling of the listfile output
dir is really needed.
* try to apply DRY to analysis and vme config serialization. grep for
"DAQConfig" and "AnalysisNG" extract that code into functions and update
callers
* multi output arraymap
* constant generator operator:
multiple outputs
stepped each time the event has data (a2_end_event)
fixed rank of 0 so that these are always stepped first, thus consumers can
count on having the data be active on the output pipes
* DAQ readout performance: try to make better use of the data buffers.
Currently one processed controller buffer ends up in the data buffer, which
means for sis it's only around 1500 bytes.
Do the following: put events into the same buffer until less than a user-set
limit is free in the buffer or a specific time interval has passed. Then
write the buffer to disk and in case of a non-local buffer put it into the
"full" queue.
Make sure the condition where an event should still go into a certain buffer
but the space inside the buffer is not enough is reported properly. In this
case the user-set limit has to be adjusted.
Reallocations of the buffer could be done but the case where a buffer is
enlarged again and again must be avoided.
* a2::Operator_KeepPrevious does not seem to set output limits. Is this bugged?
Fix the test case if needed.
--------------------------------------------------
ExpressionOperator future design plan
--------------------------------------------------
Additional/Advanced Features:
* user defined functions using the exprtk syntax. SECTION 15 -> function_compositor
Limited to returning a single scalar value and up to six input parameters
- function name
- args and their names
- code
// define function koo1(x,y,z) { ... }
compositor
.add(function_t()
.name("koo1")
.var("x").var("y").var("z")
.expression("1 + cos(x * y) / z;"));
-> one function compositor per operator. This also allows to define recursive
functions and functions that call other "dynamic" functions.
Also check out the add_auxiliary_symtab() method of the compositor. This
would allow having access to variables from arbitrary symbol tables in the
function body definition. -> use the a2 runtime library symbol table with this method.
* persistent / static variables
Could be used to keep some accumulator or the last N events or the last
result around in the step script.
statics would be (re)initialized at compile time and then kept during calls to step.
To make this work the following is needed:
for scalars: name and initial value
for arrays: name and size.
Then the expression operator can create the variables in the symbol table
instance and initialize them.
These variables will only be available in the step expression.
The size definition of arrays must be done via an expression aswell as that's
the only way to dynamically get to the input sizes. This script will have the
same symbol table as the begin expression.
Alternatively could use a return statement returning pairs of (name,
variable) and then loop through the variables and create and register the
appropriate types.
Which solution is better?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* 13:22:12: SIS3153 Warning: (buffer #109373) (partialEvent) Unexpected packetStatus: 0x89, expected 0x132. Skipping buffer. (suppressed 105 earlier messages)
packetStatus 0x132 is not valid! This should wrap.
* SIS readout: received count and Timestamp histo count have a small delta of 1
up to 3 counts. Figure out where this is coming from.
* listfile writing, MVMEStreamWriterHelper: can writing of the final EndMarker
be moved into closeEventSection? I was surprised that this had to be done
manually.
* efficiency: when the efficiency is slightly below 100% the display will still
show a 1.00 due to rounding. maybe color the number if it's not 100%.
* New Export/ListfileFilter Idea: use an operator to decide whether a full, raw
VME data event should be copied to an output listfile or be suppressed.
This is basically a copy and filter operation for listfile with the decision
being made at an arbitrary point in the analysis for the specific event.
This needs access to the raw input data.
* Transport more state of the DAQ to the analysis side. Right now the stream
processor always creates a session auto-save even if the DAQ did not start up
successfully (e.g. because the controller already was in DAQ mode). Need to
set some flags in the stream worker so that it can decide on which actions to
perform and which to skip.
* zachary use case: splitting data by time: 1st, 2nd and 3rd hour. good system
to solve these kinds of problems.
* add example of how to extract an extended timestamp (48 bits) from our modules to the documentation
* Tell Tino about SIS packet loss detection not being accurately possible
because of the sequence numbers only appearing before events, which has the
effect that with partial and/or multievent packets, the number of packets and
the sequence number do not match 1:1 anymore.
In the original SIS3153 format this wasn't an issue as there was no buffering
or partials.
A real packet number prefix would be ideal.
* Maybe add a session file browser/explorer thingy. Could even allow multiple
open sessions at once. Make sure histos and other objects clearly show which
session they belong to.
* Change analysis input selection handling to not modify the involved operators
directly. Instead create fake input pipes and connect the target operator to
those. Event better would be to create a clone of the original operator to be
edited and work on that with fake inputs.
This way no pausing and/or clearing would have to be done if the user opens
the editor, clicks around and then cancels.
Also if only the name or unit info is edited nothing actually would need to
be rebuild (NOTE: some operators might pull copies of their input unit labels
or prepare some calculation depending on the limits so be careful with
this!).
* Fix assertion when building a2 (might lead to corruption or crashes in a
release build!) in the following scenario:
- H1DSink is connected to a DataFilter by a direct index connection.
- The address bit of the filter are edited by the user. The new number of
address bits is less than the old number.
- The H1DSink is now connected to an index that doesn't exist anymore.
- The a2 adapter will assert because no histo is found for the index.
This also affects other operators. Fix this!
* Rate Monitor Widget:
- make sure rate values are scaled to the "middle" of the y axis
- improve time labels: don't show hours and minutes if time scale is less than a minute
-> This has the problem that if you look at a window of the last 60
seconds, in a run that's over an hour long you won't see the hour number. You
lose the information that you're in hour 1 plus x in the run.
I could check if the scale is past a threshold and then switch the formatting.
if (elapsed_time > 1 hour)
set scale for intervals below the hour threshold to show hours
else
use a scale that hides the hours
Does the same problem happen for other intervals?
* All filters: make sure address bits are <= 16. This is a reasonable limit and
avoids crashes due to wrong inputs into the filter fields.
* VME Script "Run" button: this still blocks the GUI.
--------------------------------------------------------------------------------
Data Export Implementation - ExportSink
--------------------------------------------------------------------------------
More ideas from Robert:
* Make it possible to export multiple events into the same file. This will need
a way to select data from different events. Also the file needs to contain
the event number for each record.
* Add a "system" event containing the current timetick count and systems stats,
rates, etc.
* The export sink needs to log which output file is being written! Right now
it's all very mysterious.
* (maybe) byte order marking. The bin file are not meant as permanent storage
but to be consumed by a user program which then stored data in e.g. ROOT
format.
* (maybe) file magic number
* (maybe) version number
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* add build date to info window thingy
* Andi has a setup where the "compression" combo box has an empty value. no/fast compression can still be selected but the initial value is empty.
* filters: add option to disable adding the random value (done for listfilters)
* Projection of combined H2D does not update axis scales after the source H1D
has been recalibrated.
* Changing H1D calibration via its widget clears all histograms. This is not
needed as only the interpretation of the data in the histo changes. Improve
this case so that it keeps existing histo data.
-> This needs fixing in Histo1DSink::beginRun(): currently the histo
mem is cleared no matter what changed.
Also the case where the analysis is not running has to be taken care of: the
AnalysisPauser won't invoke reusmeAnalysis() and thus it won't be rebuilt.
* add sis3153 firmware update utility to 'extras' directory. Also package and
install sis library code (maybe). Ask struck if this is allowed.
* Add element wise array multiplication operator and/or a calibration operator
using slope and offset values.
* add 64 bit DataFilter
* phase out data_filter.h (DataFilter and MultiWordDataFilter). stick to the a2 stuff
* Make mvme internal "system" rates available to the analysis.
Might split this into VME and Analysis rates. Think about how to feed those
to the analysis system.
Make this efficient and easily integrated.
* RateHistory needs to only be kept if the specific rate should be plotted.
Otherwise if the rate should only be displayed it's enough to keep the last
(most recent) rate value that was calculated. The GUI should be able to pick
that up.
Totals are also interesting and should be displayed alongside rates.
System design:
List / Tree of available counters. Must be able to deal with arrays of known
size, like ModuleCounters[E][M]. On the other hand at runtime it is known how
many events and modules there are so only parts of the arrays have to be
exposed.
Hierarchy is good. Names are good. Serialization of the information to restore
a counter display and the required history buffers is needed.
Then be able to find a RateHistoryBuffer via a "hierarchy path".
readout.bytes
readout.buffers
streamProc.bytes
streamProc.event[0] .. event[N] with N either begin MaxVMEEvents or the actual number of events in the VMEConfig
streamProc.event[0].module[0] with limits being the number of events and the number of modules in that event
accessor function to get the current counter value
accessor function to get the previous counter value
-> delta calc possible
accessor to get calibration variables and unit labels and additional options
accessor to get the RateHistoryBuffer for the specific counter
-> filling the buffer is possible
With a system like this different counters could be updated in different
places, e.g. some counters could be handled entirely in the GUI, others might
be updated iff it's a replay and a timetick section is processed.
The update code needs an efficient way to query the rate monitoring system for
counters it needs to process:
void foo(double sample_value)
{
// constant during the course of the run
auto byteRateHandle = get_handle_for("daqtime.streamProc.bytes");