-
Notifications
You must be signed in to change notification settings - Fork 0
/
NEWS
4820 lines (3653 loc) · 204 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This document summarizes the most important changes in the current Zeek
release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as Broker, come with their own ``CHANGES``.)
Zeek 5.2.0
==========
Breaking Changes
----------------
- Zeekctl now assigns network ports to workers starting at port 27760. This
fixes an issue where workers were starting up with ports within Linux's
ephemeral port range, and were potentially failing to startup due the ports
already being in use. This change may require changes in firewall/routing
configurations between hosts in a Zeek cluster. This should not affect
clusters running on FreeBSD, as that OS uses a different range for ephemeral
ports.
- The blank identifier ``_`` cannot be used in expressions and options anymore.
Outside of obfuscation exercises, this should have little real-world impact.
New Functionality
-----------------
- Experimental support added for building and running Zeek on Microsoft Windows
environments. This is considered experimental due to the fact that our
standard testing setup (btest) doesn't run properly on Windows. This will be
fixed in the future. In the meantime we have done some basic testing against
builds done with Visual Studio 2019. Information on how to build on Windows is
available in the Zeek documentation. Note also that Spicy is currently
unsupported and will be fixed in the future.
The Zeek team wants to give a huge thank you to the team at Microsoft for all
of their effort in completing this port.
- New ``analyzer_confirmation_info`` and ``analyzer_violation_info``
events with accompanying record types ``AnalyzerConfirmationInfo``
and ``AnalyzerViolationInfo`` have been added. These supersede
``analyzer_confirmation`` and ``analyzer_violation``, which have
been deprecated.
- Added helpers to determine protocol, packet or file analyzer based
on ``AllAnalyzers::Tag`` values named ``is_protocol_analyzer()``,
``is_packet_analyzer()`` and ``is_file_analyzer()``.
- File analyzers can now raise analyzer violations to the script-layer via
the new ``AnalyzerViolation()`` method.
- Packet and file analyzers can now be disabled and enabled at runtime using
the ``Analyzer::enable_analyzer()`` and ``Analyzer::disable_analyzer()``
wrappers. While initially for protocol analyzers only, these have
been extended to work for packet and file analyzers. This now allows to
leverage ``Analyzer::disabled_analyzers`` for these kinds of analyzers.
- The blank identifier ``_`` can now be used to ignore loop variables of
different types without type clash errors. This allows to do the following
within the same scope:
local vec = vector("a", "b", "c");
for ( _, v in vec )
print v;
for ( i, _ in vec )
print v;
Iterating over only the values of a table can be done by ignoring the full
index with a single blank identifier. Due to the internal structure of Zeek
tables, this can result in a performance improvement.
local tab = table(["a", 1, T] = "a1T", ["b", 2, F] = "b2f");
for ( _, v in tab )
print v;
It's also possible ignore individual indices of different types with the
blank identifier ``_`` as follows:
for ( [_, i, _], v in tab )
print i, v;
As noted under breaking changes, the blank identifier ``_`` cannot be
referenced in expression anymore.
- Re-introduce event groups. Allow the ``&group`` attribute on event and hook
handlers for annotating them with one or more event groups. These groups
can be disabled and enable during runtime. Disabling an event group implies
disabling all event and hook handlers that are part of it.
The main difference to a previous implementation in (very) old Zeek versions
is its granularity: It is now possible to toggle individual event handlers
(event handler bodies). The original implementation worked at the level of
events, disabling or enabling all event handlers for a given event at once.
Additionally, support for hooks was added as these are structurally similar
to events.
The BIFs ``disable_event_group()`` and ``enable_event_group()`` are
re-instantiated and allow controlling event groups based on the group
attribute.
Additionally, event and hook handlers are implicitly placed into event
module groups based on the module they are implemented in. All events
implemented in a given module can be toggled with ``disable_module_events()``
and ``enable_module_events()``.
- On Linux, the AF_PACKET packet source plugin (https://github.com/zeek/zeek-af_packet-plugin)
is included as builtin plugin by default. To select this packet source, prefix
the interface name with ``af_packet``.
zeek -i af_packet::eth0
- Usage of ``break`` and ``next`` statements is now validated. It was previously
possible to place these outside of ``for``, ``while`` or ``switch``
statements without any error indication.
- Add two BIFs ``get_identifier_declaring_script()`` and ``get_record_field_declaring_script()``
to query the declaring scripts for identifiers and record fields from Zeek scripts.
Changed Functionality
---------------------
- Violations for packet analyzers that have sessions attached with them
will be raised once only. Further, analyzer confirmations are not raised
after a violation.
- The parameter given to ``enum_names()`` can now be a string naming the
enum type, rather than the type itself.
- The ``type_name`` of enum types produced by ``record_fields()`` now
includes the actual type name rather than just ``"enum"``.
- Passing non-string ``sep`` and ``def`` arguments to ``cat_sep()`` isn't a
fatal error anymore. More descriptive error messages are produced, too.
- The number of analyzer violation events that can be raised by protocol
analyzer instances is now capped by the const ``max_analyzer_violation_events``.
- The number of analyzer violation events that can be raised by protocol and
file analyzer instances is now capped by the const ``max_analyzer_violation_events``.
Its default is 1000 and the main purpose is to prevent analyzers from
scheduling too many ``analyzer_violation_info`` events before the
DPD ``max_violations`` script-level logic has a chance to run and disable
the problematic analyzer.
- The TCP analyzer now continues processing payload for some
connections missing initial packets where it would previously have
stopped. This fixes a few cases where we already had the logic to
continue in place, but we still ended up considering them partial.
- Count underflows via ``--c`` or subtract from statements (``c = c - 1``) are
now consistently warned about. Previously, underflows through ``--c`` were
treated as runtime errors, while "subtract from" underflows were silently
accepted. The following (surprising behavior) now causes a warning, too:
$ zeek -e 'print 1 - 2'
expression warning in <command line>, line 1: count underflow (1 - 2)
18446744073709551615
- The MQTT scripts registering the analyzer and DPD signatures have
been moved from the policy folder to base and are loaded by default.
Deprecated Functionality
------------------------
- The ``analyzer_confirmation`` and ``analyzer_violation`` events have been
deprecated in favor of the more generic ``analyzer_confirmation_info`` and
``analyzer_violation_info`` events.
- The const values for toggling individual tunnel packet analyzers have
been deprecated in favor of using ``Analyzer::disable_analyzer()``
directly. This affects:
Tunnel::enable_ip
Tunnel::enable_ayiya
Tunnel::enable_teredo
Tunnel::enable_gtpv1
Tunnel::enable_gre
Setting these explicitly to F can be achieved by leveraging
``Analyzers::disabled_analyzers``, for example:
redef Analyzer::disabled_analyzers += { PacketAnalyzer::ANALYZER_GRE };
- The ``zeek::merge_type_list()`` function has been deprecated. Please consider
the partially compatible and saner ``zeek::maximal_type()`` instead. See
GH-2604 for context.
Zeek 5.1.0
==========
Breaking Changes
----------------
- The ``Packet::{l2,l3}_checksummed`` variables were reworked to correctly match
the network layers that they apply to. A new ``Packet::l4_checksummed``
variable was added to cover the transport layer. See this GitHub issue for
more detail: https://github.com/zeek/zeek/issues/2183.
- The STREAM mode of the ASCII reader now behaves like `tail -F`: when file is
removed/replaced, it will start tracking the new file. See
https://github.com/zeek/zeek/pull/2097 for more detail
- The Dictionary and PDict classes are now C++ templates. This may cause
plugin/package builds to fail due to needing to modify uses of them to match.
- By default, ``files.log`` does not have the fields ``tx_hosts``, ``rx_hosts``
and ``conn_uids`` anymore. These have been replaced with the more commonly
used ``uid`` and ``id`` fields. They can be re-instantiated by loading the
following policy script through ``local.zeek``:
@load frameworks/files/deprecated-txhosts-rxhosts-connuids
Note, however, that this script will be removed with Zeek 6.1. Consumers
of ``files.log`` should convert to using the singular ``uid`` and ``id``
fields instead.
- The ``files.log`` is now unrolled consistently. That is, when Zeek associates
multiple connections with a single file, each of these connections will result
in individual ``files.log`` entries with unique connection uids, all sharing
the same file uid.
This unrolling behavior always existed in a Zeek cluster when the network
connections involved in a file transfer are load-balanced to different
workers. Due to this affecting only a marginal ratio of files on real-world
networks, unrolling the log was chosen as the more efficient approach over
making the current logic cluster aware.
The ``seen_bytes`` and ``missing_bytes`` fields of a ``File::Info`` record
continue to represent the total number across all connections seen by the
current instance of Zeek.
- The barnyard2 policy scripts have been removed. The integration with the
Barnyard2 project used the pre-Broker Broccoli library, which got removed in
Zeek 3.0.
- The unified2 analyzer and accompanying scripts have been removed without
deprecation.
- The return value of ``packet_analysis::IP::ParsePacket`` has changed to return
enum values. This makes it easier to look at the result and immediately know
what it means. Unfortunately, because we can't overload a method on the return
value alone, we aren't able to deprecate the original version of the method.
This may cause build of packages to fail if they were using this method.
- Conditional directives (``@if``, ``@ifdef``, ``@ifndef``, ``@else`` and
``@endif``) can not be placed directly following ``if``, ``for`` or ``while``
statements anymore. This was interpreted non-intuitively and could lead to
subtle bugs. The statement following the directive was placed outside of its
intended block. Placing braces after ``if``, ``for`` or ``while`` should
result in the intended behavior.
- The ``bro`` symlink to the ``zeek`` binary has finally been removed.
- The connection's service field does not hold "negated" analyzer names
anymore (analyzers that triggered violations after a confirmation).
These analyzers are now tracked in a separate ``service_violation``
field with their respective enum values instead.
New Functionality
-----------------
- Added support for the /s regular expression modifier. Using this modifier in
patterns in Zeek scripts will cause the '.' character to also match newline
characters.
- Added a new telemetry framework for providing high-level access to Zeek's
metric subsystem. This framework allows script writers to use different metric
types (counters, gauges and histograms) for tracking metrics without using
lower-level BiFs from ``telemetry.bif``. Additionally, metrics can now be
accessed from script land using ``Telemetry::collect_metrics()`` and
``Telemetry::collect_histogram_metrics()``.
The framework is located in ``base/frameworks/telemetry``.
In addition to the Prometheus endpoint for metrics export that has existed
since Zeek 4.1, two new log streams, ``telemetry.log`` and
``telemetry_histogram.log``, can be enabled by loading
``policy/frameworks/telemetry/log``. This policy script is included in
``local.zeek`` by default.
For further details on the framework and examples, please refer to the
Zeek documentation.
- Allow redef'ing the ``&log`` attribute of record fields:
redef Notice::Info$email_dest -= { &log };
While the syntax allows for any attribute, only ``&log`` is supported. The
semantics for other record field attributes are not easy to grasp and there
were no obvious use-cases identified.
- Introduced a global ``disabling_analyzer()`` hook to allow vetoing calls
to ``disable_analyzer()``.
The contract is simple: Any script can veto a ``disable_analyzer()`` call by
breaking from this hook. The decision is local to the script taking into
account any state attached to the connection or state stored elsewhere.
A script breaking from the hook takes over responsibility to call
``disable_analyzer()`` at a later point when it finds the condition due
to which it vetoed fulfilled (which may be never).
- Add support for iterating over indices and values of a vector using the
same syntax as used for iterating over key-value pairs of tables, where
``value`` will be set to ``vec[idx]``.
local vec = vector("zero", "one", "two");
for ( idx, value in vec )
print idx, value;
- The Supervisor framework now allows better control over where to place
additional scripts in the load sequence of new nodes. It previously always
loaded such scripts after any other user scripts, which could create pitfalls
when users expected their scripts to run last. Scripts placed in
``NodeConfig``'s new ``addl_base_scripts`` and ``addl_user_scripts`` fields
will be loaded after the base scripts (and thus before any user scripts) and
after any user scripts, respectively. The old ``NodeConfig$scripts` field
still adds to the very end and is deprecated.
- Added a new script-level option ``max_changes_per_connection`` to limit the
number of ``tunnel_changed`` events that can be sent for a connection. This
helps prevent log spam from connections that regularly swap. The option
defaults to 5, and can be set to zero do disable the limiting.
- Added a new BIF ``bytestring_to_float`` for converting 4-byte bytestrings to
float values.
- Added a new BIF ``pow``.
- Added new bit-shift operators ``<<`` and ``>>`` for use in scripts.
- Added a new BIF ``table_keys`` which returns a ``set`` of keys from a table.
- Added a new BIF ``table_values`` which returns a ``vector`` of keys from a
table.
- Added new fields to the Modbus log for the Modbus PDU type, the transaction
ID, and the unit ID. See https://github.com/zeek/zeek/pull/2281 for more
information.
- Added support for parsing TCP option 27, and fixed validation of lengths for
TCP options 28, 29, and 34.
- Added new packet-analyzer to handle the DLT_LINUX_SLL2 PCAP link type.
Changed Functionality
---------------------
- The SSL analyzer now determines the direction of the SSL/TLS session by examining
the packets, and no longer assumes that the connection originator is the client.
Due to this, the ``is_orig`` field in all SSL/TLS events was renamed to ``is_client``.
Furthermore, the ``ssl_history`` now can indicate that the connection was flipped
(meaning that it is not in the normal order of the originator is the client) using
the ``^`` character. A new ``ssl_connection_flipped`` is raised when the connection
is flipped. Furthermore, a ``SSL_unclear_connection_direction`` weird is raised when
we cannot determine the connection direction, because both sides of the connection
send packets that are associated with being a client/server.
- The default logging directory is now set globally across all log
writers through ``Log::default_logdir``.
- Calling `Option::set()` when Zeek is terminating is now a noop and returns `F`.
This prevents callbacks into script-land through change handlers when parts
of the environment have already been torn down.
- When running in cluster mode, the manager by default now imports metrics from
all other cluster nodes and opens port 9911/tcp for Prometheus metrics exposition.
- The ``smb2_file_delete`` event will now be raised for SMB2 ``CREATE`` requests
marked with the ``FILE_DELETE_ON_CLOSE`` option.
- Fixed ``bytestring_to_count`` to handle 3-, 5-, 6-, and 7-byte strings.
Removed Functionality
---------------------
- The barnyard2 policy scripts have been removed.
- The unified2 analyzer and accompanying scripts have been removed.
Deprecated Functionality
------------------------
- The ``PDict`` class is now an alias to ``Dictionary`` and has been
deprecated. Use ``Dictionary`` directly, passing a pointer type to the
template.
- ``LogAscii::logdir`` and per-writer log directories have been deprecated in
favor of the new ``Log::default_logdir``.
- The ``HOOK_BRO_OBJ_DTOR`` hook and associated methods have been
deprecated. They are replaced by the ``HOOK_OBJ_DTOR`` hook and methods.
- The ``bro_int_t`` and ``bro_uint_t`` types have been deprecated and replaced
by ``zeek_int_t`` and ``zeek_uint_t``.
- The ``bro_inet_ntop.h`` and ``bro_inet_ntop.c`` files have been deprecated and
replaced by ``zeek_*`` files.
- The ``BRO_PLUGIN_API_VERSION`` has been deprecated and replaced by
``zeek::PLUGIN_API_VERSION``.
- The ``misc/scan.zeek`` script has been marked for removal in Zeek 6.1. Use
github.com/ncsa/bro-simple-scan instead.
- The Supervisor framework's ``NodeConfig$scripts`` field has been deprecated
and marked for removal in Zeek 6.1. Use ``NodeConfig$addl_user_scripts``
instead.
Zeek 5.0.0
==========
Breaking Changes
----------------
- Zeek now requires at least CMake version 3.15.0.
- If Zeek is configured with support for included Spicy (the default) we now
require at least Flex version 2.6 and its development headers, at least Bison
version 3.3, and GCC version 8.3 or Clang version 9.0 or higher.
- The script-land ``union`` and ``timer`` types have been removed. They haven't
had any actual semantics backing them for some time and shouldn't have
functioned in any useable way. We opted to skip the deprecation cycle for
these types for that reason.
- Broker now uses a new network backend with a custom network protocol that is
incompatible with the pre-5.0 backend. In practice, this means Zeek 4.x will
not be able to exchange events with Zeek 5.x. Going forward, this new backend
will allow us to keep the Broker protocol more stable and add new capabilities
in a backwards compatible way.
New Functionality
-----------------
- The Zeek cluster management framework now supports operational use of
single-instance clusters, meaning setups in which all Zeek cluster nodes
(manager, workers, etc) reside on a single machine. The framework builds upon
Zeek's Supervisor (``-j`` is a requirement) and includes three components:
(1) A cluster controller, loaded via ``policy/frameworks/management/controller`.
The controller is the central management entity for a Zeek cluster. It
interacts with one or more agents to enact instructions delivered by a
management client. The controller is stateful and can persist cluster
state to disk.
(2) One or more cluster agents, loaded via ``policy/frameworks/management/agent`.
Each agent assists the controller in managing the local instance (typically
a machine or container) of Zeek cluster nodes. Agents interact with the
local Supervisor to control nodes and peer directly with running Zeek
nodes via Broker to carry out node-local tasks.
(3) zeek-client, the cluster management client. A standalone client running
outside of Zeek, it's now installed by default, alongside the other
executables. The client supports uploading cluster configurations to the
controller, deploying them, retrieving them, checking cluster node status,
restarting individual cluster nodes, and retrieving the current value(s)
of global identifiers in one or more Zeek nodes.
Controller and agent come pre-configured for single-instance deployments of
Zeek clusters, with automated log rotation and archival via Zeek's
``zeek-archiver`` tool. For further details on the framework, please refer
to the Zeek documentation.
zeekctl remains included and offers more knobs and features than the
management framework. It remains the recommended solution for multi-machine
clusters and those needing rich management capabilities.
- Zeek now automatically warns about unused functions, hooks, and event
handlers. The intent behind these checks is to catch instances where the
script writer has introduced typos in names, or has forgotten to remove
code that's no longer needed.
For functions and hooks, "unused" means the function/hook is not exported or
in the global scope (nor deprecated), and no "live" (i.e., not "unused")
function/hook/event handler calls it. For event handlers, "unused" means
that the event engine does not generate the event, nor do any "live"
function/hook/event handler generates (and the event handler is not
deprecated).
The warnings can occasionally be or inappropriate or incorrect, such as due
to the use of conditional code (which Zeek does not try to incorporate
into its analysis), or events sent via Broker. You can mark such instances
using the ``&is_used`` attribute to suppress the associated warning.
You can also suppress all such warnings using the ``--no-unused-warnings``
command-line option.
- A new feature, ``--profile-scripts[=file]`` instructs Zeek to write (upon
termination) a profile of every function body to the given file (default:
stdout), along with some aggregate profiles. A function body is associated
with a function or a BiF (they can only have one), or a hook or event handler
(they can have multiple). The profile is in TSV, quasi-Zeek-log format. It
includes (1) the name of the script body, (2) its location, (3) its type
(e.g., "BiF" or "function"), (4) how many times it was called, (5) the total
CPU time that accumulated during its execution, (6) how much of that time was
due to execution of its "children" (other functions it called), (7) the total
memory allocated during its execution (approximately), and (8) how much of
that memory was due to its children. Note that profiling is expensive and may
not be suitable for execution on live traffic.
- Zeek now includes support for building Spicy and spicy-plugin as part of
Zeek. This feature is enabled by default, and can be turned off by passing the
``--disable-spicy`` flag to ``./configure``.
- Zeek now supports generation and replay of event traces via the new
``--event-trace`` / ``-E`` command-line options. For details, see:
https://docs.zeek.org/en/master/quickstart.html#tracing-events
- Zeek now features limited TLS decryption capabilities. This feature is
experimental and only works for TLS 1.2 connections that use the
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ciphersuite. Furthermore Zeek requires
access to the pre-master secret of each TLS connection. Typically this
functionality will be most useful when analyzing trace-files where the TLS
client recorded the key material. For more details and examples how to use
this functionality, see the TLS Decryption documentation at
https://docs.zeek.org/en/master/frameworks/tls-decryption.html
- Zeek now provides WebSocket support for exchanging events with external
clients. To enable it, call `Broker::listen_websocket()`. Zeek will then
start listening on port 9997 for incoming WebSocket connections. Zeek/Broker
communicates via JSON messages. See the Broker documentation for a description
of the message format expected over these WebSocket connections at
https://docs.zeek.org/projects/broker/en/current/web-socket.html.
- Compiling scripts to C++ (via ``-O gen-C++``) is now "feature complete", i.e.,
the compiler should be able to process any script - other than those
potentially affected by ``@if`` conditionals - regardless of which Zeek
language constructs it uses. That said, such compilation remains
experimental and only lightly tested.
- ZAM script optimization (``-O ZAM``) now includes some major performance
improvements.
- The new --with-gen-zam configure flag and its corresponding GEN_ZAM_EXE_PATH
cmake variable allow reuse of a previously built Gen-ZAM code generator. This
aids cross-compilation: the Zeek build process normally compiles Gen-ZAM on
the fly, but when cross-compiling will do so for the target platform, breaking
its use on the host platform. Gen-ZAM is similar to binpac and bifcl in this
regard. Like binpac and bifcl, it's now also available as a standalone git
repository and hooked into the Zeek distribution as a submodule.
- Zeek now uses the c-ares (https://c-ares.org) library for performing DNS
requests, replacing an old custom implementation of a DNS resolver. Switching
to this library simplifies the DNS code, adds support for IPv6 lookups, and
adds the ability to support more DNS request types in the future.
- Two new BiFs, val_footprint() and global_container_footprints(), offer
substitutes for the deprecated val_size() and global_sizes() BiFs. A value's
"footprint" is the number of elements it includes either directly or
indirectly. The number is not meant to be precise, but rather comparable:
larger footprint correlates with more memory consumption.
- The Supervisor features two new events, ``Supervisor::node_status`` and
``SupervisorControl::node_status``, to notify recipients of the fact that the
stem process has (re-)started a node. The events indicate the node's name and
its new PID, reflecting the information the stem already shares with the
Supervisor core.
- The new ``configure --statedir`` option lets you adjust the installation's
persistent state directory. It defaults to ``var/lib`` under your Zeek
installation's root directory.
- The ``base/misc/installation.zeek`` script provides your Zeek installation's
key directories.
- Zeek now lets you decrement Counting Bloom Filters using the new
``bloomfilter_decrement`` BiF. Furthermore, two Bloom Filters can be
intersected using the new ``bloomfilter_intersect`` BiF.
Changed Functionality
---------------------
- The behavior of the ``=``, ``+=``, and ``-=`` operators has been expanded and
unified. It now covers ``{ ... }`` initializer lists, supports cross-product
initialization, enables ``+=`` for table, set, vector and pattern values,
similarly allows ``-=`` for table and set values, and supports listing
multiple sets for ``+=`` initialization. For details, see:
https://docs.zeek.org/en/master/script-reference/operators.html#assignment-operators
- The is_num(), is_alpha(), and is_alnum() BiFs now return F for the empty
string.
- Type mismatch error messages now print the easier to understand type names,
instead of the fully expanded raw type.
- Errors caused by setting a filter at start-up are now more informative about
what actually caused the error, including with the default ``ip or no ip``
filter.
- Log messages about errors in input files are now more informative about where
errors occurred.
- The ``--enable-zeek-client`` configure flag has been removed and is now the
default. The new ``--disable-zeek-client`` flag allows users to skip
installation of the client.
Deprecated Functionality
------------------------
- For developers of asynchronous BiFs, or other uses of ``when`` statement
triggers, the versions of ``Trigger::Cache`` and ``Trigger::Lookup`` that take
``const CallExpr*`` parameters have been deprecated. The replacements take
``const void*`` parameters. Usually all you'll need to do is replace previous
uses of ``Frame::GetCall`` with ``Frame::GetTriggerAssoc``.
- The ``--caf-root`` option of ``zeek-config`` was made obsolete by the changes
to how we include CAF and has been marked as deprecated.
Zeek 4.2.0
==========
Breaking Changes
----------------
- The existing ``Tag`` types in C++ (``zeek::Analyzer::Tag``, etc) have been
merged into a single type called ``zeek::Tag``. This is a breaking change, and
may result in plugins failing to build where they were relying on those types
being different for function overloading and such. We attempted to include
deprecated versions of the old types, but were unable to do so because of
changes to return types from a number of methods. With this change, any uses
of the `zeek::*::Tag` types will need to be replaced by `zeek::Tag`.
- The DPD signature for SSL version 2 is no longer enabled by default. SSLv2
is basically extinct nowadays - and the protocol has a relatively high probability
of matching with random traffic and being misidentified. If you want to enable
the SSLv2 dpd signature, you can load the signature from `policy/protocols/ssl/dpd-v2.sig`
The DPD signature for SSL version 3 and up (including TLS 1.0 and above) now matches
for one-sided connections and does not require a reverst match anymore. This prevents
missed handshakes, where the client handshake contains a lot of data.
New Functionality
-----------------
- We now provide minimal official Docker images for the Zeek project via two
repositories on the Docker hub, ``zeekurity/zeek`` and ``zeekurity/zeek-dev``.
The former receives all Zeek release versions, with tag ``zeek:latest`` being
the most recent. An image corresponding to our latest merge into the master
branch is tagged at ``zeek-dev:latest``.
The images run Debian and provide a full install of the Zeek distribution into
``/usr/local/zeek``. They do not set Zeek-specific entrypoints or provide any
particular configuration for operationally running Zeek. To keep the images
lightweight they also do not contain a development toolchain as needed for
example to build a Zeek plugin. You can add any required system packages in a
derived image, or install them directly in the running container.
- Zeek now supports formatting the C++ code using clang-format. Also provided is
a configuration for ``pre-commit`` to run clang-format when add new commits via
``git``. More details can be found at https://github.com/zeek/zeek/wiki/Coding-Style-and-Conventions#clang-format.
- Experimental support for speeding up Zeek script execution by compiling
scripts to a low-level form called "ZAM". You activate this feature by
specifying ``-O ZAM`` on the command line. See
``src/script_opt/ZAM/README.md`` for more information.
- Improvements for compiling scripts to C++ (an experimental optimization
feature introduced in 4.1). The generated C++ now compiles much faster than
previously, though it can still take quite a while when using C++ optimization
on large sets of scripts. You can incrementally compile additional scripts
using ``-O add-C++``. See ``src/script_opt/CPP/README.md`` for details.
- The new flags --optimize-files=/pat/ and --optimize-funcs=/pat/ apply
to both ZAM and compile-to-C++ script optimization. The first instructs
Zeek to optimize any functions/hooks/event handlers residing in files
matching the given pattern (unanchored). The second does the same but
based on the function name, and with the pattern anchored (so for example
--optimize-funcs=foo will optimize any functions named "foo" but not
those named "foobar", or "MYSCOPE::foo"). The flags can be combined
and can also be used multiple times to specify a set of patterns.
If neither flag is used then optimization is applied to all loaded
scripts; if used, then only to those that match.
- The ``-uu`` flag for analyzing potentially unused record fields has been
removed because, due to other changes in script optimization, keeping it
would now require about 1,800 lines of code not otherwise needed.
- The DNS analyzer has initial support for the SVCB and HTTPS types. The new
events are ``dns_SVCB`` and ``dns_HTTPS``.
- The ``find_str`` and ``rfind_str`` bifs now support case-insensitive searches.
- Added a new plugin hook for capturing packets that made it through analysis
without being processed called ``Plugin::HookUnprocessedPacket``. Currently
ARP packets or packets with a valid IP-based transport header are marked as
processed. This also adds an event called ``packet_not_processed`` that
reports the same packets.
- A new command-line option ``-c`` or ``--capture-unprocessed`` will dump any
packets not marked as being processed, similar to the new hook and event
above.
- In Zeek plugins, the new cmake function ``zeek_plugin_scripts()`` should be
used alongside ``zeek_plugin_cc()`` and related functions to establish
dependency tracking between Zeek scripts shipped with the plugin and plugin
rebuilds. Previously, updates to included Zeek scripts didn't reliably
trigger a rebuild.
- Added PacketAnalyzer::register_for_port(s) functions to the packet analyzer
framework in script-land. This allows a packet analyzer to register a port
mapping with a parent analyzer just like any other numeric identifier, while
also adding that port to the now-global Analyzer::ports table used by BPF
filtering.
- Added AllAnalyzers::Tag enum type that combines the existing Analyzer::Tag,
PacketAnalyzer::Tag, and Files::Tags into a single enum. The existing types
still exist, but the new type can be used as an argument for
functions/hooks/events that need to handle any of the analyzer types.
- Added protocol detection functionality to the packet analyzer framework.
Packet analyzers can register for protocol detection using the
``PacketAnalyzer::register_protocol_detection`` script function and implement
the ``PacketAnalyzer::DetectProtocol`` method in C++. This allows packet
analyzer plugins to detect a protocol via byte matching or other heuristics
instead of relying solely on a numeric identifier for forwarding.
- The JSON logger's new LogAscii::json_include_unset_fields flag provides
control over how to handle unset "&optional" fields. By default it continues
to skip such fields entirely. When redef'ing the flag to T it includes such
fields, with a "null" value. This simplifies data import use cases that
require fields to be present at all times, regardless of their value.
- A new external testsuite, https://github.com/zeek/zeek-testing-cluster,
focuses on testing the emerging controller framework. It leverages the new
official Zeek Docker image for building docker-compose test setups, driven via
btest. The GitHub CI setup now includes a workflow that deploys and runs this
testsuite.
- The GRE analyzer now supports the Aruba WLAN protocol type.
Changed Functionality
---------------------
- Zeek now treats non-atomic index types for sets and tables more consistently.
Indexing now generally works with any types Zeek's hashing subsystem knows how
to serialize, meaning that you can now also index with sets, vectors,
patterns, and even tables.
- The traditional TSV Zeek logs are now valid UTF8 by default. It's possible to
revert to the previous behavior by setting ``LogAscii::enable_utf_8`` to
false.
- The ``SYN_packet`` record now records TCP timestamps (TSval/TSecr) when
available.
- The ``init-plugin`` script now focuses purely on dynamic Zeek plugins. It no
longer generates Zeek packages. To instantiate new Zeek packages, use the
``zkg create`` command instead.
- The ``ignore_checksums`` options and the ``-C`` command-line option now
additionally cause Zeek to accept IPv4 packets that provide a length of zero
in the total-length IPv4 header field. When the length is set to zero, the
capture length of the packet is used instead. This can be used to replay
traces, or analyze traffic when TCP sequence offloading is enabled on the
local NIC - which typically causes the total-length of affected packets to be
set to zero.
- The existing tunnel analyzers for AYIYA, Geneve, GTPv1, Teredo, and VXLAN are
now packet analyzers.
- C++ unit tests are now compiled in by default and can be disabled by
configuring the build with --disable-cpp-tests. We removed the former
--enable-cpp-tests configure flag. Unit tests now also work in (static and
dynamic) Zeek plugins.
- This release expands the emerging cluster controller framework. Most changes
concern internals of the framework. Agent/controller connectivity management
has become more flexible: configuration updates pushed by the client can now
convey the agent topology, removing the need to hardwire/redef settings
in the controller. The new ClusterController::API::notify_agents_ready event
declares the management infrastructure ready for use. zeek-client's CLI has
expanded to support the new functionality.
The framework is still experimental and provides only a small subset of
ZeekControl's functionality. ZeekControl remains the recommended tool for
maintaining your cluster.
- Entries in ``http.log`` now record the original ``HOST`` headers.
Previously, they would skip any port specification a header might
include.
Deprecated Functionality
------------------------
- The ``protocol_confirmation`` and ``protocol_violation`` events along with the
corresponding ``Analyzer::ProtocolConfirmation` and
``Analyzer::ProtocolViolation`` C++ methods are marked as deprecated. They are
replaced by ``analyzer_confirmation`` and ``analyzer_violation`` which can
also now be implemented in packet analyzers.
- Declaring a local variable in an inner scope and then accessing it in an
outer scope is now deprecated. For example,
if ( foo() )
{
local a = 5;
...
}
print a;
is deprecated. You can address the issue by hoisting the declaration
to the outer scope, such as:
local a: count;
if ( foo() )
{
a = 5;
...
}
print a;
Zeek 4.1.0
==========
New Functionality
-----------------
- Lambda functions can now use capture-list to help specify exactly which local
variables from outer scopes need to made available while evaluating the lambda
and also the method by which they're made available: deep vs. shallow copy.
For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function
- Support for VN-Tag protocol headers: the new VN-Tag packet analyzer simply
skips past the VN-Tag header, allowing for further analysis of subsequent
packet contents.
- Support for decapsulating Geneve packets to process the inner
payload, similar in operation to the existing VXLAN support.
- Support for Zeek script "Reaching Definitions" (RD) analysis: tracking the
extent to which a given variable definition (assignment) can be visible
elsewhere in the Zeek script. The analysis works on local variables in
function/event/hook bodies, but not across them.
The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a
given point in a function body) and "maximal" (what possibly could reach).
Upcoming script optimization will use the latter, but the former currently
allows identification of places for which a value is used where it
does not appear that it will necessarily be defined. Specifying the
``zeek -u`` option will generate warnings for instances where this holds for
local variables. Specifying ``zeek -uu`` turns on additional (expensive)
analysis to report instances where record fields might be used without
having previously been set.
The ``zeek -u`` option can also identify assigned-to variables that aren't
subsequently used (i.e. "dead code") and issues a warning. A new ``is_used``
attribute can be used situationally to suppress such warnings.
The base scripts have some places where the static analysis lacks sufficient
power to tell that values are being used safely (guaranteed to have been
identified). In order to enable users to employ ``zeek -u`` on their own
scripts without being distracted by these instances, this change also
includes a new attribute, ``&is_assigned``, which can be associated with a
variable or a record field to inform Zeek's analysis that the script writer
asserts the value will be set, suppressing the associated warnings.
- A Telemetry API was added to assist in gathering arbitrary runtime
metrics and allows export to Prometheus. This is still
work-in-progress, preliminary documentation for current, low-level
API lives at https://github.com/zeek/zeek/wiki/Telemetry for now.
- Experimental support for translating Zeek scripts to equivalent C++.
The generated C++ can then be compiled directly into the ``zeek`` binary,
replacing use of the interpreter and producing better runtime performance.
See ``src/script_opt/CPP/README.md`` for a guide on how to use this feature.
- Support for more generic session management. The NetSessions class has been
renamed to SessionMgr (with the old name marked deprecated). The new
class allows plugins to take advantage of session management similar to how
Connection objects were handled previously, but without the need to be based
on IP-based protocols.
- The logging framework now provides a global policy hook, ``Log::log_stream_policy``.
Like the existing filter-level hooks, handlers for the new hook can provide
additional processing and veto the log write. The new hook runs once per
write, prior to any filter-level policy hooks. Even when it vetoes,
filter-level policy hooks still run, but cannot "un-veto" the write.
- The ASCII writer gained a new option LogAscii::logdir, which can be used to
change the logging output directory.
- Added a ``--include-plugins`` argument to ``configure``. This argument
takes a semicolon separated list of paths containing plugins that will be
statically built into Zeek.
- Added a ``--plugindir`` argument to ``configure`` to set the
installation path for plugins.
- The X509 analyzer now can check if a specific hostname is valid for a
certificate. Two new BIFs were added for this, ``x509_check_hostname`` and
``x509_check_cert_hostname``. A new field ``sni_matches_cert`` that tracks
this information was added to ``ssl.log``.
- Added new functions to dynamically enable/disable file analyzers:
- ``global enable_analyzer: function(tag: Files::Tag): bool;``
- ``global disable_analyzer: function(tag: Files::Tag): bool;``
- ``global analyzer_enabled: function(tag: Files::Tag): bool;``
- Zeek now includes its own BTest tooling in the distribution, enabling other
tests (e.g. in Zeek packages) to use it. The ``$PREFIX/share/btest folder``,
reported via ``zeek-config --btest_tools_dir``, includes:
- ``scripts/`` for ``btest-diff`` canonifiers
- ``data/`` for data files, including ``random.seed``
- ``data/pcaps`` for the test pcaps
Configuring with ``--disable-btest-pcaps`` suppresses installation of the
test pcaps.
- The Supervisor now defaults to starting with a minimal set of Zeek
scripts controlled by a new init file, ``base/init-supervisor.zeek``.
One may still run it with a larger configuration by loading additional
scripts, including ``init-default.zeek``, as always. (Bare mode continues
to work as usual, reducing the configuration to a minimum.)
The ``NodeConfig`` record has two new members, providing additional
control over launched nodes. The ``env`` member allows setting environment
variables in the launched process. The ``bare_mode`` member, an optional
boolean, provides control over the bare-mode state of the new node.
When not provided, the node inherits the bare-mode status of the
Supervisor, and setting the variable enables/disables bare mode.
- Zeek now includes an incomplete, preliminary version of the future
cluster controller framework. Loading ``policy/frameworks/cluster/agent``
and/or ``policy/frameworks/cluster/agent`` in a Zeek running with the
Supervisor will launch the corresponding cluster management node(s).
An experimental management client, `zeek-client`, connects to the
controller and lets you issue commands. It requires configuration with
``--enable-zeek-client``. This does not yet provide a functional
substitute for ``zeekctl``, which users should continue to use for now.
Changed Functionality
---------------------
- The default IP-based transport protocols (UDP, TCP, and ICMP) have been
moved to the packet analysis framework. This change allows us to move other
analyzers in the future that better align with the packet analysis framework
than they do with session analysis.
- The input framework's handling of unset fields (commonly expressed as "-")
in ingested data is now safer and more consistent. When reading data into
records, Zeek now accepts unset fields in the input data only when the
corresponding record field is ``&optional``. Unset fields for non-optional
fields cause the input line to be skipped. Reading data into tables with complex
index types (such as ``table[count, count] of string``) now also skips lines
with unset fields, since such indexes require fields to be present.
Note that this may change the behavior of existing scripts if you have unset
fields in your input data.
- The version field in ssh.log is now optional and will not be set if we cannot
determine the version that was negotiated by the client and server.
- Add a new field ``email_dest`` to NOTICEs, which defines where to
send email to. The email-related NOTICE actions fill this now, and
then emails will be sent to all recorded addresses at the end of
NOTICE processing. This makes email generation more consistent and
extensible.
- Add page and email administrator to mails processed by hostnames extension.
- SSL and X509 handling was significantly overhauled with the goal to make
the data that is logged by Zeek more helpful and compact.
This change means that there are significant changes to the default log files,
as well as changes to functionality:
- ``x509.log`` is now indexed by the sha256 of the certificate, with deduplication
being automatically performed. By default, the same certificate is only logged
once per day.
This also means that the file ID is no longer present in X509 log. Similarly,
``ssl.log`` now contains hashes for X509 certificates.
The hash function that is used for indexing the certificates is changeable by
changing the ``X509::hash_function`` option.
The time period after which a certificate is logged again can be configured by
changing ``X509::relog_known_certificates_after``.
By default deduplication of certificates is done across the entire cluster using
broker. If this is not desired due to the higher communication overhead, this
behavior can be disabled using ``X509::known_log_certs_use_broker``.
- X509 certificates are, by default, no longer logged into files.log. This
behavior is configurable and the previous default can be restored by changing
the ``X509::log_x509_in_files_log`` option.
- ``x509.log`` now tracks if a certificate was encountered as a end-host certificate
or as a client certificate.
- OCSP logging is now enabled by default.
- ``ssl.log`` now no longer includes information about the certificate issuer and
subject. This information is still available in X509.log. If you need this
information in ``ssl.log``, the old behavior can be restored by changing the
``SSL::log_include_server_certificate_subject_issuer`` and
``SSL::log_include_client_certificate_subject_issuer`` configuration options.