-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathNEWS
4008 lines (3029 loc) · 166 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This document summarizes the most important changes in the current Zeek
release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as Broker, come with their own ``CHANGES``.)
Zeek 4.1.0
==========
New Functionality
-----------------
- Lambda functions can now use capture-list to help specify exactly which local
variables from outer scopes need to made available while evaluating the lambda
and also the method by which they're made available: deep vs. shallow copy.
For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function
- Support for VN-Tag protocol headers: the new VN-Tag packet analyzer simply
skips past the VN-Tag header, allowing for further analysis of subsequent
packet contents.
- Support for decapsulating Geneve packets to process the inner
payload, similar in operation to the existing VXLAN support.
- Support for Zeek script "Reaching Definitions" (RD) analysis: tracking the
extent to which a given variable definition (assignment) can be visible
elsewhere in the Zeek script. The analysis works on local variables in
function/event/hook bodies, but not across them.
The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a
given point in a function body) and "maximal" (what possibly could reach).
Upcoming script optimization will use the latter, but the former currently
allows identification of places for which a value is used where it
does not appear that it will necessarily be defined. Specifying the
``zeek -u`` option will generate warnings for instances where this holds for
local variables. Specifying ``zeek -uu`` turns on additional (expensive)
analysis to report instances where record fields might be used without
having previously been set.
The ``zeek -u`` option can also identify assigned-to variables that aren't
subsequently used (i.e. "dead code") and issues a warning. A new ``is_used``
attribute can be used situationally to suppress such warnings.
The base scripts have some places where the static analysis lacks sufficient
power to tell that values are being used safely (guaranteed to have been
identified). In order to enable users to employ ``zeek -u`` on their own
scripts without being distracted by these instances, this change also
includes a new attribute, ``&is_assigned``, which can be associated with a
variable or a record field to inform Zeek's analysis that the script writer
asserts the value will be set, suppressing the associated warnings.
- A Telemetry API was added to assist in gathering arbitrary runtime
metrics and allows export to Prometheus. This is still
work-in-progress, preliminary documentation for current, low-level
API lives at https://github.com/zeek/zeek/wiki/Telemetry for now.
- Experimental support for translating Zeek scripts to equivalent C++.
The generated C++ can then be compiled directly into the ``zeek`` binary,
replacing use of the interpreter and producing better runtime performance.
See ``src/script_opt/CPP/README.md`` for a guide on how to use this feature.
- Support for more generic session management. The NetSessions class has been
renamed to SessionMgr (with the old name marked deprecated). The new
class allows plugins to take advantage of session management similar to how
Connection objects were handled previously, but without the need to be based
on IP-based protocols.
- The logging framework now provides a global policy hook, ``Log::log_stream_policy``.
Like the existing filter-level hooks, handlers for the new hook can provide
additional processing and veto the log write. The new hook runs once per
write, prior to any filter-level policy hooks. Even when it vetoes,
filter-level policy hooks still run, but cannot "un-veto" the write.
- The ASCII writer gained a new option LogAscii::logdir, which can be used to
change the logging output directory.
- Added a ``--include-plugins`` argument to ``configure``. This argument
takes a semicolon separated list of paths containing plugins that will be
statically built into Zeek.
- Added a ``--plugindir`` argument to ``configure`` to set the
installation path for plugins.
- The X509 analyzer now can check if a specific hostname is valid for a
certificate. Two new BIFs were added for this, ``x509_check_hostname`` and
``x509_check_cert_hostname``. A new field ``sni_matches_cert`` that tracks
this information was added to ``ssl.log``.
- Added new functions to dynamically enable/disable file analyzers:
- ``global enable_analyzer: function(tag: Files::Tag): bool;``
- ``global disable_analyzer: function(tag: Files::Tag): bool;``
- ``global analyzer_enabled: function(tag: Files::Tag): bool;``
- Zeek now includes its own BTest tooling in the distribution, enabling other
tests (e.g. in Zeek packages) to use it. The ``$PREFIX/share/btest folder``,
reported via ``zeek-config --btest_tools_dir``, includes:
- ``scripts/`` for ``btest-diff`` canonifiers
- ``data/`` for data files, including ``random.seed``
- ``data/pcaps`` for the test pcaps
Configuring with ``--disable-btest-pcaps`` suppresses installation of the
test pcaps.
- The Supervisor now defaults to starting with a minimal set of Zeek
scripts controlled by a new init file, ``base/init-supervisor.zeek``.
One may still run it with a larger configuration by loading additional
scripts, including ``init-default.zeek``, as always. (Bare mode continues
to work as usual, reducing the configuration to a minimum.)
The ``NodeConfig`` record has two new members, providing additional
control over launched nodes. The ``env`` member allows setting environment
variables in the launched process. The ``bare_mode`` member, an optional
boolean, provides control over the bare-mode state of the new node.
When not provided, the node inherits the bare-mode status of the
Supervisor, and setting the variable enables/disables bare mode.
- Zeek now includes an incomplete, preliminary version of the future
cluster controller framework. Loading ``policy/frameworks/cluster/agent``
and/or ``policy/frameworks/cluster/agent`` in a Zeek running with the
Supervisor will launch the corresponding cluster management node(s).
An experimental management client, `zeek-client`, connects to the
controller and lets you issue commands. It requires configuration with
``--enable-zeek-client``. This does not yet provide a functional
substitute for ``zeekctl``, which users should continue to use for now.
Changed Functionality
---------------------
- The default IP-based transport protocols (UDP, TCP, and ICMP) have been
moved to the packet analysis framework. This change allows us to move other
analyzers in the future that better align with the packet analysis framework
than they do with session analysis.
- The input framework's handling of unset fields (commonly expressed as "-")
in ingested data is now safer and more consistent. When reading data into
records, Zeek now accepts unset fields in the input data only when the
corresponding record field is ``&optional``. Unset fields for non-optional
fields cause the input line to be skipped. Reading data into tables with complex
index types (such as ``table[count, count] of string``) now also skips lines
with unset fields, since such indexes require fields to be present.
Note that this may change the behavior of existing scripts if you have unset
fields in your input data.
- The version field in ssh.log is now optional and will not be set if we cannot
determine the version that was negotiated by the client and server.
- Add a new field ``email_dest`` to NOTICEs, which defines where to
send email to. The email-related NOTICE actions fill this now, and
then emails will be sent to all recorded addresses at the end of
NOTICE processing. This makes email generation more consistent and
extensible.
- Add page and email administrator to mails processed by hostnames extension.
- SSL and X509 handling was significantly overhauled with the goal to make
the data that is logged by Zeek more helpful and compact.
This change means that there are significant changes to the default log files,
as well as changes to functionality:
- ``x509.log`` is now indexed by the sha256 of the certificate, with deduplication
being automatically performed. By default, the same certificate is only logged
once per day.
This also means that the file ID is no longer present in X509 log. Similarly,
``ssl.log`` now contains hashes for X509 certificates.
The hash function that is used for indexing the certificates is changeable by
changing the ``X509::hash_function`` option.
The time period after which a certificate is logged again can be configured by
changing ``X509::relog_known_certificates_after``.
By default deduplication of certificates is done across the entire cluster using
broker. If this is not desired due to the higher communication overhead, this
behavior can be disabled using ``X509::known_log_certs_use_broker``.
- X509 certificates are, by default, no longer logged into files.log. This
behavior is configurable and the previous default can be restored by changing
the ``X509::log_x509_in_files_log`` option.
- ``x509.log`` now tracks if a certificate was encountered as a end-host certificate
or as a client certificate.
- OCSP logging is now enabled by default.
- ``ssl.log`` now no longer includes information about the certificate issuer and
subject. This information is still available in X509.log. If you need this
information in ``ssl.log``, the old behavior can be restored by changing the
``SSL::log_include_server_certificate_subject_issuer`` and
``SSL::log_include_client_certificate_subject_issuer`` configuration options.
- ``ssl.log`` now contains a ``ssl_history`` field, which tracks which protocol
messages were seen in an SSL/TLS connection.
- We added a policy script ``ssl-log-ext.zeek`` which greatly extends the amount
of protocol information logged to SSL.log. The script is not loaded by default.
- We added a ``disable-certificate-events-known-certs.zeek`` policy script. This script
will completely disable X509 events for known certificates over SSL/TLS connections.
For Zeek installations in settings where you encounter a lot of certificates,
this could improve the performance of your installation. Before enabling this
script, make sure that you do not use any third-party scripts that depend on the
X509 events. The script is not loaded by default.
- The ICSI SSL Notary script was deprecated. This functionality is superseeded by newer
approaches, like SCT validation (which is supported by Zeek).
- ``extract-certs-pem.zeek`` was deprecated - it never really worked in cluster modes.
A new policy script, ``log-certs-base64.zeek`` that can be used to log raw certificates
was added instead.
- The CT logs listed in ``ct-list.zeek`` are now derived from the list of CT
logs that are accepted by Google Chrome. In the past, we allowed the list
of all known CT logs. This no longer makes sense since nowadays logs exist that
contain, e.g., only outdated or invalid certificates. If the old behavior is
desired, you can re-add Logs to ``SSL::ct_logs``.
- The Mozilla CA list was updated to the state of NSS 3.67.
- SQLite was updated to 3.36.0.
- The list of subnets that are considered private addresses was updated to mostly
match IANA's list unroutable addresses. This brings it in line with Chrome, the
W3C spec, and Python's ipaddress module.
Removed Functionality
---------------------
- Support for the RocksDB Broker data store was previously broken and unusable,
so all code/options related to it are now removed.
- Support for the ``ENABLE_MOBILE_IPV6`` compiler variable has been removed. Mobile
IPv6 is now enabled by default. The ``--enable-mobile-ipv6`` option for
``configure`` now returns a warning that it will be removed in v5.1 and no
longer has any effect.
Deprecated Functionality
------------------------
- Lambda/closure support: automatic capturing of references to variables
outside a lambda's scope is now deprecated. An explicit capture
list which also specifies the desired copy-semantics is now required when
writing lambda functions that refer to local variables of an outer scope.
For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function
- The ``IterCookie`` version of iteration over ``Dictionary`` and ``PDict``
objects was marked as deprecated. It was replaced by standard-library
compatible iterators. This enables the use of standard constructs such
as ranged-for loops to iterate over those objects.
- The ``zeek::util::zeekenv()`` function is deprecated since use of all
environment variables prefixed by ``BRO_`` is now removed and calling
``getenv()`` directly with ``ZEEK_`` environment variables can be done.
- ``supervisor_rotation_format_func`` is renamed to ``archiver_rotation_format_func``
- The ``MemoryAllocation()`` function implemented by a number of interfaces
is now deprecated. In testing we found that the values returned were mostly
incorrect and weren't useful. The ``val_size`` and ``global_sizes`` BIF
methods have also both been marked deprecated.
Zeek 4.0.0
==========
New Functionality
-----------------
- Added support for EDNS0 Cookie and Keep-Alive options.
- Added new Packet Analysis plugin architecture for parsing packet headers
at layers below the existing Session analysis plugins. This allows
writing plugins to parse the various parts of a packet header separately,
chaining down into other plugins as needed.
- Add ``dce_rpc_request_stub`` and ``dce_rpc_response_stub`` events for
accessing the contents of DCE-RPC request/response stub data.
- Add support for log filter policy hooks, which supersede the current
log predicates. The hook signature is as follows:
hook(rec: any, id: Log::ID, filter: Log::Filter);
The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook
as needed. The distribution's existing log streams now come with
pre-defined hooks that users can add handlers to. The existing
predicates are deprecated for removal in 4.1 but continue to work.
See https://docs.zeek.org/en/master/frameworks/logging.html#filter-log-records
for more details.
- Added a ``udp-state`` signature condition to enforce matching against
either "originator" or "responder" flow direction of UDP packets.
- Improvements to capture-loss.zeek:
- A new option, ``CaptureLoss::initial_watch_interval``. When restarting a
Zeek cluster, one usually wants some immediate feedback as to the health of
the monitoring via capture loss. However, you previously needed to wait a
full ``CaptureLoss::watch_interval``, which defaults to 15 minutes. The
new option specifies the interval for the first-time report. So the new
default behavior provides stats after 1 minute and then after
15 minutes afterward.
- A new notice type, ``CaptureLoss::Too_Little_Traffic``.
If a Zeek process sees less than ``CaptureLoss::minimum_acks`` ACKs in a
given interval, this notice gets raised. This can be a useful diagnostic
if, for whatever reason, a Zeek process stops seeing traffic, but
capture-loss.zeek would have previously only reported that "0 gaps and 0
ACKs is 0% loss".
- A new ``zeek_script_args`` variable contains a list of arguments passed
to a script. E.g. either when explicitly executing Zeek like
``zeek -- myscript.zeek -arg1 -arg2``, or when using Zeek to interpret
executable scripts that contain a hashbang line at the top like::
#!/usr/local/zeek/bin/zeek --
- Added a new ``generate_all_events`` bif, which can be used to always raise
events, even when they are not used by scripts. This can be used by the
``dump-events.zeek`` script to log all events that happen; the script
got a new option to enable this behavior.
- Added new unknown_protocols.log that will log analyzer and protocol pairs
via the packet analysis framework for packet protocols that aren't
supported by Zeek. It can be enabled by loading the
``policy/misc/unknown-protocols`` script. The script adds a new
``unknown_protocol`` event.
- Added support for DNS resource records LOC, SSHFP, NSEC3PARAM, and custom
BIND9 signaling. The associated events are:
- dns_LOC
- dns_SSHFP
- dns_NSEC3PARAM
- dns_BINDS
- Zeek now supports SSH clients/servers that advertise SSH version 1.99, which
is a special version indicating that the server/client supports both SSH2 and
SSH1.
- Added ``count_to_double()`` and ``int_to_double()`` type-conversion BIFs.
- Added these string-processing BIFs:
- count_substr
- find_str
- rfind_str
- starts_with
- ends_with
- is_num
- is_alpha
- is_alnum
- ljust
- rjust
- swap_case
- to_title
- zfill
- remove_prefix
- remove_suffix
- Added a new ``Weird::sampling_global_list`` option to configure global
rate-limiting of certain weirds instead of per connection/flow.
- Added a ``Pcap::findalldevs()`` for obtaining available network devices.
- Added ``enum_names()`` BIF to return names of an enum type's values
- Added ``type_aliases`` BIF for introspecting type-names of types/values
- Added composite-index support for ``&backend`` (Broker-backed tables).
An example of a set with composite index is ``set[string, count, count]``.
- Sumstats now allows manual epochs. If an ``epoch`` interval of 0 is specified,
epochs will have to be manually ended by callis ``SumStats::next_epoch``. This
can be convenient because epochs can be synced to other events.
- The Zeek distribution now includes Zeek's package manager, zkg. Its
code, configuration, and state reside in Zeek's installation tree,
as follows:
- The toplevel script, ``zkg``, installs alongside ``zeek`` in the
distribution's ``$prefix/bin`` folder.
- The config file installs into ``$prefix/etc/zkg/config``. The
distribution's zkg command uses it by default, but you can switch
to a different one via the ``ZKG_CONFIG_FILE`` environment
variable or the ``--configfile`` command-line flag.
- zkg's package state resides in ``$prefix/var/lib/zkg``. This
implies that parallel Zeek installations now automatically
separate their package installations.
These folders have the same ownership and access permissions as the
rest of the installation, , meaning that in order to manage zkg
packages you need to run zkg as a user with corresponding access.
Apart from these location overrides, the bundled zkg installation
behaves as usual.
local.zeek now contains a (commented out) ``@load`` statement you
can use to source zkg's package state automatically.
zkg's own Python module resides in ``zeek/python/zeekpkg`, in the
installation tree's library folder. See below for additional changes
around the library folder.
zkg has external Python module dependencies. The Zeek configuration
does not verify whether these dependencies are met. A new warning
message at zkg launch flags missing packages and how to install them
(e.g. via pip).
Configuring with ``--disable-zkg`` disables the zkg inclusion. You
can continue to install and use zkg independently. You're also free
to use the config file in ``$prefix/etc/zkg/config`` with other zkg
installations.
The zkg source tree resides in ``auxil/package-manager`` as an
additional Git submodule.
- Added a new ``ssl_probable_encrypted_handshake_message`` event, which
is raised for encrypted TLS 1.3 handshake messages.
Changed Functionality
---------------------
- ``NetControl::DROP`` had 3 conflicting definitions that could potentially
be used incorrectly without any warnings or type-checking errors.
Such enum redefinition conflicts are now caught and treated as errors,
so the ``NetControl::DROP`` enums had to be renamed:
- The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG``
- The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to
``NetControl::DROP_REQUESTED``
- The use as enum of type ``NetControl::RuleType`` is unchanged and still
named ``NetControl::DROP``
- The extract_email_addrs_vec() BIF now returns all occurrences of emails,
including duplicates, with preserved order of occurrence. This seems like
the original/documented intent of the function, but the previous
implementation did not preserve ordering or duplicates.
- The Dictionary implementation is replaced (no API changes). The new version
uses clustered hashing, a variation of Robinhood / Open Addressing hashing.
This implementation generally performs better and utilizes less memory
than the previous one. A detailed explanation of the implementation is here:
https://jasonlue.github.io/algo/2019/08/20/clustered-hashing.html
- The ``p`` fields of ``Cluster::Node`` records now use a
``&default=0/unknown`` attribute with ``0/unknown`` meaning that the node is
not pre-configured to listen for incoming connections from other cluster
nodes.
- The ``|x|`` operator, where ``x`` is an expression with an integral result,
no longer performs an implicit coercion of that result into a signed
``int`` type. This was actually the behavior before Zeek 3.0 as well, but
the attempt to prevent mistakes that easily result from integer literals in
Zeek being unsigned like ``|5 - 9|`` causing an overflow/wraparound and
yielding a very large number is not generally consistent since overflows
are still generally able to happen in other ways and also in other contexts
besides just the absolute-value operator. So the preference was to revert
to a behavior that favors consistency. For reference, see
https://github.com/zeek/zeek/pull/251#issuecomment-713956976
- The Zeek installation tree is now more consistent in using a ``lib64/``
(rather than ``lib/``) subdirectory for platforms where that's the common
convention. If the old hardcoded ``lib/`` path exists while installing Zeek
4.0 and the new subdirectory differs, the old ``lib/`` remains untouched.
This clutters the installation but is safe: the new installation does not
require the old location, and any files you might require still in the old
tree (e.g. ZeekControl plugins) remain available.
Due to Zeek 4's reorganization of the installation tree we recommend
a clean-slate install when possible.
- Python modules installed with the Zeek distribution now reside in a
common ``zeek/python`` directory below the library path (such as
``lib64/zeek/python``) and no longer assume ZeekControl. The
``zeek/python/zeekctl`` folder now contains only ZeekControl's own
functionality, ``zeek/python/zeekpkg`` contains zkg's Python module, and
Broker's Python bindings live in ``zeek/python/broker``. ``zeek-config
--python_dir`` now reports this new ``zeek/python`` folder. Several
new configure options allow you to customize the Python folder location,
depending on your needs.
As with the new libdir, no cleanup of the existing Python tree occurs.
- Continued renaming/namespacing of many classes into either ``zeek`` or
``zeek::detail`` namespaces as already explained in Zeek 3.2's release notes.
Deprecation warnings should generally help notify plugin developers of these
changes.
- Changed HTTP DPD signatures to trigger analyzer independent of peer state.
This is to avoid missing large sessions where a single side exceeds
the DPD buffer size. It comes with the trade-off that now the analyzer
can be triggered by anybody controlling one of the endpoints (instead
of both). For discussion, see https://github.com/zeek/zeek/issues/343.
- TLS 1.3 support was improved in several ways:
* In the past, some TLS 1.3 sessions were misidentified as using session
resumption when, in fact, they were not resumed. This was caused by
the TLS session ID which no longer has any meaning in TLS 1.3. This was
fixed.
* Similarly, in the past, TLS 1.3 sessions that use TLS 1.3 PSKs for
session resumption were not marked as resumed. This also was fixed.
* The way in which session establishment for TLS 1.3 is performed was
rewritten. This causes the ``ssl_encrypted_data`` event to be correctly
raised; in the past this did not work for some sessions. A new
``ssl_probable_encrypted_handshake_message`` event was added that is
raised for encrypted TLS 1.3 handshake packets.
* In the same vein, hello retry requests in TLS 1.3 should now always
be handled correctly; in the past this only happened in some cases.
Please note: When a connection uses Hello Retry requests you will see
two client hello and two server hello events in a single connection.
This happened in the past, but may become more common now; this might
trigger unexpected behavior in your scripts.
Removed Functionality
---------------------
- The counter type was removed. This type was never fully functional/used
anywhere.
- Removed the PRI_PTR_COMPAT_INT, PRI_PTR_COMPAT_UINT, and PRI_SOURCE_ID
macros. There are no deprecation warnings for these because they were C
macros. Use the PRIdPTR and PRIuPTR macros from the standard library
instead.
- The ``successful_connection_remove`` and ``connection_successful`` events
as well as ``connection$successful`` field that were added in Zeek v3.1.0 are
now removed. They were found to be unintuitive along with having unresolved
corner cases. The original goal/intent to avoid the overhead and scalability
issues with every new protocol-analysis adding a new
``connection_state_remove`` handler can now be resolved with a less-confusing
approach: see the ``Conn::register_removal_hook`` function.
- Python 2 is no longer supported. Python 3.5 is the new minimum requirement.
- CMake versions less than 3.5 are no longer supported.
- CAF version 0.18 is now required and, by default, that is bundled with
the Zeek distribution and will get built unless overridden with the
``--with-caf=`` configuration option.
- ``server_appdata`` and ``client_appdata`` were removed from ``SSL::Info``.
These variables were only used internally, and did not give a correct counts
in all circumstances.
Deprecated Functionality
------------------------
- Marked the Continuation.h and PacketDumper.h files as deprecated. The code
contained within them is unused by Zeek.
- ``Type::GetAliases()`` and ``Type::AddAlias()`` are deprecated, use
``Type::Aliases()`` and ``Type::RegisterAlias()``.
- The ``ssh1_server_host_key`` event's modulus and exponent parameters,
*e* and *p*, were named in misleading way (*e* is the modulus)
and now deprecated in favor of the new *modulus* and *exponent* parameters.
Zeek 3.2.0
==========
New Functionality
-----------------
- X509 Certificate caching:
Zeek now caches certificates if they have (by default) been encountered
more than 10 times in 62 seconds. Information for cached certificates is
retained; if the certificate is encountered again it does not have to
be re-parsed and already existing information is used to raise the events.
This should especially help with performance in environments where the
same certificates are seen very often.
Certificate caching is very configureable; it is possible to disable the
feature, change the time intervals or even suppress X509 events.
For details see ``scripts/base/files/x509/main.zeek``.
- Add parsing support for Remote Desktop Protocol UDP Transport Extension
(RDPEUDP versions 1 and 2). This primarily only adds "rdpeudp" to
connection record service fields when an RDPEUDP session handhake is
detected, but also provides a few other events related to the RDPEUDP
connection establishment.
- Add the ``udp_content_ports`` configuration option. Any port added to
this set will cause the ``udp_contents`` event to be raised.
- Add the ``udp_content_delivery_ports_use_resp`` option which can be used
to specify how the destination port for the ``udp_content_delivery_ports_orig``
and ``udp_content_delivery_ports_orig`` options is determined. The current value
keeps behavior as it was in previous versions of Zeek.
- Add a file signature to identify ISO9660 disk images (application/x-iso9660-image)
- Add file signature to identify Python bytecode (application/x-python-bytecode)
- Events and hooks are now allowed to have multiple, alternate prototype
declarations. This allows for extending event/hook parameters in a way that
won't break an existing user's handlers and also allows users to define their
own custom event/hook prototypes that consume a subset of the parameters
(convenience of typing/memory/etc). This feature is documented in detail
here: https://docs.zeek.org/en/current/script-reference/types.html#type-event
- Add ``flags`` parameters to ``rdp_connect_request``,
``rdp_negotiation_response``, and ``rdp_negotiation_failure`` events.
- ``Reporter::conn_weird`` now correctly handles weirds for expired connections,
for which no connection state information is available in the core anymore. These
cases will raise the new ``expired_conn_weird`` event.
- Broker Store table synchronization (experimental).
Zeek now supports synchronizing tables/sets across clusters using a backing Broker
store. The same feature also allows persistent storage of data in tables/sets
over Zeek restarts. This feature is implemented using the new ``&backend`` attribute.
To synchronize a table over a cluster, you can, e.g., use:
global t: table[string] of count &backend=Broker::MEMORY;
This feature is documented in detail here:
https://docs.zeek.org/en/current/frameworks/broker.html#broker-store-backed-zeek-tables-for-data-synchronization-and-persistence
Note: this feature is experimental and the syntax/specifics can change in the future.
Changed Functionality
---------------------
- Several C++ functions have been changed to pass smart pointers
(``class IntrusivePtr<>``) instead of raw pointers. This makes the
code more robust. External plugins may need to be updated to this
API change.
- BIFs that use @ARG@, @ARGS@, or @ARGC@ may break since their type has
changed: BIF arguments are now passed as a ``std::vector<IntrusivePtr<Val>>``
rather than a ``val_list`` (i.e. ``List<Val*>``).
- RocksDB support for Broker data stores is now opt-in instead of automatically
detected and used at configuration-time. Use the ``--enable-rocksdb`` and
``--with-rocksdb=`` flags to opt-in.
- At the C++ level, a large number of functions had their return
values and/or arguments changed to use ``bool`` types instead of
``int``. This includes some virtual methods, which may cause build
failures in plugins that were overriding those methods. Those
plugins will need to be updated to match these API changes. To make
sure to catch changes you need to make to your plugin, double-check
that all virtual method implementations use "override".
- Many C++ classes were marked "final" which also has some performance benefits
due to devirtualization optimizations.
- Data members of many C++ classes/structs were reordered to achieve better
packing and smaller memory footprint.
- "using namespace std" was removed from the Zeek header files; Zeek now always
explicitly specifies std when using STL functionality in headers. This may
necessitate small changes in external plugins, if they relied on the using
statement in Zeek headers.
- The ``connection_external`` event was removed. This functionality that could
raise this event (injecting connections via broccoli) was removed a while ago;
the event handler served no purpose anymore.
- Reorganize the file signatures to break them out into more groups. This may
break scripts that had been explicitly loading any signature files that moved.
- The DCE/RPC operation string of "NetrLogonSamLogonWithFlags" has been
corrected from "NetrLogonSameLogonWithFlags".
- ``AsRecord()`` and ``AsNonConstRecord()`` have changed to return
``std::vector<IntrusivePtr<Val>>*``.
- ``AsVector()`` has changed to return ``std::vector<IntrusivePtr<Val>>*``.
- Moved a large number of classes from the global namespace into either the
``zeek`` or ``zeek::detail`` namespace. See https://github.com/zeek/zeek/issues/266
for the rationale behind these changes. Most types that were moved and functions
that used them have been marked as deprecated and will generate compiler
warnings if used (a few exceptions will not generate compiler warnings,
but the Deprecated Functionality section below will mention those
ones specifically).
This includes a number of renames to classes, removing 'Bro' from their
names. Deprecation warnings should notify plugin developers of these
name changes.
Removed Functionality
---------------------
- The fmt() function which takes a va_list argument is replaced, use
the new vfmt() function for equivalent functionality. The former is
deprecated because overloading it with the variadic fmt() function
can cause the unintended overload to be chosen depending on how the
platform implements va_list.
Deprecated Functionality
------------------------
- The ``plugin::Plugin::HookCallFunction()`` method is deprecated. Note
that compilers will not emit a deprecation warning, but the replacement
method to now use is called ``HookFunctionCall`` and uses ``IntrusivePtr``
arguments and return value.
- The ``Func::Call(val_list*, ...)`` method is now deprecated. Use ``Invoke()``
instead which takes a ``zeek::Args`` (``std::vector<IntrusivePtr<Val>>``).
There's also a variadic template for ``Invoke()`` that forwards all arguments
into a ``zeek::Args`` for you.
- The ``EventMgr::QueueEvent()`` and EventMgr::QueueEventFast()`` methods
are now deprecated, use ``EventMgr::Enqueue()`` instead.
- The ``Connection::ConnectionEvent()``, ``Connection::Event()``, and
``Connection::ConnectionEventFast()`` methods are now deprecated, use
``Connection::EnqueueEvent()`` instead.
- The ``file_analysis::File::FileEvent()`` methods taking ``val_list``
arguments are now deprecated, use the overload that takes a ``zeek::Args``
instead.
- The ``analyzer::Analyzer::ConnectionEvent()``, ``analyzer::Analyzer::Event``,
and ``analyzer::Analyzer::ConectionEventFast()`` methods are deprecated, use
``analyzer::Analyzer::EnqueueConnEvent()`` instead.
- All ``val_mgr`` methods starting with "Get" are deprecated, use the new
``val_mgr`` methods that return ``IntrusivePtr``.
- ``Connection::BuildConnVal()`` is deprecated, use ``Connection::ConnVal()``.
- ``Analyzer::BuildConnVal()`` is deprecated, use ``Analyzer::ConnVal()``.
- ``BifEvent::generate_`` functions are deprecated, use ``zeek::BifEvent::enqueue_``.
- ``binpac::bytestring_to_val()`` is deprecated, use ``binpac::to_stringval()``.
- ``binpac::string_to_val()`` is deprecated, use ``StringVal`` constructor.
- Returning ``Val*`` from BIFs is deprecated, return ``IntrusivePtr`` instead.
- Various methods of converting protocol structures, like IP or packet headers,
to associated ``Val`` type are now deprecated, the deprecation warning
message will advise what new method to use instead.
- Various methods of ``Tag`` classes are deprecated with the warning
message advising what new method to use instead.
- The ``utf16_bytestring_to_utf8_val()`` function is deprecated, use
``utf16_to_utf8_val()`` instead.
- ``RecordType::FieldType()`` is deprecated, use ``RecordType::GetFieldType()``
- ``BroType::HasField()`` and ``BroType::FieldType()`` are deprecated, use
the methods of ``RecordType`` directly.
- ``BroType::YieldType()`` is deprecated, use ``BroType::Yield()``.
- ``ID::AsType()`` is deprecated, use ``ID::IsType()`` and ``ID::GetType()``.
- ``ID::Type()`` is deprecated, use ``ID::GetType()``.
- ``ID::ID_Val()`` is deprecated, use ``ID::GetVal()``.
- ``internal_type()`` is deprecated, use ``zeek::id::find_type()``.
- ``internal_val()`` and ``internal_const_val()`` are deprecated, use
``zeek::id::find_val()`` or ``zeek::id::find_const()``.
- ``internal_func()`` is deprecated, use ``zeek::id::find_func()``.
- ``opt_internal_val()`` is deprecated, use ``lookup_ID()`` or
``zeek::id::find_val()``.
- ``Val::Type()`` is deprecated, use ``Val::GetType``.
- Most global type/value pointers in NetVar.h are deprecated, but one can
still always perform the lookup themselves.
- ``Scope::Lookup()`` is deprecated, use ``Scope::Find()``.
- All generated ``BroType*`` names in the ``BifType::`` namespaces are
deprecated, but there's an equivalent name in ``zeek::BifType::`` of
``IntrusivePtr`` type to use instead.
- All generated ``BifConst::`` names are deprecated, but there's an
equivalent name now in ``zeek::BifCont::``, and changed to ``IntrusivePtr``
if the old name was some ``Val*`` type.
- Constructors for ``Val`` types that take a ``BroType*`` are all generally
deprecated, with alternatives that instead take an ``IntrusivePtr`` argument.
- ``FuncType::Args()`` is deprecated, use ``FuncType::Params()``.
- ``FuncType::ArgTypes()`` is deprecated, use ``FuncType::ParamList()``.
- ``RecordVal::Assign(int, Val*)`` is deprecated, use the overload taking
``IntrusivePtr``.
- ``RecordVal::Lookup(int)`` is deprecated, use ``RecordVal::GetField(int)``.
- ``RecordVal::LookupWithDefault(int)`` is deprecated, use
``RecordVal::GetFieldOrDefault(int)``.
- ``RecordVal::Lookup(const char*, bool)`` is deprecated, use either
``RecordVal::GetField()`` or ``RecordVal::GetFieldOrDefault()``.
- ``TableVal::Assign`` methods taking raw ``Val*`` are deprecated, use the
overloads taking ``IntrusivePtr``.
- ``TableVal::Lookup()`` is deprecated, use ``TableVal::Find()`` or
``TableVal::FindOrDefault()``.
- ``VectorVal::Assign`` and ``Insert`` methods taking raw ``Val*`` are
deprecated, use the methods that take ``IntrusivePtr``.
- ``VectorVal::Lookup()`` is deprecated, use ``VectorVal::At()``.
- The file analysis/analyzer API has deprecated methods taking raw
``RecordVal*`` for analyzer arguments and replaced those with methods
taking ``IntrusivePtr``.
- The ``Val(double, TypeTag)`` constructor is deprecated, use either
``IntervalVal()``, ``TimeVal()`` or ``DoubleVal()`` constructors.
- The "BroString.h" file is deprecated, use "ZeekString.h"
- The str_split() BIF is deprecated, use str_split_indices(). Note
that the former returns a vector with indices starting at 1 while the
later returns a vector with indices starting at 0.
- The ``icmp_conn`` parameter of ICMP events is deprecated, there's an
alternate version with an ``icmp_info`` parameter to use instead.
The ``icmp_conn`` record passed to ICMP events has always been re-used
amongst all events within an ICMP "connection", so the
``itype``, ``icode``, ``len``, and ``hlim`` fields as inspected in
handlers never appears to change even if the underlying packet data
has different values for those fields. However, it's not known if
anyone relied on that behavior, so the new ``icmp_info`` record is
introduced with the more-expected behavior of being created and
populated for each new event. It also removes the orig_h/resp_h
fields since those are redundant with what's already available in
the connection parameter.
- External plugins should include Zeek header files like
``#include <zeek/Foo.h>`` instead of ``#include <Foo.h>``. The later
style is considered deprecated. Reliance on ``zeek-config --include_dir``
to contain ``$prefix/include/zeek`` is also deprecated: its replacement
output is expected to be just ``$prefix/include``, with it currently
outputting both paths, delimited by a colon, during the deprecation period.
Zeek 3.1.0
==========
New Functionality
-----------------
- Add a new supervisor framework that enables Zeek to operate clusters
of processes itself without any external help.
The Supervisor framework provides an entirely new deployment mode
for Zeek, one that supervises a set of Zeek processes that are meant
to be persistent. A Supervisor automatically revives any process
that dies or exits prematurely and also arranges for an ordered
shutdown of the entire process tree upon its own termination. This
Supervisor mode for Zeek provides the basic foundation for process
configuration/management that could be used to deploy a Zeek cluster
similar to what ZeekControl does, but is also simpler to integrate
as a standard system service.
This mode is still experimental and will evolve over time. The
command-line argument of ``-j`` toggles Zeek to run in "Supervisor
mode" to allow for creation and management of child processes. If
you're going to test this, please note that you will need some
custom script code to configure the processes you want Zeek to run.
See the documentation for more information:
https://docs.zeek.org/en/stable/frameworks/supervisor.html
- Add a new option, ``dpd_late_match_stop``, which can be used in conjuction
with the option ``dpd_match_only_beginning`` and the new event
``protocol_late_match`` to help annotate the conn.log with a field
to speculate on the protocol/service in cases where the DPD buffer
was already exhausted and can't analyze the full connection anymore,
but where there was still a late signature match. A new script,
``policy/protocols/conn/speculative-service.zeek``, was added as an
example of how to perform this tuning and add a "speculative_service"
field to conn.log, but it's not loaded by default.
- There is now a new ``tcp_options`` event that is raised for each TCP header
that contains options.
- Added a new option, ``Log::print_to_log`` that can be set to automatically
redirect the output from "print" statements to a real log stream (e.g.
instead of writing to stdout).
- There is now a new ``&on_change`` attribute that can be used to be notified
of changes to tables and sets.
Changed Functionality
---------------------
- A C++17-capable compiler and CMake 3.0+ are now required to compile Zeek
- The backwards-compability wrappers & work-arounds introduced in 3.0
for the "Bro to Zeek rename" have either changed their operation, or in some
cases been removed. Generally, anything that reported a
naming-related warning in 3.0 now aborts with a corresponding error
message. In cases where 3.0 silently continued to accept old names,
3.1 now reports warnings. Most importantly, that's loading of
scripts with ".bro" endings, which are now flagged and should be
renamed.
- Broker has switched versions for the underlying CAF communication
library from 0.16 to 0.17. CAF has changed its wireformat between
those versions, which means that previous Broker and Zeek versions
won't be able to connect to the new ones. In other words, all Zeek
instances, as well as other Broker clients, need to upgrade at the
same time. In case of version mismatches, Broker now reports better
error messages to point out the problem.
- The key type of ``Known::service_store`` has changed to
``Known::AddrPortServTriplet`` and ``Known::services`` is now a table
instead of just a set.
- The DNS class name for Hesiod in the ``DNS::classes`` table is now spelled
correctly as "C_HESIOD" instead of "C_HESOD". For reference, this
class name string may appear in the dns.log file or in any custom
script logic that inspects the ``qclass`` field of ``DNS::Info``
after a ``dns_request`` event.
- The configuration input reader now ignores trailing spaces at the end of
configuration lines.
- The tcp_option event is now correctly raised.
- The base scripts shipped with Zeek now use the new
``successful_connection_remove`` event instead of
``connection_state_remove`` where possible (when the logic doesn't
pertain to unestablished TCP connections). There's a performance
benefit to this switch, but it may potentially break custom scripts
that depended on accessing or modifying state via specific
``&priority`` ordering of ``connection_state_remove`` event
handlers. However, since most of Zeek's base scripts previously just
used that event with the intention of writing a finalized log as the
last thing executed for a given connection, and the new
``successful_connection_remove`` event handlers all run after
``connection_state_remove``, it's not likely this change to the base
scripts causes any incompatibility with user scripts.
There's also a new event called ``connection_successful`` and a new
``connection`` record field named "successful" to help indicate this
new property of connections.
- The JSON output formatters now use the RapidJSON library. This
improves their performance considerably over the library that was
previously used. Output from the formatters remains nearly
identical.
- The ``decompose_uri`` function no longer raises an error when parsing
URIs with an empty port number (e.g. ``http://example.org:/``). Instead,
the ``portnum`` component of the returned ``URI`` value is left
uninitialized.
- Replace old ``select``-based IO loop with a new architecture that doesn't
spin checking for active IO sources. The new architecture now waits for the
the sources to actively notify it when activity occurs and only processes
data once it's ready. This helps heavily reduce the CPU usage on idle
network connections. This includes a couple of breaking changes:
- Only a single packet source is allowed to be specified from the
command-line now. If you pass combinations of multiple ``-r`` and/or
``-i`` flags, Zeek will return an error at startup.
- The IOSource API changed fairly wildly. The ``GetFds()`` and
``NextTimestamp`` methods no longer exist. If you had previously
implemented a custom IOSource, you will need to look at the new API
and make changes to your code to accomodate it. This does not include
packet sources, which should remain functional with little to no
changes, since the entirety of the changes should be in ``PktSrc``.
- Remove a large number of headers from being included by various files across
the entire code base, which leads to a sizeable build time improvement. This
set of changes has the potential to cause plugins to not build anymore. The
fixes for this potential breakage should just be a matter of including the