forked from zeek/zeek
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathNEWS
6443 lines (4891 loc) · 279 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This document summarizes the most important changes in the current Zeek
release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as Broker, come with their own ``CHANGES``.)
Zeek 7.2.0
==========
Breaking Changes
----------------
New Functionality
-----------------
- Some DNS events are not raised when ``dns_skip_all_addl`` is set to true.
Zeek now raises a warning when a script declares these events while this
option is set to true.
- Types can now be used as constants in Zeek script. This allows types to be
directly passed into BIFs without aliasing.
- A new ``enc_part`` field was added to the Kerberos ``KRB_Response`` record
passed as part of the ``krb_as_response`` event. This field contains the
encrypted session information from a Kerberos response, including the cipher
and encrypted data.
- Geneve tunnel options of the current packet can be extracted from scripts
using the new PacketAnalyzer::Geneve::get_options() builtin function.
Changed Functionality
---------------------
- The ``service`` field in the connection log is now sorted in the order that
protocol analyzers raise their confirmation events.
Since the time at which the protocol confirmation is raised depends on the
indivivual implementation of each analyzer, there is no specific meaning
to the order that the services appear. However, the order should be
deterministic between runs. It also will in many cases represent
the order in which layered protocols are parsed (e.g. "quic,ssl").
- The way that protocol violations are handled by the dynamic protocol
detection (DPD) changed. Now, a violation that is raised by an analyzer
before it is confirmed will immediately disable the analyzer. This adjusts
the behavior back to the historically desired state, and aligns it with
the treatment of confirmed analyzers.
As a consequence of this, the option ``DPD::max_violations`` is no longer used.
It will be retained till Zeek 8.1 to prevent script errors, and raises a
deprecation warning.
The way failed services interact with the ``service`` field in the connection
log also changed. In the past, protocol analyzers that were confirmed and
later failed were removed from the ``service`` field in some cases. This
commonly lead to the case that a protocol log exists, while the service is
not listed in the connection.log - so, e.g., an etry in ``http.log`` existing
without ``http`` showing up in the connection log.
Now, protocol analyzers that raised a confirmation event will always be added to
the ``service`` field in the connection log, and the entry will be retained
even if the analyzer raises a violation later.
To extend the visibility of protocol violations, a new option
``DPD::track_removed_services_in_connection`` was added. Enabling it causes
failed analyzers to be logged to the ``service`` field of the connection log,
with a prepended "-". So a connection that attached the ``ssl`` analyzer
which later failed due to a protocol error will be logged as ``ssl,-ssl``.
This change also adds a new policy script,
``protocols/conn/failed-service-logging.zeek``. Loading this script adds the
column ``failed_service`` to the connection.log. This column contains the
list of protocol analyzers that failed due to a protocol error.
- Command line options processing will no longer print usage whenever there
is an error. Instead, issues in command line processing will print an error,
then prompt to use --help. The --help usage will now print to standard output
rather than standard error.
- Saving seeds with ``--save-seeds`` will now put Zeek into deterministic mode.
A subsequent ``--load-seeds`` run with the same scripts and traces will produce
identical UID values as the original ``--save-seeds` run.
- The `policy/protocols/dns/detect-external-names.zeek` script now no longer logs
names that were found in mDNS broadcasts by default. This is configurable with
the new `DNS::skip_resp_host_port_pairs` option.
Furthermore, the script now supports and logs IPv6 results.
- The mkdir(), rmdir(), unlink(), and rename() functions now trigger reporter
warnings instead of builtin errors when hitting trouble. This allows Zeek to
continue gracefully in case of such problems, particularly during zeek_init().
- The RDP analyzer now also parses connections that do not contain the cookie
field, which were previously rejected.
Removed Functionality
---------------------
Deprecated Functionality
------------------------
Zeek 7.1.0
==========
We would like to thank Aashish Sharma (@initconf), Andras Gemes (@gemesa),
Anthony Kasza (@anthonykasza), Benjamin Grap (@blightzero), Chiragdeshlehra27,
@cooper-grill, Craig Leres (@leres), Eldon Koyle (@ekoyle), Emmanuele Zambon
(@zambo99), Fox-IT Data Science (@fox-ds), Fupeng Zhao (@AmazingPP), Jan
Grashöfer (@J-Gras), Jordan Barnartt (@JordanBarnartt), Jürgen Löhel (@jloehel),
Justin Azoff (@JustinAzoff), Lucas (@Lucasmeteenc), Martin van Hensbergen
(@martinvanhensbergen), Matti Bispham (@mbispham), Matteo (@skorpion98), Mike
Dopheide (@dopheide-esnet), Mike Peters (@MP-Corelight), Mohan Dhawan
(@Mohan-Dhawan), Pierre (@p-l-), @robinkou, Rodrigo Rojo (@r-rojo), @scyllaever,
Seth Hall (@sethhall), Simeon Miteff (@simeonmiteff), @Sonderino, @superzerosec,
Sven (@svenvanhal), Theo Buehler (@botovq), @timo-mue, @Zopazz, and
@zrobinette12 for their contributions to this release.
Breaking Changes
----------------
- The ``OpaqueVal::DoSerialize`` and ``OpaqueVal::DoUnserialize`` methods were
marked as deprecated in v7.0 and have now been removed as per the Zeek
deprecation policy. Plugins that were overriding these methods and were not
updated will fail to compile. Those plugins should be updated to override the
new ``OpaqueVal::DoSerializeData`` and ``OpaqueVal::DoUnserializeData``
methods.
- Certain internal methods on the broker and logging classes have been changed to
accept std::vector<threading::Value> parameters instead of threading::Value**
to leverage automatic memory management, reduce the number of allocations
and use move semantics to express ownership.
The DoWrite() and HookLogWrite() methods which can be provided by plugins
are not affected by this change, so we keep backwards compatibility with
existing log writers.
- ``Func::Name()`` was deprecated, use ``Func::GetName()`` instead.
New Functionality
-----------------
- The following dependencies have had updates:
- The bundled version of Spicy was updated to 1.12.0. See
https://github.com/zeek/spicy/releases/tag/v1.12.0 for notes on what's new
with Spicy.
- The bundled version of c-ares has been updated to v1.34.2, which required
some updates to Zeek's internal DNS resolver due to changes in the c-ares
API. At least version v1.28.0 is now required to build Zeek.
- Python 3.9 is now required for Zeek and all of it's associated subprojects.
- IP-based connections that were previously not logged due to using an unknown
IP protocol (e.g. not TCP, UDP, or ICMP) now appear in conn.log. All conn.log
entries have a new ``ip_proto`` column that indicates the numeric IP protocol
identifier used by the connection. A new policy script at
``policy/protocols/conn/ip-proto-name-logging.zeek`` can be loaded to also add
an ``ip_proto_name`` column with a string version of the ``ip_proto`` value.
This entire feature can be disabled by loading the new
``policy/protocols/conn/disable-unknown-ip-proto-support.zeek`` policy script.
- New ``Cluster::publish()``, ``Cluster::subscribe()`` and ``Cluster::unsubscribe()``
functions have been added. In contrast to their ``Broker`` counterparts, these
will operator on whichever cluster backend is enabled. Going forward, in-tree
``Broker::publish()`` usages will be replaced with ``Cluster::publish()`` and
script writers should opt to prefer these over the Broker-specific functions.
- Zeek now includes a PostgreSQL protocol analyzer. This analyzer is enabled
by default. The analyzer's events and its ``postgresql.log`` should be
considered preliminary and experimental until the arrival of Zeek's next
long-term-stable release (8.0).
If you observe unusually high CPU consumption or other issues due to this
analyzer being enabled by default, the easiest way to disable it is via the
``Analyzer::disabled_analyzers`` const as follows:
redef Analyzer::disabled_analyzers += {
Analyzer::ANALYZER_POSTGRESQL,
};
If you observe PostgreSQL traffic in your environment, please provide feedback
about the analyzer and structure of the new log.
- Broker's message I/O buffering now operates on per-peering granularity at the
sender (it was previously global) and provides configurable overflow handling
when a fast sender overwhelms a slow receiver, via the following new tunables
in the ``Broker`` module:
const peer_buffer_size = 2048 &redef;
const peer_overflow_policy = "disconnect" &redef;
const web_socket_buffer_size = 512 &redef;
const web_socket_overflow_policy = "disconnect" &redef;
When a send buffer overflows (i.e., it is full when a node tries to transmit
another message), the sender may drop the message and unpeer the slow receiver
(policy ``disconnect``, the default), drop the newest message in the buffer
(``drop_newest``), or drop the oldest (``drop_oldest``). Buffer sizes are
measured in number of messages, not bytes. Note that "sender" and "receiver"
are independent of the direction in which Zeek established the peering. After
disconnects Zeek automatically tries to re-establish peering with the slow
node, in case it recovers.
Zeek notifies you in two ways of such disconnects:
* A cluster.log entry for the sending node indicates that a slow peered node
has been removed. Here node ``worker01`` has removed a peered ``proxy01`:
1733468802.626622 worker01 removed due to backpressure overflow: 127.0.0.1:42204/tcp (proxy01)
* The labeled counter metric ``zeek_broker_backpressure_disconnects_total``
in the telemetry framework tracks the number of times such disconnects
happen between respective nodes. The following scraped telemetry indicates
the same disconnect as above:
zeek_broker_backpressure_disconnects_total{endpoint="worker01",peer="proxy01"} 1
To implement custom handling of a backpressure-induced disconnect, add a
``Broker::peer_removed`` event handler, as follows:
event Broker::peer_removed(endpoint: Broker::EndpointInfo, msg: string)
{
if ( "caf::sec::backpressure_overflow" !in msg )
return;
# The local node has disconnected the given endpoint,
# add your logic here.
}
These new policies fix a problem in which misbehaving nodes could trigger
cascading "lockups" of nodes, each ceasing to transmit any messages.
- The LDAP analyzer now supports handling of non-sealed GSS-API WRAP tokens.
- StartTLS support was added to the LDAP analyzer. The SSL analyzer is enabled
for connections where client and server negotiate to TLS through the extended
request/response mechanism.
- The ``unknown_protocols()`` event now includes the name of all packet
analyzer used for processing the packet when the event is raised. The
``unknown_protocol.log`` file was extended to include this information.
- The MySQL analyzer now generates a ``mysql_user_change()`` event when the user
changes mid-session via the ``COM_USER_CHANGE`` command.
- The DNS analyzer was extended to support TKEY RRs (RFC 2390). A corresponding
``dns_TKEY`` event was added.
- The ``signature_match()`` and custom signature events now receive the end of
match offset within the ``data`` parameter as an optional parameter named
``end_of_match``.
event signature_match(state: signature_state, msg: string, data: string, end_of_match: count);
- A new plugin hook ``InitPreExecution()`` has been added to allow introspection
of Zeek's AST after ZAM optimizations ran. This hook executes right before
the ``zeek_init()`` event is enqueued.
- The SQLite logger now supports setting the value of the SQLite synchronous mode,
as well as of the journal mode. For example, WAL mode can be enabled by setting:
redef LogSQLite::journal_mode=LogSQLite::SQLITE_JOURNAL_MODE_WAL;
- A pseudo protocol analyzer StreamEvent has been added. Attaching this analyzer
to TCP connections allows processing the connection's stream data in the
scripting layer. One example use-case is interactive terminal sessions over
HTTP connections upgraded to TCP.
redef HTTP::upgrade_analyzers += {
["tcp"] = Analyzer::ANALYZER_STREAM_EVENT,
};
event stream_deliver(c: connection, is_orig: bool, data: string);
This comes with performance caveats: For use-cases with high-data rates
a native protocol analyzer with dedicated events will be far more efficient.
- Experimental support for pluggable cluster backends has been added. New plugin
components have been introduced to support switching Zeek's Broker-based
publish-subscribe and remote logging functionality to alternative implementations.
redef Cluster::backend = Cluster::CLUSTER_BACKEND_ZEROMQ;
Besides the backend, the serialization format used for events and log-writes
has become pluggable as well.
- The Zeek distribution now includes an experimental ZeroMQ based cluster backend.
To experiment with it, load the following script on each cluster node.
@load frameworks/cluster/backend/zeromq/connect
Note that Broker-dependent scripts or integrations will become non-functional
when doing so as Zeek nodes will not listen on Broker ports anymore, nor will
they establish a peering to other nodes.
- Zeek now ships with an experimental Spicy-based SSL analyzer, which is
disabled by default. This analyzer can be enabled using the
``--enable-spicy-ssl`` conifgure-time option. The Spicy-based analyzer has
full support for SSL and TLS, just like the current binpac analyzer. It does,
however, not support any version of DTLS. Enabling it will disable DTLS
parsing in Zeek.
The analyzer is currently mostly interesting if you want to experiment with
SSL; we do not yet recommend to enable it in normal Zeek deployments.
- The majority of the metrics reported via stats.log are also now reported via
the Telemetry framework, and are visible in the output passed to Prometheus.
- A new weird ``DNS_unknown_opcode`` was added to the DNS analyzer to report
when it receives opcodes that it cannot process.
Changed Functionality
---------------------
- Heuristics for parsing SASL encrypted and signed LDAP traffic have been
made more strict and predictable. Please provide input if this results in
less visibility in your environment.
- The MySQL analyzer has been improved to better support plugin authentication
mechanisms, like caching_sha2_password, as well as recognizing MySQL query
attributes.
- The ``mysql.log`` for user change commands will contain *just* the username
instead of the remaining parts of the command, including auth plugin data.
- The POP3 parser has been hardened to avoid unbounded state growth in the
face of one-sided traffic capture or when enabled for non-POP3 traffic.
Concretely, the Redis protocol's AUTH mechanism enables the POP3 analyzer
for such connections through DPD.
- Batching and flushing for local log writers can now be controlled via the
options ``Log::flush_interval`` and ``Log::write_buffer_size``. Previously
the ``Threading::heartbeat_interval`` was used for flushing and the buffer
size fixed at 1000.
- Logging of the FTP PASS command in ``ftp.log`` now honors ``FTP::default_capture_password``
and the password is blanked with "<hidden>". Previously, the argument for the PASS
command would be logged in clear.
- The ASCII input reader now suppresses warnings for consecutive invalid lines,
producing a summary of total suppressions once a valid line is encountered.
- The `Telemetry::sync()` hook is now invoked on demand. Either when the metrics
of a node are scraped via the Prometheus HTTP endpoint, or one of the collect
methods is invoked from Zeek script.
- The community-id-logging.zeek policy script was used to set ``c$conn$community_id``
during ``new_connection()`` rather than ``connection_state_remove()``, allowing
other scripts to reuse its value early.
- Calling ``Broker::publish()`` now uses the event time of the currently
executing event as network time metadata attached to the remote event.
Previously, ``network_time()`` was used. This matters if ``Broker::publish()``
is called within scheduled events or called within remote events.
- The SSL analyzer now reports the correct version when an SSLv2 client hello is
used. Zeek previously always reported these as v2, even when the v2 client
hello indicated support for a later version of SSL.
Deprecated Functionality
------------------------
- The ``Broker::auto_publish()`` function has been deprecated and should
be replaced with explicit ``Broker::publish()`` invocations that are
potentially guarded with appropriate ``@if`` or ``@ifdef`` directives.
- The misspelled ``complte_flag`` in the ``dns_binds_rr`` record has been deprecated.
The new ``complete_flag`` uses type ``count`` instead of ``string``.
Zeek 7.0.0
==========
Breaking Changes
----------------
- The Telemetry framework has had a major rework, and includes a number of
breaking changes. The biggest change is a move towards a Prometheus-first
model. This removes the internal aggregation of metrics from nodes onto the
manager node, replacing it with a Prometheus service discovery endpoint. The
usage of this endpoint is described in the updated documentation for the
Telemetry framework. Using this endpoint also requires adding a new
``metrics_port`` field to each node in the cluster configuration, denoting
what port to connect to for each node.
All of the metrics-related script-level options, type, and methods have been
moved to the Telemetry framework:
* Option ``Broker::metrics_port` is now ``Telemetry::metrics_port``
* Option ``Broker::metrics_export_endpoint_name`` is now ``Telemetry::metrics_endpoint_name``
The following options have been removed:
* ``Broker::metrics_export_interval``
* ``Broker::metrics_export_topic``
* ``Broker::metrics_import_topics``
* ``Broker::metrics_export_prefixes``
The ``unit`` field has been removed from the telemetry log.
All of the ``BROKER_METRICS_*`` environment variables have been removed. Use
the equivalent script-level variables for setting configuration settings. For
cluster settings, a new ``metrics_port`` option was added to the ``node.cfg``
options and can be used to set the port number. The cluster layout record
defined in ``cluster-layout.zeek`` has the same option, and it can be
automatically populated by ``zeekctl`` during the ``install`` or ``deploy``
commands.
The instruments that previously supported ``count`` in scripts and ``int64_t``
in C++ were removed in favor of only providing ``double`` versions. Prometheus
only handles ``double`` underneath the Zeek code, so it makes sense to only
support it directly in Zeek as well. This also simplifies the code
significantly.
The ``is_sum`` argument has been removed from the constructors/creation
methods for all of the instruments. This again follows how Prometheus works,
where ``counter`` instruments are always considered sums and ``gauge``
instruments are not. ``Histogram`` instruments don't have the concept of
summing.
- Zeekctl now sets `FileExtract::prefix` to `spool/extract_files/<node>` to avoid
deletion of extracted files when stopping worker nodes. To revert to the
previous behavior, set `FileExtractDir` to an empty string in `zeekctl.cfg`.
If you never enabled Zeek's file extraction functionality, there's no impact.
New Functionality
-----------------
- Support ``delete`` on tables, sets and vectors to clear their contents.
global v = vector(1, 2, 3);
delete v;
assert |v| == 0;
- A new helper function ``can_load()`` backed by a new bif ``find_in_zeekpath()``
was added to determine if a non-relative ``@load`` directive might work. This
can be used to guard ``@load`` directives when script packages may or may not
be installed.
@if ( can_load("my-package") )
@load my-package
@endif
- Zeek packagers can now include a "local" addition into Zeek's version string.
Configure your Zeek build with ``--localversion=XXX`` to add the provided
string, dash-separated, at the end of Zeek's regular version string. The build
setup will refuse strings that contain dashes, to avoid confusing the version
components. For debug builds, the ``-debug`` suffix remains at the end. For
example, the version string on a debug-enabled build with local addition "XXX"
might be "7.0.0-dev.114-XXX-debug". For Docker builds, the ``LOCALVERSION``
environment variable configures the addition.
- SMB2 packets containing multiple PDUs now correctly parse all of the headers,
instead of just the first one and ignoring the rest. This may cause increased
CPU load on SMB2-heavy networks.
- The new built-in function ``lookup_connection_analyzer_id()`` retrieves the
numeric identifier of an analyzer associated with a connection. This enables
the use of the ``disable_analyzer()`` BiF outside of the analyzer
confirmation/violation events that have so far been the only providers of
those identifiers. For example, this allows the suppression of an analyzer
from the outset for specific connections:
event connection_established(c: connection):
{
if ( no_http_for_this_conn_wanted(c) )
{
local aid = lookup_connection_analyzer_id(c$id, Analyzer::ANALYZER_HTTP);
if ( aid > 0 )
disable_analyzer(c$id, aid, T, T);
}
}
Use ``Analyzer::get_tag()`` if you need to obtain an analyzer's tag from its
name (such as "HTTP").
- The ``from_json()`` function now supports ingesting JSON representations of
tables as produced by the ``to_json()`` function. It now also supports reading
the object-based representation of ports that ``to_json()`` generates for that
Zeek type.
- The ``analyzer.log`` now optionally supports logging of disabled analyzers
through the new option ``Analyzer::logging::include_disabling``.
Changed Functionality
---------------------
- The ``ftp.log`` fuid field is now cleared after handling a command with a fuid
associated with it. Previously, fuid was sticky and any subsequent FTP command
would reproduce the same fuid, even if the command itself did not result in
a file transfer over a data connection (e.g., CWD, DEL, PASV, SIZE).
- The type_name field populated by ``global_ids()`` now aligns with the value
returned by ``type_name()`` for each identifier. E.g, ``Site::local_nets``
has a type_name of ``set[subnet]`` rather than ``table``.
- The ISO 9660 file signature has been moved into the policy directory. The
signature has previously been non-functional due to implicit anchoring. Further,
this signature requires users to significantly increase their
``default_file_bof_buffer_size``. Users can now enable this signature by loading
``frameworks/signatures/iso-9660`` which also increases the BOF buffer sufficiently.
Note, doing so may increase memory and CPU usage significantly.
- The ``val_footprint()`` BiF now factors in the size of strings when reporting
footprints, roughly equating a string's size with the number of elements
comparable to that length. As before, footprints are not meant to be precise
but mainly for providing comparisons, which is why this is not a breaking
change.
- The tuning/defaults policy has been deprecated and will be removed in
v7.1. This policy was already being loaded by default via local.zeek. The
settings contained within have become the overall defaults for Zeek now,
instead of having to load the policy. The two changes here are that fragments
now timeout after 5 minutes by default instead of no timeout, and extracted
files now have a default size limit of 100MB instead of unlimited.
- If a Spicy protocol analyzers feeds data into file analysis, it now
needs to call Zeek's `Files::register_protocol()` and provide a
callback for computing file handles. If that's missing, Zeek will
issue a warning. While this was not necessary in previous versions,
it aligns with the same requirement for traditional analyzers and
enables customizing file handles for protocol-specific semantics.
- The Supervisor's API now returns NodeConfig records with a cluster table whose
ClusterEndpoints have a port value of 0/unknown, rather than 0/tcp, to
indicate that the node in question has no listening port.
Removed Functionality
---------------------
Deprecated Functionality
------------------------
- The ``--disable-archiver`` configure flag no longer does anything and will be
removed in 7.1. zeek-archiver has moved into the zeek-aux repository.
- The policy/frameworks/telemetry/prometheus.zeek script has been deprecated
and will be removed with Zeek 7.1. Setting the ``metrics_port`` field on a
``Cluster::Node`` implies listening on that port and exposing telemetry
in Prometheus format.
Zeek 6.2.0
==========
We would like to thank Anthony Verez (netantho), Bijoy Das (mute019), Jan
Grashöfer (J-Gras), Matti Bispham (mbispham), Phil Rzewski (philrz), and
xb-anssi for their contributions to this release.
Breaking Changes
----------------
- The methods ``Dispatcher::Lookup()`` and ``Analyzer::Lookup()`` in the packet_analysis
namespace were changed to return a reference to a std::shared_ptr instead of a copy
for performance reasons.
- Zeek's ``OPENSSL_INCLUDE_DIR`` is not automatically added to an external plugin's
include path anymore. A plugin using OpenSSL functionality directly can use the
following explicit entry to re-use Zeek's ``OPENSSL_INCLUDE_DIR``:
zeek_add_plugin(
Namespace Name
INCLUDE_DIRS "${OPENSSL_INCLUDE_DIR}"
SOURCES ...
)
- The "segment_profiling" functionality and ``load_sample`` event have been removed
without deprecation. This functionality was unmaintained and not known to be used.
- Certain ``ldap.log`` and ``ldap_search.log`` fields have been renamed from
plural to singular and their types changed to scalars. This maps better onto
the expected request-response protocol used between client and server. Additionally,
it removes the burden of working with non-scalar columns from downstream systems.
Specifically, for ``ldap.log``:
* ``arguments: vector of string`` is now ``argument: string``
* ``diagnostic_messages: vector of string`` is now ``diagnostic_message: string``
* ``objects: vector of string`` is now ``object: string``
* ``opcodes: set[string]`` is now ``opcode: string``
* ``results: set[string]`` is now ``result: string``
For ``ldap_search.log``, the following fields were changed:
* ``base_objects: vector of string`` is now ``base_object: string``
* ``derefs: set[string]`` is now ``deref_aliases: string``
* ``diagnostic_messages: vector of string`` is now ``diagnostic_message: string``
* ``results: set[string]`` is now ``result: string``
* ``scopes: set[string]`` is now ``scope: string``
In the unlikely scenario that a request-response pair with the same message
identifier is observed, containing different values for certain fields, new
weirds are raised and will appear in ``weird.log``, including the old and new
values as well as the LDAP message identifier. The value within the LDAP logs
will be the most recently observed one.
- BIF methods now return a ``ValPtr`` directly instead of a ``BifReturnVal`` object
which was just a thin wrapper around ``ValPtr``. This may cause compilation errors
in C++ code that was calling BIF methods directly.
New Functionality
-----------------
- The table type was extended to allow parallel regular expression matching
when a table's index is a pattern. Indexing such tables yields a vector
containing all values of matching patterns for keys of type string.
As an example, the following snippet outputs ``[a, a or b], [a or b]``.
global tbl: table[pattern] of string;
tbl[/a/] = "a";
tbl[/a|b/] = "a or b";
tbl[/c/] = "c";
print tbl["a"], tbl["b"];
Depending on the patterns and input used for matching, memory growth may
be observed over time as the underlying DFA is constructed lazily. Users are
advised to test with realistic and adversarial input data with focus on
memory growth. The DFA's state can be reset by removal/addition of a single
pattern. For observability, a new bif ``table_pattern_matcher_stats()``
can be used to gather ``MatcherStats``.
- Support for delaying log writes.
The logging framework offers two new functions ``Log::delay()`` and ``Log::delay_finish()``
to delay a ``Log::write()`` operation. This new functionality allows delaying of
a specific log record within the logging pipeline for a variable but bounded
amount of time. This can be used, for example, to query and wait for additional
information to attach to the pending record, or even change its final verdict.
Conceptually, delaying a log record happens after the execution of the global
``Log::log_stream_policy`` hook for a given ``Log::write()`` and before the
execution of filter policy hooks. Any mutation of the log record within the
delay period will be visible to filter policy hooks. Calling ``Log::delay()``
is currently only allowed within the context of the ``Log::log_stream_policy`` hook
for the active ``Log::write()` operation (or during the execution of post delay callbacks).
While this may appear restrictive, it makes it explicit which ``Log::write()``
operation is subject to the delay.
Interactions, semantics and conflicts of this feature when writing the same
log record multiple times to the same or different log streams need to be taken
into consideration by script writers.
Given this is the first iteration of this feature, feedback around usability and
use-cases that aren't covered are more than welcome.
- A WebSocket analyzer has been added together with a new ``websocket.log``.
The WebSocket analyzer is instantiated when a WebSocket handshake over HTTP is
recognized. By default, the payload of WebSocket messages is fed into Zeek's dynamic
protocol detection framework, possibly discovering and analyzing tunneled protocols.
The format of the log and the event semantics should be considered preliminary until
the arrival of the next long-term-stable release (7.0).
To disable the analyzer in case of fatal errors or unexpected resource usage,
use the ``Analyzer::disabled_analyzers`` pattern:
redef Analyzer::disabled_analyzers += {
Analyzer::ANALYZER_WEBSOCKET,
};
- The SMTP analyzer was extended to recognize and properly handle the BDAT command
from RFC 3030. This improves visibility into the SMTP protocol when mail agents
and servers support and use this extension.
- The event keyword in signatures was extended to support choosing a custom event
to raise instead of ``signature_match()``. This can be more efficient in certain
scenarios compared to funneling every match through a single event.
The new syntax is to put the name of the event before the string used for the
``msg`` argument. As an extension, it is possible to only provide an event name,
skipping ``msg``. In this case, the framework expects the event's parameters to
consist of only state and data as follows:
signature only-event {
payload /.*root/
event found_root
}
event found_root(state: signature_state, data: string) { }
Using the ``msg`` parameter with a custom event looks as follows. The custom
event's parameters need to align with those for ``signature_match()` event:
signature event-with-msg {
payload /.*root/
event found_root_with_msg "the-message"
}
event found_root_with_msg(state: signature_state, msg: string, data: string) { }
Note, the message argument can currently still be specified as a Zeek identifier
referring to a script-level string value. If used, this is disambiguated behind
the scenes for the first variant. Specifying ``msg`` as a Zeek identifier has
been deprecated with the new event support and will be removed in the future.
Note that matches for signatures with custom events will not be recorded in
``signatures.log``. This log is based on the generation of ``signature_match()``
events.
- The QUIC analyzer has been extended to support analyzing QUIC Version 2
INITIAL packets (RFC 9369). Additionally, prior draft and some of
Facebook's mvfst versions are supported. Unknown QUIC versions will now be
reported in ``quic.log`` as an entry with a ``U`` history field.
- Conditional directives (``@if``, ``@ifdef``, ``@ifndef``, ``@else`` and
``@endif``) can now be placed within a record's definition to conditionally
define or extend a record type's fields.
type r: record {
c: count;
@if ( cond )
d: double;
@else
d: count;
@endif
};
Note that generally you should prefer record extension in conditionally loaded
scripts rather than using conditional directives in the original record definition.
- The 'X' code can now appear in a connection's history. It is meant to indicate
situations where Zeek stopped analyzing traffic due to exceeding certain limits or
when encountering unknown/unsupported protocols. Its first use is to indicate
``Tunnel::max_depth`` being exceeded.
- A new ``Intel::seen_policy`` hook has been introduced to allow intercepting
and changing ``Intel::seen` behavior:
hook Intel::seen_policy(s: Intel::Seen, found: bool)
- A new ``NetControl::rule_added_policy`` hook has been introduced to allow modification
of NetControl rules after they have been added.
- The IP geolocation / ASN lookup features in the script layer provide better
configurability. The file names of MaxMind databases are now configurable via
the new ``mmdb_city_db``, ``mmdb_country_db``, and ``mmdb_asn_db`` constants,
and the previously hardwired fallback search path when not using an
``mmdb_dir`` value is now adjustable via the ``mmdb_dir_fallbacks``
vector. Databases opened explicitly via the ``mmdb_open_location_db`` and
``mmdb_open_asn_db`` functions now behave more predictably when updated or
removed. For details, see:
https://docs.zeek.org/en/master/customizations.html#address-geolocation-and-as-lookups
- The ``zeek-config`` script now provides a set of ``--have-XXX`` checks for
features optionally compiled in. Each check reports "yes"/"no" to stdout and
exits with 0/1, respectively.
Changed Functionality
---------------------
- The ``split_string`` family of functions now respect the beginning-of-line ^ and
end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere
in the input string.
- The ``sub()`` and ``gsub()` functions now respect the beginning-of-line ^ and
end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere
in the input string.
- Ed25519 and Ed448 DNSKEY and RRSIG entries do not cause weirds anymore.
- The OpenSSL references in ``digest.h`` and ``OpaqueVal.h`` headers have been
hidden to avoid unneeded dependencies on OpenSSL headers. Plugins using the
detail API from ``digest.h`` to compute hashes likely need to accommodate for
this change.
- The ``Tunnel::max_depth`` default was changed from 2 to 4 allowing for more than
two encapsulation layers. Two layers are already easily reached in AWS GLB
environments.
- Nested MIME message analysis is now capped at a maximum depth of 100 to prevent
unbounded MIME message nesting. This limit is configurable with ``MIME::max_depth``.
A new weird named ``exceeded_mime_max_depth`` is reported when reached.
- The ``netcontrol_catch_release.log`` now contains a plugin column that shows which
plugin took an action. The logs also contain information when errors or existing
rules are encountered.
- The ``Cluster::PoolSpec`` record no longer provides default values for its
``topic`` and ``node_type`` fields, since defaults don't fit their intended
use and looked confusing in generated documentation.
Removed Functionality
---------------------
- Zeek no longer automatically subscribes to topics prefixed with "bro/" whenever
subscribing to topics prefixed with "zeek/". This was a leftover backward-
compatibility step in the Broker code that should have been removed long ago.
Deprecated Functionality
------------------------
- The virtual functions ``DoSerialize``and ``DoUnserialize`` of the ``OpaqueVal``
(and ``BloomFilter``) class will be removed with Zeek 7.1. Unfortunately, code
implementing the deprecated methods does not produce compiler warnings.
Plugin authors implementing an ``OpaqueVal`` subclass need to convert to
``DoSerializeData`` and ``DoUnserializeData``:
* ``std::optional<BrokerData> OpaqueVal::DoSerializeData() const``
* ``bool OpaqueVal::DoUnserializeData(BrokerDataView data)``
When overriding ``DoSerializeData()``, return ``std::nullopt`` (or a
default-constructed ``optional``) for values that cannot be serialized.
Otherwise, the canonical way to create a ``BrokerData`` for serialization is
by using a ``BrokerListBuilder``. For example, creating a ``BrokerData`` that
contains ``true`` and the count ``42`` could be implemented as follows:
BrokerListBuilder builder;
builder.Add(true);
builder.AddCount(42u);
return std::move(builder).Build();
Please refer to the respective class documentation for a full list of member
functions on ``BrokerListBuilder`` and ``BrokerDataView``.
For plugins that are using the macro ``DECLARE_OPAQUE_VALUE`` to generate the
function prototypes for the serialization functions: please use
``DECLARE_OPAQUE_VALUE_DATA`` instead to generate prototypes for the new API.
Plugin authors that need to support multiple Zeek versions can use the
``ZEEK_VERSION_NUMBER`` macro to conditionally implement the new and old
methods. Provide the new versions with Zeek 6.2 (60200) or later, otherwise
keep the old signature. The default implementations for the new functions
as used by Zeek will call the old signatures and convert the results.
- The ``Cluster::Node$interface`` field has been deprecated. It's essentially
unneeded, unused and not a reliable way to gather the actual interface used
by a worker. In Zeekctl deployments the field will be populated until its
removal. The ``packet_source()`` bif should be used on worker processes to
gather information about the interface.
- The ``policy/misc/load-balancing`` script has been deprecated in favor of
AF_PACKET PF_RING, Netmap or other NIC specific load balancing approaches.
- Time machine related enums, options and fields have been marked for removal.
- The ``check_for_unused_event_handlers`` options the related ``UsedHandlers()``,
``UnusedHandlers()`` and their related ``SetUsed()`` and ``Used()`` methods
have been marked for removal. The feature of finding unused event handlers is
provided by default via the ``UsageAnalyzer`` component.
- Using a Zeek identifier for the ``msg`` argument within a signatures's ``event``
keyword has been deprecated.
Zeek 6.1.0
==========
Breaking Changes
----------------
- ``assert`` is now a reserved keyword for the new ``assert`` statement.
- The ``__bro_plugin__`` file that gets generated as part of plugin builds was
renamed to ``__zeek_plugin__``. This will affect the ability for older
versions of ``zkg`` to use the ``zkg unload`` and ``zkg load`` commands. This
should only cause breakage for people using a version of ``zkg` that doesn't
come bundled with Zeek (which we generally don't recommend doing).
- Zeek does not traverse into dot directories to find plugins or hlto files
anymore. Any dot directories found below the directories specified in
ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are now skipped. Dot directories
explicitly listed in ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are not
skipped.
- External plugins will fail to configure if their minimum required CMake
version is below 3.15. This was a warning with Zeek 6.0, but has caused user
confusion due to unhelpful error messages around the IN_LIST operator policy.
- The FindBISON, FindOpenSSL, FindPackageHandleStandardArgs, FindPackageMessage,
and SelectLibraryConfigurations cmake files were removed from our cmake
repository in favor of the versions that come with CMake. This should not
cause any breakage, but it is possible in the case that someone was using
these in a plugin.
New Functionality
-----------------
- Zeek now includes the LDAP protocol analyzer from the zeek/spicy-ldap project
(https://github.com/zeek/spicy-ldap). This analyzer is enabled by default. The
analyzer's events and its ``ldap.log`` and ``ldap_search.log`` should be
considered preliminary and experimental until the arrival of Zeek's next
long-term-stable release (7.0).
If you observe unusually high CPU consumption or other issues due to this
analyzer being enabled by default, the easiest way to disable it is via the
``Analyzer::disabled_analyzers`` const as follows:
redef Analyzer::disabled_analyzers += {
Analyzer::ANALYZER_LDAP_UDP,
Analyzer::ANALYZER_LDAP_TCP,
};
Please do report issues to us including diagnostic information in case this is
necessary in your environment. We're also open to general feedback about the
structure of the new logs.
- Zeek now includes the QUIC protocol analyzer from the zeek/spicy-quic project
(https://github.com/zeek/spicy-quic). This project is a fork of Fox-IT's
initial implementation (https://github.com/fox-ds/spicy-quic).
As for the LDAP analyzer, the analyzer's events and the new ``quic.log``
should be considered preliminary and experimental until the arrival of Zeek's
next long-term-stable release (7.0). As above, any feedback and contributions
to this analyzer and the new log are welcome.
The analyzer's functionality is limited to decryption of the INITIAL packets
of QUIC version 1. If decryption of these packets is successful, the handshake
data is forwarded to Zeek's SSL analyzer. An ``ssl.log`` entry will appear in
``ssl.log`` for QUIC connections. The entry in the ``conn.log`` will contain
``quic`` and ``ssl`` in the service field.
To disable the analyzer in case of issues, use the following snippet:
redef Analyzer::disabled_analyzers += {
Analyzer::ANALYZER_QUIC,
};
- Added a new ``assert`` statement for assertion based testing and asserting
runtime state.
assert <expr: bool>[, <message: string>];
This statement comes with two hooks. First, ``assertion_failure()`` that is
invoked for every failing assert statement. Second, ``assertion_result()``
which is invoked for every assert statement and its outcome. The latter allows
to construct a summary of failing and passing assert statements. Both hooks
receive the location and call stack for the ``assert`` statement via a
``Backtrace`` vector.
A failing assert will abort execution of the current event handler similar to
scripting errors. By default, a reporter error message is logged. Using the
break statement within ``assertion_failure()`` or ``assertion_result()``
allows to suppress the default message.
- Add a new ``&default_insert`` attribute for tables. This behaves as
``&default`` with the addition that the default value is inserted into the
table upon a failed lookup. Particularly for tables with nested container
values, the ``&default`` behavior of not inserting the value can be of little
use.
- The ``from_json()`` function now takes an optional key_func argument to
normalize JSON object key names. This can be useful if the keys in a JSON
object are not valid Zeek identifiers or reserved keywords.
- Module names are now included in ``global_ids()``. Their key in the returned
table is prefixed with "module " and their value will have the ``type_name``
field set to "module".
- Identifiers in the global scope can now be referenced and defined from within
modules by prefixing their names with ``::``. Previously, these required an
explicit ``GLOBAL::`` prefix to be used. Using ``GLOBAL::`` has been
deprecated.
- The ``as`` keyword now supports casting between ``set`` and ``vector`` values
with the same element type. Converting ``set`` values with multiple index
values is not supported. We plan to extend the use of the ``as`` keyword to
support more type conversions in the future.
- Added new packet analyzer to handle PCAP files DLT_PPP link type.
- Fixed appending of ``any`` to ``vector of any``.
- The ModBus analyzer's function support was expanded, with new handling of the
Encapsulation Interface Transport (function 28) And Diagnostics (function 8)
functions. This adds new ``modbus_encap_interface_transport_{request,response}``
and ``modbus_diagnostics_{request,response}`` events.
- The ModBus file record read and write events now provide the full data from
the request and response messages as part of the event data.
- The full PDU length was added to the ``ModBusHeader`` record type passed with
all of the ModBus events.
Changed Functionality
---------------------
- A connection's value is now updated in-place when its directionality is
flipped due to Zeek's heuristics (for example, SYN/SYN-ACK reversal or
protocol specific approaches). Previously, a connection's value was discarded
when flipped, including any values set in a ``new_connection()`` handler. A
new ``connection_flipped()`` event is added to allow updating custom state in
script-land.
- Loading ``policy/frameworks/notice/community-id.zeek`` now also automatically
community ID logging. In the past, loading the script had no effect unless
``policy/protocols/conn/community-id-logging.zeek`` was loaded before. This
was fairly unusual and hard to debug behavior.
- Connections to broadcast addresses are not flipped based on
``likely_server_ports`` anymore. Previously, broadcast packets originating
from a likely server port resulted in 255.255.255.255 being the originator in
``conn.log``.
- When too many HTTP requests are pending, Zeek will now log them at once and
reset request/response correlation instead of running into unbounded state
growth. This behavior is configurable via a new option
``HTTP::max_pending_requests``. The default is ``100``.
- Fix deferred initialization of nested records containing non-const &default
attributes.
- Parameter lists for functions, events and hooks now use commas instead of
semicolons in error messages or when printing such functions.