-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGES
26565 lines (16673 loc) · 935 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
4.1.0-dev.539 | 2021-04-20 15:21:03 -0700
* Fix LGTM finding in zeek-cut with uses of localtime and gmtime (Tim Wojtulewicz, Corelight)
4.1.0-dev.537 | 2021-04-20 14:34:48 -0700
* Restore support for vectors with holes and improve test cases (Vern Paxson, Corelight)
There's no longer breaking behavior changes to how vector-holes work from
previous Zeek versions.
* Fix cloning/copying vectors that contain holes (Vern Paxson, Corelight)
* fix using ++/-- to vectors that contain holes (Vern Paxson, Corelight)
4.1.0-dev.526 | 2021-04-16 16:03:06 -0700
* GH-1506: Fix Broker unserialization of set/table function indices (Jon Siwek, Corelight)
Zeek function types are serialized as a broker::vector, but the
unserialization logic for Zeek set/table types that use a function for
as an index incorrectly identified it as a composite-index, which also
use broker::vector, rather than a singleton-index, and makes such
unserialization fail.
A general example where this failure can happen in practice is when
trying to unserialize a connection record for which there's a
Conn::RemovalHook, since that's a set[function], and a specific case of
that is use of the Intel Framework in a Zeek cluster.
4.1.0-dev.524 | 2021-04-16 08:08:38 -0700
* Move an assert() in input/Manager.cc to account for ValueToVal errors (Jon Siwek, Corelight)
* Add test for config framework (Tim Wojtulewicz, Corelight)
* Fix similar issues with ValueTo* methods in the input framework (Tim Wojtulewicz, Corelight)
* GH-1487: Handle error from ValueToVal instead of ignoring it (Tim Wojtulewicz, Corelight)
4.1.0-dev.519 | 2021-04-15 17:44:16 -0700
* Fix indexing of set/table types with a vector (Jon Siwek, Corelight)
That previously caused an internal error/crash
* GH-1507: Tolerate junk data before SIP requests (Jon Siwek, Corelight)
This allows for data that won't match a SIP request method to precede an
actual request and generates a new 'sip_junk_before_request' weird when
encountering such a situation.
* GH-1503: Improve &expire_func compatibility type-checking (Jon Siwek, Corelight)
Previously, incompatible &expire_funcs could mistakenly be used, such as
when using that attribute on the unspecified table()/set()
initializations/assignments, resulting in invalid function calls that
eventually crash Zeek.
4.1.0-dev.512 | 2021-04-15 11:54:44 -0700
* GH-1496: Fix build on armv7 architecture (Jon Siwek, Corelight)
4.1.0-dev.510 | 2021-04-14 16:05:55 -0700
* GH-1450: Improve printing/logging of large double/interval/time values
The modp_dtoa/modp_dtoa2 functions aren't capable of handling double
values larger than INT_MAX and fallback on using sprintf() in that
situation. Previously, the format string to that sprintf() was "%e",
defaulting to a precision of 6, which is already too few digits to
represent a number known to be larger than INT_MAX. Now, an sprintf()
is still performed for values larger than INT_MAX and still uses a
scientific notation format, but in a way that uses as many decimal
digits as needed to preserve information. (Jon Siwek, Corelight)
4.1.0-dev.508 | 2021-04-14 15:54:03 -0700
* Add btest for unterminated pattern parsing error behavior
And clarified the error message that it's more about finding an
unterminated pattern than knowing for sure there's remaining pattern
text spanning multiple lines. (Jon Siwek, Corelight)
* GH-1497: Support CRLF line-endings in Zeek scripts and signature files (Jon Siwek, Corelight)
* Escape non-printables in "unrecognized character" parser error messages (Jon Siwek, Corelight)
4.1.0-dev.504 | 2021-04-14 15:51:39 -0700
* Add a check for null packet data in pcap IOSource
Some libpcaps (observed in Myricom's) may claim to have read a packet,
but either did not really read a packet or at least provide no way
to access its contents, so this adds a check for null-data to
handle those cases. (Jon Siwek, Corelight)
4.1.0-dev.501 | 2021-04-12 18:03:53 -0700
* Add explanation of vector holes/in-operator changes to NEWS (Jon Siwek, Corelight)
4.1.0-dev.500 | 2021-04-09 12:18:06 -0700
* Reference the manual's explanation of originator and responder from conn_id (Christian Kreibich, Corelight)
4.1.0-dev.498 | 2021-04-07 18:30:58 -0700
* Disable Broker tests in Coverity Scan cronjob build (Jon Siwek, Corelight)
4.1.0-dev.497 | 2021-04-07 15:54:21 -0700
* GH-1493: Fix build with -DENABLE_MOBILE_IPV6 (Tim Wojtulewicz, Corelight)
4.1.0-dev.495 | 2021-04-07 11:12:13 -0700
* Change TableVal::ToMap() to return ValPtr-indexed maps (Jon Siwek, Corelight)
* Add std::hash specialization for IntrusivePtr (Jon Siwek, Corelight)
* Move IntrusivePtr relational operators to zeek namespace
Otherwise some cases relying on argument-dependent lookup (ADL) fail. (Jon Siwek, Corelight)
4.1.0-dev.490 | 2021-04-05 14:13:48 -0700
* Add Telemetry API (C++/BIFs) for gathering runtime metrics (Dominik Charousset, Corelight)
This relies on the CAF metrics API/implementation and allows potential
export to Prometheus. These typical metric types are supported:
counters, gauges, histograms, timers.
4.1.0-dev.475 | 2021-04-03 09:39:10 -0700
* Increase timeout of plugins.reader btest (Jon Siwek, Corelight)
* optionally hash original, not extended, form of records for profiling (Vern Paxson, Corelight)
profile types associated with global initializations
incorporate type name, if present, into type hash
* fixed out-of-range enum constant: OpenFlow::INVALID_COOKIE (Vern Paxson, Corelight)
(Though it's not currently an enum value)
* Add method to add a fully-qualified enum name (module name included) (Vern Paxson, Corelight)
* enum types track whether they've had values added via "redef" (Vern Paxson, Corelight)
* tracking of original size of records (pre redef'ing) (Vern Paxson, Corelight)
* Added TableVal::ToMap to retrieve a table's entire contents as a unordered_map (Vern Paxson, Corelight)
* factoring out some replicated code in the parser (Vern Paxson, Corelight)
* some micro-preening of parse.y; no semantic changes (Vern Paxson, Corelight)
* for readability, removed explicit scoping in parse.y; no semantic changes (Vern Paxson, Corelight)
4.1.0-dev.461 | 2021-04-01 14:11:44 -0700
* function profiling rewritten - more detailed info, supports global profiling (Vern Paxson, Corelight)
Hashes for Zeek script types are now done globally rather than
per-function-body, which can save considerable time due to the complexity
of some commonly used types (such as connection records).
Hashing has been expanded to provide more robust distinctness
(lack of collisions in practice) and determinism (consistently computing
the same hash across compilations).
* track whether a given function/body should be included/skipped for optimization (Vern Paxson, Corelight)
4.1.0-dev.451 | 2021-03-31 11:58:08 -0700
* Add ssh to Alpine Dockerfile for retrieving external test repos (Jon Siwek, Corelight)
4.1.0-dev.449 | 2021-03-31 10:47:22 -0700
* Change ci/init-external-repos.sh to use `base64 -d` (Jon Siwek, Corelight)
The BusyBox version of `base64` does not have `--decode`.
4.1.0-dev.448 | 2021-03-31 10:29:17 -0700
* Teach ci/init-external-repos.sh to consider user permissions (Jon Siwek, Corelight)
Any errors while setting up external/private test repo should fail the
task for PRs submitted by a user with write/admin permission.
4.1.0-dev.447 | 2021-03-31 09:55:05 -0700
* CI: Add OpenSUSE Leap 15.2 (Johanna Amann, Corelight)
* Add CI task for Alpine Linux (Jon Siwek, Corelight)
* Separate stdout from stderr in btest baselines (Jon Siwek, Corelight)
Redirecting both to the same file can show platform-specific differences
(e.g. Alpine), likely due to different buffering defaults.
* Remove newline-eof canonification attempt in diff-remove-timestamps (Jon Siwek, Corelight)
* Change a <sys/errno.h> include to <errno.h> (Jon Siwek, Corelight)
Some systems (e.g. Alpine) may warn that that the former is incorrect.
4.1.0-dev.440 | 2021-03-30 14:57:07 -0700
* Fix incomplete-type for struct timeval (Andrew Benson)
This fixes building on musl (e.g. Void, Alpine, etc.)
4.1.0-dev.438 | 2021-03-29 15:42:25 -0700
* "balance" tests with multiple Zeek scripts to load the same elements (Vern Paxson, Corelight)
* put global statements into a quasi-function to support script optimization (Vern Paxson, Corelight)
4.1.0-dev.434 | 2021-03-29 13:18:18 -0700
* Fix sign-compare compiler warning in coerce_to_record() (Jon Siwek, Corelight)
* Fix maybe-uninitialized warning in ZVal::ToVal() (Jon Siwek, Corelight)
* Change RecordVal::GetFieldAs() to use std::vector::operator[] (Jon Siwek, Corelight)
Since the method claims it's up to the user to ensure the field exists
before calling, the extra bounds-checking done by std::vector::at()
isn't needed.
* Add RecordVal::AssignField() and use it in supervisor code (Jon Siwek, Corelight)
This is a convenience method to assign a known record field value by
field name. May also be useful to reduce warnings from static analysis
(e.g. Coverity) about not checking for negative return values before
assigning since that now flows through a [[noreturn]] error path.
* GH-960: Fix include order of bundled header files (Jon Siwek, Corelight)
Previously, a system-wide installation of any bundled auxil/ software
(like CAF) may get found/included rather than the bundled version and
possibly break the build.
4.1.0-dev.427 | 2021-03-27 14:18:16 -0700
* Update bundled CAF to 0.18.2 (Jon Siwek, Corelight)
4.1.0-dev.426 | 2021-03-26 17:17:45 -0700
* GH-1463: Rename supervisor_rotation_format_func to archiver_rotation_format_func (Vlad Grigorescu)
And expose it for non-supervised setups. Also deprecates the old name.
4.1.0-dev.423 | 2021-03-26 16:44:26 -0700
* GH-1454: Heartbleed: fix substraction order. (Johanna Amann)
Fixes incorrect/overflowed `n` value for `SSL_Heartbeat_Many_Requests`
notices where number of server heartbeats is greater than number of client
heartbeats.
The larger number was substracted from the smaller one leading to an
integer overflow. However, no information was lost due to everything
also being present in the notice message.
4.1.0-dev.421 | 2021-03-26 16:39:22 -0700
* Fix crash in Analyzer::ForwardPacket due to recursive analyzer calls. (Tim Wojtulewicz, Corelight)
The change in 44f558df7b5a85bae40945de653bcb2448e0a7f4 that made analyzer_list
a std::vector instead of a std::list doesn't take into account that in some
cases an analyzer may chain back into itself, such as with UDP-in-UDP tunnels.
In these cases, the second call to ForwardPacket may cause iterator
invalidation, leading to a crash, so this reverts back to using an std::list.
* Include git sha in request to benchmark host (Tim Wojtulewicz, Corelight)
4.1.0-dev.417 | 2021-03-25 11:37:55 -0700
* test suite update due to factoring out coerce_to_record() (Vern Paxson, Corelight)
* removal of vestigial #include's (breaking an include loop in the process) (Vern Paxson, Corelight)
* better method name: HasCopySemantics() (Vern Paxson, Corelight)
* Use STL functionality to initialize coercion map (Vern Paxson, Corelight)
* comments for factored-out index slice functions (Vern Paxson, Corelight)
* support for subclassing ScriptFunc's, esp. for alternate lambda closures (Vern Paxson, Corelight)
* factor out record coercion; modernize management of coercion "map" (Vern Paxson, Corelight)
* lower-level method for adding fields to records (Vern Paxson, Corelight)
* factor out "print" statement's execution functionality (Vern Paxson, Corelight)
* functions for indexing slices and strings (Vern Paxson, Corelight)
* new function for getting the location, if any, associated with the current call (Vern Paxson, Corelight)
4.1.0-dev.404 | 2021-03-24 16:58:50 -0700
* Fix missing `user_agent` existence check in smtp/software.zeek (Michael Dopheide)
Without the check there can be extra reporter.log error noise, but
no actual difference in functionality.
4.1.0-dev.402 | 2021-03-24 15:49:52 -0700
* Add `bin_dir` setting for default zkg config file (Jon Siwek, Corelight)
Which defaults to `${zeek_install_prefix}/bin`, same as what a
standalone `zkg autoconfig` would use.
4.1.0-dev.400 | 2021-03-23 20:44:19 -0700
* whitespace tweaks (Vern Paxson, Corelight)
* resolved some TODO comments (Vern Paxson, Corelight)
* remove unnecessary casts, and change necessary ones to use static_cast<> (Vern Paxson, Corelight)
* explain cmp_func default (Vern Paxson, Corelight)
* change functions for ZVal type management to static members (Vern Paxson, Corelight)
* fix some unsigned/signed integer warnings (Vern Paxson, Corelight)
* address lint concern about uninitialized variable (Vern Paxson, Corelight)
* Remove use of obsolete forward-declaration macros (Tim Wojtulewicz, Corelight)
* Merge branch 'topic/vern/zval' of github.com:zeek/zeek into topic/vern/zval (Vern Paxson, Corelight)
* fix #include's that lack zeek/ prefixes (Vern Paxson, Corelight)
* overlooked another way in which vector holes can be created (Vern Paxson, Corelight)
* initialize vector holes to the correct corresponding type (Vern Paxson, Corelight)
* explicitly populate holes created in vectors (Vern Paxson, Corelight)
* fix other instances of GetField().get() assuming long-lived ValPtr's (Vern Paxson, Corelight)
* fix for now-incorrect assumption that GetField always returns an existing ValPtr (Vern Paxson, Corelight)
* ... and porting over memory management for assignment to vector elements (Vern Paxson, Corelight)
* ... porting over memory management for assignment to record fields (Vern Paxson, Corelight)
* dagnabbit, forgot to port over destructor cleanup from ZAM_vector/ZAM_record (Vern Paxson, Corelight)
* fix #include's that lack zeek/ prefixes (Vern Paxson, Corelight)
* explicitly populate holes created in vectors (Vern Paxson, Corelight)
* fixes for now-incorrect assumption that GetField always returns an existing ValPtr (Vern Paxson, Corelight)
* memory management for assignment to vector elements (Vern Paxson, Corelight)
* memory management for assignment to record fields (Vern Paxson, Corelight)
* destructor cleanup from ZAM_vector/ZAM_record (Vern Paxson, Corelight)
* test suite update for Smith-Waterman now using 0-based vectors (Vern Paxson, Corelight)
* test suite updates for expanded sort() BiF semantics (bools, doubles) (Vern Paxson, Corelight)
* test suite updates reflecting that vectors no longer support arbitrary "holes" (Vern Paxson, Corelight)
* migration to using new differentiated methods for setting record fields (Vern Paxson, Corelight)
* RecordVal methods for differentiated setting of record fields (Vern Paxson, Corelight)
* fix for fencepost error when looping over vectors (Vern Paxson, Corelight)
* migrate to using RecordVal's new Remove() method (Vern Paxson, Corelight)
* new RecordVal Remove() method to clear a field (Vern Paxson, Corelight)
* use differentiated record field accessors (Vern Paxson, Corelight)
* base support for differentiated record field accessors (Vern Paxson, Corelight)
* migrate to differentiated vector "At" accessors to support future efficiency (Vern Paxson, Corelight)
* fix for subtle bug due to GetField now potentially returning a new ValPtr (Vern Paxson, Corelight)
* use HasField() rather than GetField() where appropriate (Vern Paxson, Corelight)
* moving sort()/order() functionality into VectorVal (Vern Paxson, Corelight)
* convert VectorVal's to use vector<ZVal> as internal representation (Vern Paxson, Corelight)
* convert RecordVal's to use vector<ZVal> as internal representation (Vern Paxson, Corelight)
* added ZVal for low-level representations of Zeek script values (Vern Paxson, Corelight)
* fix mis-typed record fields that happened to work in original RecordVal impl. (Vern Paxson, Corelight)
* fix off-by-one assumption regarding vector indexing dating to 2011 (Vern Paxson, Corelight)
4.1.0-dev.356 | 2021-03-23 19:33:44 -0700
* Increase timeout of openflow.log-cluster btest (Jon Siwek, Corelight)
4.1.0-dev.355 | 2021-03-23 19:05:10 -0700
* add option for deterministic descriptions of sets & tables (Vern Paxson, Corelight)
* determinism for concurrent Zeek test suite invocations; split out deprecations (Vern Paxson, Corelight)
* disambiguate descriptions of enum types; include attributes when describing record types (Vern Paxson, Corelight)
* more liberal view of attribute equality; allow suppressing attr type-checking (Vern Paxson, Corelight)
* support for operations on sets that return new values (Vern Paxson, Corelight)
* add low-level method for adding new enum values (Vern Paxson, Corelight)
* sundry accessors/cast-ers; RE_Matcher's track their construction values (Vern Paxson, Corelight)
* add convenience functions for comparing IP addresses (Vern Paxson, Corelight)
4.1.0-dev.346 | 2021-03-23 17:25:02 -0700
* Add new ``./configure --plugindir`` option (Vlad Grigorescu)
Useful for customizing the default installation path for plugins.
4.1.0-dev.344 | 2021-03-23 16:29:27 -0700
* Clean up iterator usage in Analyzer (Tim Wojtulewicz, Corelight)
The ``zeek::analyzer::analyzer_list`` type-alias changed from an
``std::list`` to ``std::vector`` which, in practice, is not expected to be
used from plugins in API-incompatible way and may result in ~1-2% overall
performance benefit.
4.1.0-dev.341 | 2021-03-18 12:45:04 -0700
* fix for associating current scope with the name of enums; name tidying (Vern Paxson, Corelight)
* avoid infinite recursion in same_type() if it is analyzing recursive types (Vern Paxson, Corelight)
* remove iffy reliance on type punning that relies on interpreter's behavior (Vern Paxson, Corelight)
* fixes for propagating optimization options, and pruning script function analysis (Vern Paxson, Corelight)
* tidying with respect to "const", and streamlining OuterIDBindingFinder (Vern Paxson, Corelight)
* making Exec methods non-const, so execution can manage side-information (Vern Paxson, Corelight)
* removing unneeded BoolVal constructor that can cause overload ambiguity issues (Vern Paxson, Corelight)
* lint fixes: ensuring functions return values, robustness to nil Val's (Vern Paxson, Corelight)
* correcting some simple typos (Vern Paxson, Corelight)
4.1.0-dev.330 | 2021-03-17 13:48:34 -0700
* Prevent use of LeakSanitizer on FreeBSD (Jon Siwek, Corelight)
4.1.0-dev.328 | 2021-03-17 10:35:59 -0700
* Add tests for ERSPAN Type I patch
Co-authored-by: Markel Elorza Alvarez <[email protected]>
Co-authored-by: Ivan Arrizabalaga Cupido <[email protected]> (Gorka Olalde Mendia)
* Add ERSPAN Type I patch
Co-authored-by: Markel Elorza Alvarez <[email protected]>
Co-authored-by: Ivan Arrizabalaga Cupido <[email protected]> (Gorka Olalde Mendia)
4.1.0-dev.323 | 2021-03-11 12:43:21 -0800
* ts fields in SMB logs now default to network_time() (Seth Hall, Corelight)
This avoids a problem identified by amanbansal2709 in pull
request #1288. I fixed it in a different way than that pull request
by making sure the ts field is always set so that this isssue doesn't
return in the future.
4.1.0-dev.320 | 2021-03-10 12:29:13 -0800
* GH-1432: Use buffered IO for file extraction (Jon Siwek, Corelight)
This can improve performance significantly: ~3.5x faster when tested on
a large file passing data to the file analysis framework in small chunks
of 20 bytes.
4.1.0-dev.318 | 2021-03-10 12:05:47 -0800
* Avoid searching a directory for dynamic plugins multiple times (Jon Siwek, Corelight)
4.1.0-dev.316 | 2021-03-10 13:00:27 +0000
* Fix potential mime type detection bug in IRC/FTP file_transferred event
The files framework uses strncpy to copy data into the buffer that is
used for IRC/FTP mime type detection. From all I can tell that means
that, in these cases, currently mime type detection will be messed up if
the data being passed in contains zero bytes. (Johanna Amann, Corelight)
4.1.0-dev.314 | 2021-03-08 18:28:22 -0800
* new "opt" btest alternative (Vern Paxson, Corelight)
* update to "xform" test suite baseline due to recent line number changes (Vern Paxson, Corelight)
* logic for driving AST optimization when requested (Vern Paxson, Corelight)
* methods implementing AST optimization (aliasing, constant propagation, CSE) (Vern Paxson, Corelight)
* helper class checking if common-subexpression elimination opportunity is valid (Vern Paxson, Corelight)
* helper function for comparing collections of definition points (Vern Paxson, Corelight)
* track more information about temporary variables (Vern Paxson, Corelight)
* simplify and tidy up some script-opt interfaces (Vern Paxson, Corelight)
* enable setting "-O optimize-AST" option (Vern Paxson, Corelight)
* fix for reducing operations on boolean vectors (Vern Paxson, Corelight)
* Add flag to FuncType::CheckArgs() to control type-equivalence warning output (Vern Paxson, Corelight)
* remove unneeded IndexExpr::Traverse() virtual method (same as inherited method) (Vern Paxson, Corelight)
* accessors for additional Expr subclasses (Vern Paxson, Corelight)
* convenience functions to check for aggregate types (Vern Paxson, Corelight)
* fix header includes to have proper zeek/ prefixes (Vern Paxson, Corelight)
4.1.0-dev.288 | 2021-03-08 12:29:14 -0800
* GH-1426: Improve handling of Broker data store creation failures (Jon Siwek, Corelight)
Broker::create_master() and Broker::create_clone() now return
a valid value even when there's a failure to open the backend database
(e.g. SQLite filesystem error). In that case, the returned value can
still be passed into other data store operations, but they'll fail
immediately with an error. Broker::is_closed() can now also be used to
determine whether the data store creation calls failed.
4.1.0-dev.286 | 2021-03-08 11:24:38 +0000
* Allow non-TCP based protocols to use SSL analyzer. (Keith Jones)
4.1.0-dev.284 | 2021-03-05 17:37:03 -0800
* Simplify some copy_string() usages in threading::Field (Jon Siwek, Corelight)
* Use const-reference in plugin::Manager::MetaHookPost for minor performance gain (Tim Wojtulewicz, Corelight)
* Fix missing assigmnent operator/copy constructor pairings reported by LGTM (Tim Wojtulewicz, Corelight)
* Fix variable shadowing issues reported by LGTM (Tim Wojtulewicz, Corelight)
4.1.0-dev.277 | 2021-03-05 16:14:23 -0800
* Avoid reporting paths repeatedly in zeek-config --include_dir (Christian Kreibich, Corelight)
* Fix comment in IP analyzer (Tim Wojtulewicz, Corelight)
4.1.0-dev.272 | 2021-03-01 11:36:06 -0800
* Change update-zeekygen-docs.sh to set release branch in Sphinx config (Jon Siwek, Corelight)
Allows release branches to generate correct links to GitHub source code.
* Teach Zeekygen to produce source-code-range information (Jon Siwek, Corelight)
Related to https://github.com/zeek/zeek-docs/issues/56
* Add normalize_script_path() zeek::zeekygen::detail namespace (Jon Siwek, Corelight)
* Add starts_with()/ends_with() to zeek::util namespace (Jon Siwek, Corelight)
4.1.0-dev.266 | 2021-02-25 08:45:49 -0700
* Avoid superfluous string copies when adding to zeek::detail::sig_files
Coverity 1387015 (Jon Siwek, Corelight)
* Initialize an RD_Decorate member via std::move (Jon Siwek, Corelight)
* Fix invalid iterator comparison in UseDefs::FindSuccUsage()
Coverity 1447018 (Jon Siwek, Corelight)
4.1.0-dev.260 | 2021-02-24 15:52:31 -0800
* Remove obsolete ZEEK_FORWARD_DECLARE_NAMESPACED macros (Tim Wojtulewicz, Corelight)
4.1.0-dev.258 | 2021-02-24 15:10:51 -0700
* Minor Fedora 32/33 updates in CI
- Switch Fedora package management from yum to dnf, the default
- Minimize set of requested RPMs (Christian Kreibich, Corelight)
* Add CentOS Stream 8 to CI
There's no official Docker image on the Docker Hub, it's on quay.io:
https://quay.io/repository/centos/centos?tab=tags
At the moment, "Stream" and "Stream 8" are synonymous. (Christian Kreibich, Corelight)
4.1.0-dev.255 | 2021-02-24 12:35:47 -0800
* Support explicit disabling of file analyzers (Jon Siwek, Corelight)
This adds various methods/BIFs the enable/disable file analyzers
or query whether they're currently enabled.
4.1.0-dev.252 | 2021-02-23 12:01:52 -0800
* "xform" btest alternative baseline update (Vern Paxson, Corelight)
* Test baselines updates related to "unused assignment" warnings (Vern Paxson, Corelight)
* USE ZEEK_USAGE_ISSUES environment variable in tests rather than -u/-uu (Vern Paxson, Corelight)
Since it has precedence over the flag arguments and don't want btest
alternatives to override it.
* splitting out "usage" test suite alternative into -u/-uu versions (Vern Paxson, Corelight)
* adding &is_used attribute for base scripts (Vern Paxson, Corelight)
Not actually needed yet, but will be once optimization is added.
* removing unused assignments from base scripts (Vern Paxson, Corelight)
* enhancements/changes to the Reduce class in preparation for use-defs (Vern Paxson, Corelight)
* Add "-Odump-uds" and "ZEEK_DUMP_UDS" flag/environment variable for dumping use-defs (Vern Paxson, Corelight)
* Add &is_used attribute to suppress set-but-not-used usage warnings (Vern Paxson, Corelight)
* Support for "unused assignment" dead-code detection/warning via -u/-uu analysis (Vern Paxson, Corelight)
4.1.0-dev.229 | 2021-02-16 12:32:23 -0800
* Update Broker submodule for performance/credit-tracking fixes (Jon Siwek, Corelight)
4.1.0-dev.228 | 2021-02-16 11:10:40 -0800
* Fix `major_subsys_version` field in `pe_optional_header` event
It was incorrectly set the same as the `minor_subsys_version` field
of the `PE::OptionalHeader` record. (Jon Siwek, Corelight)
4.1.0-dev.225 | 2021-02-12 14:40:14 -0800
* Fix CentOS 8 CI Dockerfile (Jon Siwek, Corelight)
The "PowerTools" repoid changed to "powertools":
https://bugs.centos.org/view.php?id=17920
4.1.0-dev.224 | 2021-02-12 11:18:32 -0800
* GH-1398: Fix buffer overread in ascii formatter (Johanna Amann, Corelight)
When a text with an (escaped) zero byte was passed to ParseValue, only
the part of the string up to the zero byte was copied, but the length of
the full string was passed to the input framework.
This leads to the input manager reading over the end of the buffer.
4.1.0-dev.220 | 2021-02-11 11:10:46 -0800
* GH-1399: Remove RocksDB configure script options (Jon Siwek, Corelight)
4.1.0-dev.219 | 2021-02-11 10:56:16 -0800
* Add a script to cleanup the file listing before sending to Coveralls (Tim Wojtulewicz)
GCC has a "feature" where it doesn't realize that files in the
src/zeek/... tree are the same as files in the normal 'src/' tree. This
leads the coverage script to send duplicates to Coveralls and pollute
the display with them. The new script scrapes the intermediate output
from lcov and de-duplicates anything in src/zeek/ to be just from src/.
4.1.0-dev.216 | 2021-02-08 13:06:07 -0800
* Allow --parse-only to work with --usage-issues flag (Jon Siwek, Corelight)
Use of --parse-only would previously exit before --usage-issues had a
chance to analyze scripts and report any discovered issues and it can be
useful to combine both flags for checking scripts for mistakes without
actually executing any code.
This also improves the behavior of --parse-only in combination with
reporting problems in signature files (previously, it exited before
even reading them).
4.1.0-dev.212 | 2021-02-05 10:56:23 -0800
* Support for Zeek script "Reaching Definitions" (RD) analysis (Vern Paxson, Corelight)
RD analysis invovles tracking the extent to which a given variable
definition (assignment) can be visible elsewhere in the Zeek script. The
analysis works on local variables in function/event/hook bodies, but not
across them.
The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a
given point in a function body) and "maximal" (what possibly could reach).
Upcoming script optimization will use the latter, but the former currently
allows identification of places for which a value is used where it
does not appear that it will necessarily be defined. Specifying the
``zeek -u`` option will generate warnings for instances where this holds
for local variables. Specifying ``zeek -uu`` turns on additional
expensive analysis to report instances where record fields might be used
without having previously been set.
The base scripts have some places where static analysis lacks sufficient
power to tell that values are being used safely (guaranteed to have been
identified). In order to enable users to employ ``zeek -u`` on their own
scripts without being distracted by these instances, this change also
includes a new attribute, ``&is_assigned``, which can be associated with a
variable or a record field to inform Zeek's analysis that the script writer
asserts the value will be set, suppressing the associated warnings.
For the test suite, there's new ``btest -a usage`` alternative to exercise
both ``zeek -uu`` and ``&is_assigned`` functionality.
4.1.0-dev.173 | 2021-02-03 11:16:04 -0800
* GH-1389: Add VN-Tag packet analyzer that skips VN-Tag headers (Tim Wojtulewicz, Corelight)
4.1.0-dev.171 | 2021-02-03 09:53:12 -0700
* Prefix #includes of .bif.h files with zeek/
This enables locating the headers within the install-tree using the
dirs provided by `zeek-config --include_dir`.
To enable locating these headers within the build-tree, this change also
creates a 'build/src/include/zeek -> ..' symlink. (Jon Siwek, Corelight)
4.1.0-dev.169 | 2021-02-03 09:50:33 -0700
* Bump macOS jobs in CI from 4 to 12 cores
A recent Cirrus upgrade enables this and saves 4-5 minutes for the
build and test jobs (combined), for both Catalina and Big Sur. (Christian Kreibich, Corelight)
4.1.0-dev.165 | 2021-02-01 15:53:56 -0800
* Update documentation to include "Book of Zeek" revisions (Jon Siwek, Corelight)
4.1.0-dev.164 | 2021-02-01 12:13:38 -0800
* Deprecate zeekenv() and use getenv() directly (Jon Siwek, Corelight)
4.1.0-dev.162 | 2021-01-29 21:05:41 -0800
* GH-1377: Add missing zeek/ to header includes (Jon Siwek, Corelight)
4.1.0-dev.158 | 2021-01-29 16:37:56 -0800
* Fix language.init-in-anon-function btest due to changes to log filter predicates (Tim Wojtulewicz, Corelight)
* Remove deprecated log filter predicates for 4.1 (Christian Kreibich, Corelight)
Update the logging framework tests: since hooks operate
by name, they cannot be anonymous. I'm also dropping the &optional
attribute from the status field, since here know that the values are
actually defined, and access to an optional status field should
normally be guarded by the existence test operator.
* Remove deprecated Plugin::HookCallFunction (Tim Wojtulewicz, Corelight)
* Remove deprecated support for .bro script extension and BRO_ environment variables (Tim Wojtulewicz, Corelight)
* Remove deprecated ICMP events (Tim Wojtulewicz, Corelight)
* Remove some deprected methods/events from bif files (Tim Wojtulewicz, Corelight)
* Remove deprecated TYPE_COUNTER (Tim Wojtulewicz, Corelight)
* Remove all of the random single-file deprecations (Tim Wojtulewicz, Corelight)
* Remove all fully-deprecated files (Tim Wojtulewicz, Corelight)
4.1.0-dev.146 | 2021-01-29 10:20:28 -0800
* Skip negative integers in input framework where not type-permissible (Christian Kreibich, Corelight)
Script-layer counts, when provided as negative integers in an input
file, got cast to unsigned values because strtoull() does not complain
about negative values. For example, input string "-1" would lead to
value 18446744073709551615 (an all-ones 64-bit int) on x86_64. This is
more likely to be an error than an intent to get very large,
platform-dependent values, so these input lines are now skipped with
according messaging in the reporter.log/stderr.
This also affected ports: -1/tcp got cast to unsigned and only thrown
out because PortVal rejects values > 65535, mapping them to 0. We now
skip such inputs as well.
4.1.0-dev.144 | 2021-01-29 09:54:01 -0700
* Fix various broken links in script documentation (Jon Siwek, Corelight)
4.1.0-dev.141 | 2021-01-25 12:32:48 -0800
* Update broker submodule, embedded CAF to 0.18 (Jon Siwek, Corelight)
4.1.0-dev.137 | 2021-01-21 17:29:40 -0800
* Update SSL consts (Johanna Amann)
There have been a bunch of mostly TLS 1.3 related changes.
4.1.0-dev.135 | 2021-01-21 16:57:01 -0800
* GH-1352: Added flag to stop processing SMTP headers in attached messages (Jon Oakley)
Previously, header values from sub-entities could overwrite the top-level
MIME entity values, causing the SMTP log to contain misleading information.
4.1.0-dev.132 | 2021-01-21 13:59:35 -0800
* GH-1296: fix type-checks related to list-type equality (Jon Siwek, Corelight)
List-types as used in composite table/set indices, for example,
previously had incorrect same_type() comparisons due to flattening
of the list-type into a single type without checking whether the
number and kind of types all match.
This patch simply removes the flatten_type() call from same_type() since
it was already contradicting/preventing a subsequent full-comparison
between elements of two TYPE_LISTs.
There was also a superfluous special-case of the `in` operator's
type-checking for testing whether a record is in a table/set. It's
superfluous because the general case will already do the type-checking
from MatchesIndex() after first wrapping the record operand in a
ListExpr. The previous logic was incorrectly relying on the
flatten_type() for testing equality of a record-type against a
list-type, whereas the general case correctly normalizes to testing
equality of two list-types.
The special-cased type-checking logic for assigning a record value to a
table index during its initialization similarly needed minor
re-organization in order to maintain the same error messages as before.
* Guard against nullptr deref after AST-reduction consistency checks (Jon Siwek, Corelight)
4.1.0-dev.128 | 2021-01-21 13:37:10 -0800
* Install BTest with Zeek (Christian Kreibich, Corelight)
Configuring with --disable-btest skips the installation. The main
reason for doing this is that many Zeek packages come with testsuites
that nearly always require btest, so providing btest out of the box
makes sense. Note that zkg's recent additions to its PATH environment
variable management mean that this btest instance is automatically
found also when the Zeek installation's binary folder isn't in the
path.
4.1.0-dev.125 | 2021-01-21 11:27:14 +0000
* More precise type information in a config framework error message
When an option's value is a reader-incompatible table or set, Zeek now
renders the type as expressed in the script layer (e.g. "set[addr,addr]")
as opposed to the internal type tag (which'd here be "table", including
for sets). (Christian Kreibich, Corelight)
* Explicitly don't support sets with multiple index types in input/config frameworks
The input framework's Manager::IsCompatibleType() already rejected
sets with multiple index types that aren't all the same (i.e. that are
not pure). Pure ones (e.g. "set[addr,addr]") slipped through and could
cause Zeek to segfault elsewhere in the config framework due to type
comparison subtleties. Note that the ASCII reader can't read such sets
anyway, so this method now rejects sets with any kind of index-type
tuple.
In the config framework, the script-level change handler has a risky
conversion from any to set[bool], which can trigger segfaults when the
underlying set's index is a type tuple. We now prevent this code path
by ensuring it only applies to sets with a single index type.
(Christian Kreibich, Corelight)
* Fix logo/website link in README (Jon Siwek, Corelight)
4.1.0-dev.120 | 2021-01-19 21:08:11 -0800
* Update Cirrus CI naming for MacOS images (Christian Kreibich, Corelight)
The docs on https://cirrus-ci.org/guide/macOS/ changed from
"osx_instance" to "macos_instance", so let's reflect that.
4.1.0-dev.118 | 2021-01-18 19:14:30 -0700
* Remove superfluous nullptr checks in Val/RecordVal methods (Jon Siwek, Corelight)
4.1.0-dev.116 | 2021-01-14 19:29:58 -0800
* Implement standard-library-compatible iterators for Dictionary (Tim Wojtulewicz, Corelight)
4.1.0-dev.114 | 2021-01-14 15:00:48 -0800
* Add support for rewriting the ASTs for script functions in "reduced" form
In reduced ASTs, expressions have only variables or constants for operands
(no subexpressions - those are replaced by temporary variables). In
addition, some expressions are transformed into alternatives (new specialized
expressions, or in some cases new internal types of statements) to
more directly encapsulate their operation.
The functionality is enabled by using -O xform. You can dump the
before-and-after ASTs using -O dump-xform (very large output!). The new
command-line option --optimize-only=X restricts the processing to only the
function named X, and also automatically turns on dump-xform.
* Add new btest alternative for script transformation (xform) (Vern Paxson, Corelight)
* split bifs.string_utils into a non-error test and an only-errors test (Vern Paxson, Corelight)
To help control for differing error propagation
* Add convenience function, obj_desc(), for accessing object descriptions (Vern Paxson, Corelight)
* removed unused EventBodyList subclass (Vern Paxson, Corelight)
4.1.0-dev.82 | 2021-01-14 13:27:23 -0700
* Use static_cast instead of dynamic_cast for Val conversions (Tim Wojtulewicz)
* Remove GET_FIELD_AS macro, replace with template methods (Tim Wojtulewicz, Corelight)
* Add templated As() method to Val, use in various places we were using dynamic_cast (Tim Wojtulewicz, Corelight)
* Move ValImpl classes into detail namespace (Tim Wojtulewicz)
* Some review cleanup in Val.cc (Tim Wojtulewicz, Corelight)
* Changes addressing pending issues per PR discussion (Vern Paxson, Corelight)
* Remove BroValUnion by hoisting underlying Val subclass values into subclasses (Vern Paxson, Corelight)
4.1.0-dev.74 | 2021-01-14 11:57:48 -0700
* Use xcode 11 on the Catalina CI image (Tim Wojtulewicz, Corelight)
4.1.0-dev.71 | 2021-01-11 18:03:25 -0800
* Add []-style variable-capture-list for Zeek lambda functions (Vern Paxson, Corelight)
The previous behavior of automatically capturing references to variables
outside a lambda's scope is now deprecated. An explicit capture list which
also specifies the desired copy-semantics will be required when
writing lambda functions that refer to local variables of an outer scope.
Examples of the new capture-list syntax are described at
https://docs.zeek.org/en/master/script-reference/types.html#type-function
* nit: fixed some 0/1 values that should instead be false/true (Vern Paxson, Corelight)
* factored some complexity of begin_func() into static functions for clarity (Vern Paxson, Corelight)
* error propagation fix: don't complain about "unused" values that themselves are due to errors (Vern Paxson, Corelight)
* corrected & reflowed some comments, plus a whitespace tweak (Vern Paxson, Corelight)
4.1.0-dev.52 | 2021-01-11 11:11:13 -0800
* Remove unusable/broken RocksDB code and options (Jon Siwek, Corelight)
The Broker RockSDB data store backend was previously unusable
and broken, so all code and options related to it are now removed.
4.1.0-dev.51 | 2021-01-07 17:02:28 -0800
* Virtualize Obj::GetLocationInfo() (Vern Paxson, Corelight)
Cleaner approach for localizing errors associated with duplicated ASTs.
* Add support for inlining of Zeek script functions (Vern Paxson, Corelight)
* Add support for duplicating Zeek ASTS (Vern Paxson, Corelight)
* Update COPYING to 2021 (Johanna Amann, Corelight)
4.1.0-dev.27 | 2021-01-06 20:42:35 -0800
* GH-1347: Update cmake module to fix ZeekPluginDynamic's find_package(CAF) (Jon Siwek, Corelight)
4.1.0-dev.25 | 2020-12-28 15:37:20 -0700