-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGES
23509 lines (14865 loc) · 814 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
3.2.0-dev.874 | 2020-07-08 17:46:20 -0700
* scan.l: Remove "constant" did_module_restore logic / dead code (Arne Welzel)
3.2.0-dev.872 | 2020-07-08 12:25:11 -0700
* Fix FreeBSD CI script to install right SWIG package (Jon Siwek, Corelight)
3.2.0-dev.870 | 2020-07-08 15:41:15 +0000
* Fix WhileStmt to call Stmt(Tag) constructor. (Jon Siwek,
Corelight)
3.2.0-dev.868 | 2020-07-08 15:39:46 +0000
* GH-928: use realpath() instead of inode to de-duplicate scripts.
(Jon Siwek, Corelight)
3.2.0-dev.864 | 2020-07-06 21:06:41 -0700
* GH-1040: Add zero-indexed version of str_split (Tim Wojtulewicz, Corelight)
3.2.0-dev.862 | 2020-07-06 20:40:44 -0700
* GH-1041: Move compress_path to a bif that uses normalize_path (Tim Wojtulewicz, Corelight)
3.2.0-dev.859 | 2020-07-06 14:55:00 +0000
* Add backtrace() and print_backtrace() BIF functions. (Jon Siwek,
Corelight)
3.2.0-dev.855 | 2020-07-03 14:29:55 -0700
* Fix --enable-mobile-ipv6 build (Jon Siwek, Corelight)
3.2.0-dev.854 | 2020-07-03 14:17:42 -0700
* Fix namespace of GetCurrentLocation() to zeek::detail (Jon Siwek, Corelight)
3.2.0-dev.853 | 2020-07-02 19:24:09 -0700
* Rename BroString files to ZeekString (Tim Wojtulewicz, Corelight)
* Rename BroObj to Obj (Tim Wojtulewicz, Corelight)
* Rename BroString to zeek::String (Tim Wojtulewicz, Corelight)
* Move Func up to zeek namespace, rename BroFunc to ScriptFunc (Tim Wojtulewicz, Corelight)
* Mark global val_mgr as deprecated and fix uses of it to use namespaced version (Tim Wojtulewicz, Corelight)
* Update binpac and bifcl submodules with review changes (Tim Wojtulewicz, Corelight)
* Move Location to zeek::detail and BroObj to zeek (Tim Wojtulewicz, Corelight)
* Move BroString to zeek namespace (Tim Wojtulewicz, Corelight)
* Move Dictionary/PDict, List/PList, and Queue/PQueue to zeek namespace (Tim Wojtulewicz, Corelight)
* Remove typedef that should have been removed in 3.1 (Tim Wojtulewicz, Corelight)
* Move Func and associated classes into zeek::detail namespace (Tim Wojtulewicz, Corelight)
* Move Frame and Scope to zeek::detail namespace (Tim Wojtulewicz)
* Move all Val classes to the zeek namespaces (Tim Wojtulewicz, Corelight)
* Use type aliases for IntrusivePtr definitions (Tim Wojtulewicz, Corelight)
* Move deprecation macro to zeek-config.h.in to avoid having to over-include util.h (Tim Wojtulewicz, Corelight)
* Move IntrusivePtr and utility methods to the zeek namespace (Tim Wojtulewicz, Corelight)
3.2.0-dev.834 | 2020-07-02 15:57:09 -0700
* GH-786: fix Val::As methods casting incomplete types (Jon Siwek, Corelight)
* GH-786: fix Stmt::As methods casting incomplete types (Jon Siwek, Corelight)
* GH-786: fix Expr::As methods casting incomplete types (Jon Siwek, Corelight)
3.2.0-dev.830 | 2020-07-02 11:36:28 -0700
* GH-786: Move Type::As methods to cc file so they have type info for casting safely (Tim Wojtulewicz, Corelight)
3.2.0-dev.828 | 2020-07-01 09:57:23 -0700
* Fix a deprecation warning compiling with GCC (Jon Siwek, Corelight)
3.2.0-dev.827 | 2020-06-30 16:54:22 -0700
* Deprecate plugin::HookType and plugin::component::Type in a different way (Tim Wojtulewicz, Corelight)
* Deprecate init_class and IDScope in another way. (Tim Wojtulewicz, Corelight)
This also renamed init_class to InitClass for consistency.
* Deprecate TypeTag and friends in a different way (Tim Wojtulewicz, Corelight)
* Deprecate attr_tag in a different way, rename to AttrTag (Tim Wojtulewicz, Corelight)
3.2.0-dev.822 | 2020-06-30 15:54:38 -0700
* Revert Attributes::Attrs back to return an attr_list and mark it deprecated (Tim Wojtulewicz, Corelight)
* GH-1034: Revert TypeList::Types() back to return a type_list* and mark it deprecated (Tim Wojtulewicz, Corelight)
3.2.0-dev.817 | 2020-06-29 19:23:21 +0000
* GH-1036: change print.log to log network time instead of current (Jon Siwek, Corelight)
3.2.0-dev.815 | 2020-06-29 10:25:05 -0700
* reduce memory usage of ConnPolling (Justin Azoff)
Instead of scheduling the event with the full 'connection' record,
schedule it with the smaller 'conn_id' record.
3.2.0-dev.813 | 2020-06-26 16:25:34 +0000
* Fix several issues with command-line option redefs
* Variables of `string` type can now be set to an empty string
* Trying to set a variable with non-`string` type to an empty value
now emits an error instead of silently doing nothing
* Providing an invalid identifier now emits an "unknown identifier"
error instead of silently doing nothing (Jon Siwek, Corelight)
* Fix "possibly-truncated" compiler warning in BuildJSON snprintf() (Jon Siwek, Corelight)
* GH-1025: allow copying/cloning of `opaque of Broker::Store`
Implemented simply as a reference count increment of the
data store handle. (Jon Siwek, Corelight)
* Fix shadowed variable that breaks lookup_hostname(). (Jon Siwek, Corelight)
3.2.0-dev.804 | 2020-06-25 23:58:59 -0700
* Extend dns_request, dns_reject, and dns_query_reply events with original_query param (Ryan Victory)
Where the "query" parameter is downcased, the "original_query" is
passed unmodified to these events.
Added a policy/protocols/dns/log-original-query-case.zeek script that adds
the original_query to the dns.log file as well.
* Convert to using permissions to check for access to cirrus variables in benchmark script (Tim Wojtulewicz, Corelight)
3.2.0-dev.798 | 2020-06-19 13:25:02 -0700
* Remove last_access_time from TableEntryVal. (Johanna Amann, Corelight)
Turns out - this was no longer used. And it takes up at least 8 bytes
of space for every single table/set entry.
3.2.0-dev.796 | 2020-06-18 20:40:59 +0000
* cmake: Make musl support more distro agnostic (Andrew Benson)
3.2.0-dev.794 | 2020-06-18 20:04:06 +0000
* Minimize data published for expected FTP data channel analysis
Previously, more data than could effectively be utilized by any remote
Zeek was published (e.g. full list of pending commands or other
transient state that may add up to non-trivial amount of bytes). (Jon Siwek, Corelight)
* GH-998: Fix Reporter::conn_weird() to handle expired connections
This introduces a new sampling state-map for expired connections to fix
segfaults that previously occured when passing in a `connection` record
to `Reporter::conn_weird()` for which the internal `Connection` object
had already been expired and deleted. This also introduces a new event
called `expired_conn_weird`, which is similar to `conn_weird`, except
the full `connection` record is no longer available, just the `conn_id`
and UID string. (Jon Siwek, Corelight)
* Place build file in explicit location for benchmarking to work correctly (Tim Wojtulewicz, Corelight)
* Update highwayhash submodule to upstream.
This includes a patch by us, which fixes the compile on AARCH64.
Fixes GH-1006. (Johanna Amann, Corelight)
3.2.0-dev.788 | 2020-06-15 11:04:20 -0700
* Fix location where CI places build.tgz (Jon Siwek, Corelight)
3.2.0-dev.787 | 2020-06-15 10:27:48 -0700
* Add new CI task for running benchmarks on a remote host (Tim Wojtulewicz, Corelight)
3.2.0-dev.783 | 2020-06-11 23:21:41 -0700
* Compare pcap_next_ex() result to PCAP_ERROR/PCAP_ERROR_BREAK (Jon Siwek, Corelight)
* GH-977: Improve pcap error handling (Jon Siwek, Corelight)
Switches from pcap_next() to pcap_next_ex() to better handle all error
conditions. This allows, for example, to have a non-zero exit code for
a Zeek process that fails to fully process all packets in a pcap file.
* Remove not-useful code in iosource::Manager::OpenPktSrc (Jon Siwek, Corelight)
3.2.0-dev.779 | 2020-06-11 23:17:46 -0700
* Rename BroType to zeek::Type (Tim Wojtulewicz, Corelight)
* Move Type types to zeek namespace (Tim Wojtulewicz, Corelight)
* Move Flare/Pipe from the bro namespace to zeek::detail (Tim Wojtulewicz, Corelight)
* Move Attr to the zeek::detail namespace (Tim Wojtulewicz, Corelight)
* Move Trigger into the zeek::detail namespace (Tim Wojtulewicz, Corelight)
* Move ID to the zeek::detail namespace (Tim Wojtulewicz, Corelight)
* Move Anon.h into zeek::detail namespace (Tim Wojtulewicz, Corelight)
* Mark all of the aliased classes in plugin/Plugin.h deprecated (Tim Wojtulewicz, Corelight)
And fix all of the plugins that were using them
* Move all of the base plugin classes into the zeek::plugin namespace (Tim Wojtulewicz, Corelight)
* Expr: move all classes into zeek::detail (Tim Wojtulewicz, Corelight)
* Stmt: move Stmt classes into zeek::detail namespace (Tim Wojtulewicz, Corelight)
* Add utility macro for creating namespaced aliases for classes (Tim Wojtulewicz, Corelight)
3.2.0-dev.763 | 2020-06-10 16:34:31 -0700
* Optimize Connection::RemovalEvent() for bare-mode usage (Jon Siwek, Corelight)
It was creating RecordVals even if they wouldn't be used by any event
handler and that situation is common/expected for `zeek -b` mode.
Normally, there's at least the tunnel scripts with a `new_connection`
handler causing the connection RecordVals to be built.
3.2.0-dev.762 | 2020-06-09 15:19:15 -0700
* Restrict Cirrus CI to only zeek repo's branches (Jon Siwek, Corelight)
3.2.0-dev.761 | 2020-06-09 09:39:31 -0700
* Prevent IP fragment reassembly on packets without minimal IP header (Jon Siwek, Corelight)
The IP fragment reassembly process assumes a packet contains at least
the minimum IP header, but such a check did not previously occur,
resulting in a heap buffer over-read. For example, a self-reported
IPv4 IHL field with a value less than minimum IPv4 header length of
20 bytes. Such packets likely aren't routable on their own, but one
can create an artifical pcap like that or possibly encapsulate it
within another protocol to trigger this bug.
* Fix potential leak of Analyzers added to tree during Analyzer::Done (Jon Siwek, Corelight)
Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22630
* Fix multiple content-transfer-encoding headers causing a memory leak (Jon Siwek, Corelight)
The MIME analyzer may also leak if it sees many multipart boundary
parameters.
Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22871
* Fix potential stack overflow in NVT analyzer (Jon Siwek, Corelight)
The NVT_Analyzer (e.g. as instantiated to support the FTP analyzer)
uses a recursive parsing function that may only advance one byte at a
time and can easily cause a stack overflow as a result. This change
replaces the recursive calls with equivalent iterative logic.
Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22898
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22972
* Fix NVT analyzer memory leak from multiple telnet authn name options (Jon Siwek, Corelight)
Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23069
3.2.0-dev.751 | 2020-06-08 22:30:18 +0000
* GH-999: Stop formatting DHCP Client ID Hardware Type 0 as MAC
For `DHCP::ClientID$hwtype` fields equal to 0, the `hwaddr` field is
no longer misformatted as a MAC and instead just contains the raw bytes
seen in the DHCP Client ID Option. (Jon Siwek, Corelight)
3.2.0-dev.749 | 2020-06-08 11:22:45 -0700
* Remove inline from some static KeyedHash members (Jon Siwek, Corelight)
Coverity Scan builds currently encounter catastrophic error, claiming
alignas requires use on both declaration and definition, so appears to
actually not understand "static inline" in combo with alignas.
* Improve Func.h inclusion (Jon Siwek, Corelight)
Now forward declares some Broker types since Broker/CAF headers
generally slow things down and also Coverity Scan currently has a
catastrophic error on some CAF headers.
Also a few other changes to EventHandler/BifReturnVal to reduce number
of places that depend on Func.h.
* Rename aux/ to auxil/ (Jon Siwek, Corelight)
Since "aux" is not an allowed file/dir name on Windows.
3.2.0-dev.744 | 2020-06-04 15:11:56 -0700
* Fix use-after-move of proc_status_file breaking -U flag (Jon Siwek, Corelight)
* Change Cirrus CI config to use org-level secured variable (Jon Siwek, Corelight)
3.2.0-dev.742 | 2020-06-03 13:29:12 -0700
* GH-973: Fix the return type of topk_get_top()
e.g. when using `string` elements it should return `vector of string`,
not `vector of list of string` like it did before. (Jon Siwek, Corelight)
3.2.0-dev.740 | 2020-06-03 13:17:26 -0700
* Deprecate Val(double, TypeTag) ctor, add TimeVal/DoubleVal subclasses
This also updates all usages of the deprecated Val ctor to use
either IntervalVal, TimeVal, or DoubleVal ctors. The reason for
doing away with the old constructor is that using it with TYPE_INTERVAL
isn't strictly correct since there exists a more specific subclass,
IntervalVal, with overriden ValDescribe() method that ought to be used
to print such values in a more descriptive way. (Jon Siwek, Corelight)
3.2.0-dev.738 | 2020-06-02 18:13:50 -0700
* GH-985: Fix descriptions of double_to_interval() return values (Jon Siwek, Corelight)
The BIF was not returning an IntervalVal which has an overriden
ValDescribe() method that allows for prettier printing like "6.0 secs"
instead of just "6.0".
3.2.0-dev.736 | 2020-06-02 12:37:56 -0700
* Decrease number of CPUs/memory for Cirrus CI tasks (Jon Siwek, Corelight)
3.2.0-dev.735 | 2020-06-02 10:55:43 -0700
* Change CI script to compile from build/Makefile (Jon Siwek, Corelight)
3.2.0-dev.734 | 2020-06-01 20:44:29 -0700
* Add missing include in util.cc (Jon Siwek, Corelight)
3.2.0-dev.733 | 2020-06-01 19:25:37 -0700
* Remove Analyzer.h from bro-bif.h (Tim Wojtulewicz, Corelight)
* Remove IPAddr.h from Reporter.h (Tim Wojtulewicz, Corelight)
* Remove the inclusion of Func.h from NetVar.h (Tim Wojtulewicz, Corelight)
3.2.0-dev.727 | 2020-06-01 10:57:16 -0700
* Integrate review feedback
* Add deprecation for MIME_Entity::ContentType(), use GetContentType()
* Add deprecation for MIME_Entity::ContentSubType(), use GetContentSubType()
* Add deprecation for MIME_Message::BuildHeaderVal(), use ToHeaderVal()
* Add deprecation for MIME_Message::BuildHeaderTable(), use ToHeaderTable()
* Add deprecation for mime::new_string_val(), use mime::to_stringval()
* Add deprecation for ARP_Analyzer::ConstructAddrVal(), use ToAddrVal()
* Add deprecation for ARP_Analyzer::EthAddrToStr(), use ToEthAddrStr()
* Change the Func::Call() replacement to be named Func::Invoke() (Jon Siwek, Corelight)
* Switch Broker Val converter visitor to return IntrusivePtr (Jon Siwek, Corelight)
* Change BroFunc ctor to take const-ref IntrusivePtr<ID> (Jon Siwek, Corelight)
* Add version of Frame::SetElement() taking IntrusivePtr<ID>
Expect the version using raw ID* could go away eventually, but this is
convenience for the meantime. (Jon Siwek, Corelight)
* Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>> (Jon Siwek, Corelight)
* Change Scope::GenerateTemporary() to return IntrusivePtr (Jon Siwek, Corelight)
* Deprecate Scope::ReturnType(), replace with GetReturnType() (Jon Siwek, Corelight)
* Deprecate Scope::ScopeID(), replace with GetID() (Jon Siwek, Corelight)
* Switch parsing to use vector<IntrusivePtr<Attr>> from attr_list
This allows improved passing/storing of Attr references to Exprs,
TypeDecl, Scope, etc. (Jon Siwek, Corelight)
* Deprecate TableVal::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight)
* Deprecate TypeDecl::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight)
* Deprecate ID::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight)
* Deprecate Attributes::FindAttr(), replace with Find() (Jon Siwek, Corelight)
* Deprecate Attributes::AddAttrs(Attributes*)
Replaced with version taking an IntrusivePtr parameter (Jon Siwek, Corelight)
* Add Attributes ctor that takes IntrusivePtrs (Jon Siwek, Corelight)
* Change Attributes to store std:vector<IntrusivePtr<Attr>>
This also changes the return type of Attributes::Attrs() from attr_list* (Jon Siwek, Corelight)
* Change Attr::SetAttrExpr() to non-template (Jon Siwek, Corelight)
* Deprecate Attr::AttrExpr(), replace with GetExpr() (Jon Siwek, Corelight)
* Deprecate ID::Attrs(), replace with GetAttrs() (Jon Siwek, Corelight)
* Remove weak_ref param from ID::SetVal()
It was not used anywhere. (Jon Siwek, Corelight)
* Store weak ref boolean along with Frame element Vals (Jon Siwek, Corelight)
* Deprecate Frame::GetElement(ID*), replace with GetElementByID() (Jon Siwek, Corelight)
* Deprecate Frame::NthElement(), replace with GetElement() (Jon Siwek, Corelight)
* Switch Frame::SetElement() to use IntrusivePtr (Jon Siwek, Corelight)
* Store IntrusivePtrs in Frame (Jon Siwek, Corelight)
* Deprecate Plugin::HookCallFunction(), replace with HookFunctionCall()
This also changes the argument type of Func::operator() to zeek::Args*
to allow plugins to be able to alter function arguments in place as
was previously documented. (Jon Siwek, Corelight)
* Switch plugin::Manager::HookCallFunction() to return IntrusivePtr
The plugin::Plugin side of things is not (yet) changed. (Jon Siwek, Corelight)
* Port remaining file analysis API to use IntrusivePtr (Jon Siwek, Corelight)
* Deprecate file analyzer construction methods taking raw RecordVal*
Replaced with versions that instead take IntrusivePtr (Jon Siwek, Corelight)
* Deprecate file_analysis::Analyzer::Args(), replace with GetArgs() (Jon Siwek, Corelight)
* Deprecate file_analysis::File::GetVal(), replace with ToVal() (Jon Siwek, Corelight)
* Change file_analysis::File::GetID() to return const-ref (Jon Siwek, Corelight)
* Fix build: some don't like IntrusivePtr default arg w/ incomplete type (Jon Siwek, Corelight)
* Change bro_broker::DataVal::ScriptDataType() to IntrusivePtr (Jon Siwek, Corelight)
* Change find_nested_record_types() to take IntrusivePtr (Jon Siwek, Corelight)
* Deprecate TypeType::Type(), replace with GetType() (Jon Siwek, Corelight)
* Add is_assignable() overload taking TypeTag (Jon Siwek, Corelight)
* Add is_atomic_type() overloads for IntrusivePtr (Jon Siwek, Corelight)
* Add same_type() overloads for IntrusivePtr args (Jon Siwek, Corelight)
* Change merge_types() to take IntrusivePtrs (Jon Siwek, Corelight)
* Deprecate IndexType::Indices(), replace with GetIndices() (Jon Siwek, Corelight)
* Add is_vector() methods taking const-ref IntrusivePtr (Jon Siwek, Corelight)
* Factor static-local nil IntrusivePtrs to global locations
Minor optimization to remove any run-time impact. (Jon Siwek, Corelight)
* Deprecate VectorVal::Lookup(), replace with At() (Jon Siwek, Corelight)
* Switch VectorVal BroValUnion to store std::vector<IntrusivePtr<Val>>
This changes the return type of AsVector() from std::vector<Val*>* (Jon Siwek, Corelight)
* Deprecate VectorVal::Insert() taking raw Val*, use IntrusivePtr (Jon Siwek, Corelight)
* Deprecate VectorVal::Assign methods taking raw Val*
And adapt usages to pass in to alternate method taking IntrusivePtr (Jon Siwek, Corelight)
* Merge remote-tracking branch 'origin/master' into topic/jsiwek/gh-893-intrusive-ptr-migration (Jon Siwek, Corelight)
* Change EventHandler to store IntrusivePtr<Func>
Also deprecates the LocalHandler() and SetLocalHandler() methods,
replaced with GetFunc() and SetFunc(). (Jon Siwek, Corelight)
* Add Val::AsFuncPtr() convenience method
Since it's not trivial to store IntrusivePtr in BroValUnion and also
not satisfying to store IntrusivePtr*. (Jon Siwek, Corelight)
* Deprecate StringVal::Substitute(), replace with Replace() (Jon Siwek, Corelight)
* Switch a TableVal::CallChangeFunc param to IntrusivePtr (Jon Siwek, Corelight)
* Fix ambiguous ODesc::Add() call (Jon Siwek, Corelight)
* Minor TableVal::Assign() ref-counting optimization (Jon Siwek, Corelight)
* Improve TableVal HashKey management
* Deprecated ComputeHash() methods and replaced with MakeHashKey()
which returns std::unique_ptr<HashKey>
* Deprecated RecoverIndex() and replaced with RecreateIndex()
which takes HashKey& and returns IntrusivePtr.
* Updated the new TableVal Assign()/Remove() methods to take either
std::unique_ptr<HashKey> or HashKey& as appropriate for clarity of
ownership expectations. (Jon Siwek, Corelight)
* Add back in a deprecated TableVal ctor taking raw pointers (Jon Siwek, Corelight)
* Deprecate TableVal::Attrs(), replace with GetAttrs() (Jon Siwek, Corelight)
* Deprecate TableVal::Delete(), replace with Remove() (Jon Siwek, Corelight)
* Switch Func::Call(val_list*) back to returning Val*
And renamed the method returning IntrusivePtr to operator().
This corrects the deprecation process for Func::Call(val_list*). (Jon Siwek, Corelight)
* Deprecate TableVal::Lookup(), replace with Find()/FindOrDefault() (Jon Siwek, Corelight)
* Deprecate TableVal IsSubsetOf and EqualTo taking Val*, use Val& (Jon Siwek, Corelight)
* Deprecate ComputeHash(Val*) methods, replace with ComputeHash(Val&) (Jon Siwek, Corelight)
* Deprecate TableVal::Intersect(), replace with Intersection() (Jon Siwek, Corelight)
* Deprecate TableVal::Assign methods with Val*, add IntrusivePtr overloads (Jon Siwek, Corelight)
* Deprecate RecordVal::Lookup(const char*, bool)
Replace with GetField(const char*) and GetFieldOrDefault(const char*). (Jon Siwek, Corelight)
* Deprecate RecordVal::LookupWithDefault() replace with GetFieldOrDefault()
(The former was previously changed during this release cycle to return
Intrusive pointer, but this just changes it back to return Val* and
deprecates it). (Jon Siwek, Corelight)
* Deprecate RecordVal::Lookup(int), replace with GetField(int) (Jon Siwek, Corelight)
* Change BroValUnion to use IntrusivePtr for record field storage
This also changes the AsRecord() and AsNonConstRecord() accessors
to return std::vector<IntrusivePtr<Val>>* instead of val_list* (Jon Siwek, Corelight)
* Switch ASN1 Val conversion functions to return IntrusivePtr (Jon Siwek, Corelight)
* Deprecate RecordVal::Assign(int, Val*)
And adapt all usages to the existing overload taking IntrusivePtr. (Jon Siwek, Corelight)
* Switch RPC analyzers to use IntrusivePtr (Jon Siwek, Corelight)
* Switch RecordVal::CoerceTo() to use IntrusivePtr (Jon Siwek, Corelight)
* Deprecate TableEntryVal::Value(), replace with GetVal() (Jon Siwek, Corelight)
* Switch OpaqueVal::SerializeType() to IntrusivePtr (Jon Siwek, Corelight)
* Switch BlommFilterVal/CardinalityVal to use IntrusivePtr<BroType> (Jon Siwek, Corelight)
* Switch some TopkVal methods to use IntrusivePtr (Jon Siwek, Corelight)
* Switch TopkVal to store IntrusivePtr<BroType> (Jon Siwek, Corelight)
* Switch OpaqueVal::UnserializeType() to return IntrusivePtr (Jon Siwek, Corelight)
* Deprecate FuncType::ArgTypes(), replace with ParamList() (Jon Siwek, Corelight)
* Deprecate FuncType::Args(), replace with Params() (Jon Siwek, Corelight)
* Deprecate EventHandler::FType(), replace with GetType() (Jon Siwek, Corelight)
* Deprecate BroFile::FType(), replace with GetType() (Jon Siwek, Corelight)
* Deprecate Func::FType(), replace with Func::GetType() (Jon Siwek, Corelight)
* Change {Get,Set}ContentsFile() to use IntrusivePtr (Jon Siwek, Corelight)
* Deprecate BroFile::GetFile(), replace with BroFile::Get() (Jon Siwek, Corelight)
* Deprecate Val(BroFile*) ctor, replace with one using IntrusivePtr (Jon Siwek, Corelight)
* Deprecate Val(Func*) ctor, replace with one using IntrusivePtr (Jon Siwek, Corelight)
* Deprecate OpaqueVal/HashVal ctors that take OpaqueType*
Replaced with ones that take IntrusivePtr (Jon Siwek, Corelight)
* fixup! Deprecate Scope::Lookup(), replace with Scope::Find() (Jon Siwek, Corelight)
* Switch EnumType::GetVal() to return const-ref (Jon Siwek, Corelight)
* Change protected EnumVal ctor to use IntrusivePtr (Jon Siwek, Corelight)
* Deprecate RecordVal(RecordType*) ctor
Replaced with one that takes IntrusivePtr (Jon Siwek, Corelight)
* Switch zeek::id::lookup to zeek::id::find
For parity with Scope since it now uses Find instead of Lookup (Jon Siwek, Corelight)
* Use zeek::BifEvent:: for enqueue_ functions instead of BifEvent:: (Jon Siwek, Corelight)
* Replace deprecated usage of BifFunc:: with zeek::BifFunc::
Names of functions also changed slightly, like bro_fmt -> fmt_bif.
Should generally be unusual/unexpected to see somone calling these
directly from C++ in their plugin, but since technically possible in
previous versions, I also removed the "private" restriction on accessing
the BifReturnVal member. (Jon Siwek, Corelight)
* Deprecate names in BifConst, replace with zeek::BifConst
Some Val* types are also replaced with IntrusivePtr at the new location (Jon Siwek, Corelight)
* Deprecate all BroType* in BifType:: namespace
Replaced with equivalently named IntrusivePtr in zeek::BifType:: (Jon Siwek, Corelight)
* Change EventRegistry/EventHandler methods to use std::string{_view} (Jon Siwek, Corelight)
* Deprecate internal_handler(), replace with EventRegistry::Register()
Added a couple explicit event declarations that were missing: "net_done"
and "dns_mapping_name_changed". (Jon Siwek, Corelight)
* Deprecate remaining "opt_internal" functions in Var.h (Jon Siwek, Corelight)
* Remove signal_val declaration from Var.h (Jon Siwek, Corelight)
* Change zeek::id::lookup functions to use std::string_view (Jon Siwek, Corelight)
* Change Scope::Find() and Scope::Remove() to use std::string_view (Jon Siwek, Corelight)
* Change lookup_ID() to return a const-reference (Jon Siwek, Corelight)
* Deprecate Scope::Lookup(), replace with Scope::Find() (Jon Siwek, Corelight)
* Move various elements into ID.h and zeek::id namespace
* A handful of generic/useful/common global type pointers that used
to be in NetVar.h
* Lookup functions that used to be Var.h (Jon Siwek, Corelight)
* Trim the list of "global type pointers" from NetVar.h further
Most of them are deprecated now, with usage sites now doing the lookup
themselves. (Jon Siwek, Corelight)
* Deprecate global Val pointers in NetVar.h
All of these have fairly niche uses, so better maintained as
lookup/static closer to the usage site. (Jon Siwek, Corelight)
* Deprecate global type pointers in NetVar.h
There's analogous IntrusivePtrs in zeek::vars (Jon Siwek, Corelight)
* Add RecordVal ctor that takes IntrusivePtr (Jon Siwek, Corelight)
* Remove unused FlattenExpr (Jon Siwek, Corelight)
* Add missing "vector_coerce" to expr_name() (Jon Siwek, Corelight)
* Deprecate Expr::Type(), replace with GetType() (Jon Siwek, Corelight)
* Deprecate Val::Type(), replace with GetType() (Jon Siwek, Corelight)
* Change Val to store IntrusivePtr (Jon Siwek, Corelight)
* Remove VectorVal::vector_type member (Jon Siwek, Corelight)
* Deprecate VectorVal(VectorType*) ctora
Adds a new one taking an IntrusivePtr. (Jon Siwek, Corelight)
* Deprecate internal_list_val() (Jon Siwek, Corelight)
* Deprecate opt_internal_val() (Jon Siwek, Corelight)
* Deprecate internal_func(), replace with zeek::lookup_func() (Jon Siwek, Corelight)
* Deprecate internal_val() and internal_const_val()
Replaced with zeek::lookup_val() and zeek::lookup_const() (Jon Siwek, Corelight)
* Deprecate internal_type(), replace with zeek::lookup_type() (Jon Siwek, Corelight)
* Deprecate ID::ID_Val(), replace with ID::GetVal() (Jon Siwek, Corelight)
* Use std::move in some zeekygen::Manager methods (Jon Siwek, Corelight)
* Add Val TypeType constructor taking an IntrusivePtr (Jon Siwek, Corelight)
* Deprecate ID::Type(), replace with GetType() (Jon Siwek, Corelight)
* Deprecate ID::AsType(), add ID::IsType() and ID::GetType() (Jon Siwek, Corelight)
* Deprecate BroType::YieldType(), replace with Yield() (Jon Siwek, Corelight)
* Change base_type() to return const-ref, deprecate base_type_no_ref() (Jon Siwek, Corelight)
* Deprecate BroType::GetField() and BroType::HasField() (Jon Siwek, Corelight)
* Deprecate RecordType::FieldType(), replace with GetFieldType() (Jon Siwek, Corelight)
* Migrate TypeList to store IntrusivePtrs
This changes return types of TypeList::Types() and
IndexType::IndexTypes() to return std::vector instead of type_list* (Jon Siwek, Corelight)
* Deprecate TypeList::PureType(), replace with TypeList::GetPureType() (Jon Siwek, Corelight)
* Deprecate SetType::SetElements(), replace with SetType::Elements() (Jon Siwek, Corelight)
* Remove unused TableType::ExpandRecordIndex() (Jon Siwek, Corelight)
* Add cast_intrusive() and make use of it in two spots (Jon Siwek, Corelight)
* Give make_intrusive() access to protected EnumVal ctor (Jon Siwek, Corelight)
* Remove two superfluous IntrusivePtr NewRefs in Type.cc (Jon Siwek, Corelight)
* Change BroType::ShallowClone() to return IntrusivePtr (Jon Siwek, Corelight)
* Migrate IP.cc to use IntrusivePtr (Jon Siwek, Corelight)
* Migrate SMB analyzer to use IntrusivePtr
Deprecates the utf16_bytestring_to_utf8_val() function with replacement
being utf16_to_utf8_val(). (Jon Siwek, Corelight)
* Migrate ARP analyzer to use IntrusivePtr (Jon Siwek, Corelight)
* Migrate HTTP/MIME analyzers to use IntrusivePtr (Jon Siwek, Corelight)
* Migrate ICMP analyzer to use IntrusivePtr (Jon Siwek, Corelight)
* Migrate DNS analyzer to use IntrusivePtr (Jon Siwek, Corelight)
* Migrate Tag classes to use IntrusivePtr
Deprecates various methods that previously took raw pointers (Jon Siwek, Corelight)
* Deprecate TableVal::ConvertToList() and TableVal::ConvertToPureList()
Replaced with ToListVal() and ToPureListVal() that return IntrusivePtr (Jon Siwek, Corelight)
* Switch ListVal to store IntrusivePtrs
* Deprecates ListVal::Index() methods and replaces with ListVal::Idx()
* Replaces ListVal::Vals() method with one that returns
std::vector<IntrusivePtr<Val>> rather than val_list (Jon Siwek, Corelight)
* Deprecant ListVal::Append(Val*) and add IntrusivePtr version (Jon Siwek, Corelight)
* Deprecate ListVal::ConvertToSet(), add ListVal::ToSetVal() (Jon Siwek, Corelight)
* Deprecate TunnelEncapsulation BuildRecordVal/BuildVectorVal methods
Replaced with ToVal methods that return IntrusivePtr (Jon Siwek, Corelight)
* Deprecate various IP/packet header Val-building methods
And supply new alternatives that use IntrusivePtr (Jon Siwek, Corelight)
* Update submodule(s)
[nomail] (Jon Siwek, Corelight)
3.2.0-dev.573 | 2020-05-29 17:13:36 -0700
* Add "Known::service_udp_requires_response" option (Jon Siwek, Corelight)
Determines whether to require UDP server response before considering
an active service to log in known_services.log.
* Update known-services.zeek logic for "active" TCP services (Jon Siwek, Corelight)
To better check for only endpoints that either have observed a TCP handshake
or else assumed to have done one in the past (partial connections, missed
the handshake, but now see data/acks from server)
* Log services with unknown protocol names (Michael Dopheide)
* GH-989: Fix crash on using some deprecated environment variables (Jon Siwek, Corelight)
3.2.0-dev.570 | 2020-05-28 14:49:10 -0700
* Add negotiation flags parameter to some RDP events (Anthony Kasza, Corelight)
Namely rdp_connect_request, rdp_negotiation_response, and rdp_negotiation_failure
* Update test baselines for new Broker connection status/error strings (Jon Siwek, Corelight)
3.2.0-dev.565 | 2020-05-26 21:55:54 +0000
* Add DCE-RPC constants from BZAR project (V)
3.2.0-dev.562 | 2020-05-26 11:06:34 -0700
* GH-979: Update libkqueue to fix use on CentOS 6 (Jon Siwek, Corelight)
3.2.0-dev.561 | 2020-05-21 13:05:39 -0700
* GH-983: Fix opaque Broker types lacking a Type after (de)serialization (Jon Siwek, Corelight)
3.2.0-dev.559 | 2020-05-21 13:04:19 -0700
* Make SendEvent callable from all threads
This commit refactors the SendEvent call and moves it from the Input
ReaderBackend to to MsgThread. This allows all other types of threads
to access this functionality.
This necessitated a few more changes. Most importantly, one of the
ValueToVal methods was moved over to SerialTypes. Whereit arguably
belongs - there was nothing that was input-framework specific in
that method - and the functionality could come in useful in a number
of cases. (Johanna Amann, Corelight)
3.2.0-dev.557 | 2020-05-21 11:41:12 -0700
* Speed up FuzzBuffer ChunkCount validity check (Justin Azoff, Corelight)
3.2.0-dev.555 | 2020-05-20 11:19:08 -0700
* Disable output of Reporter messages to stderr in fuzz targets (Jon Siwek, Corelight)
3.2.0-dev.554 | 2020-05-20 10:56:46 -0700
* Improve standalone fuzz driver timing output (Jon Siwek, Corelight)
* Skip fuzz inputs that have more than 64 chunks (Justin Azoff, Corelight)
3.2.0-dev.550 | 2020-05-19 10:50:42 -0700
* Upgrade to latest Broker changes for CAF 0.18 (Dominik Charousset, Corelight)
* Include pcap.h instead of pcap/dlt.h in packet-fuzzer (Tim Wojtulewicz, Corelight)
3.2.0-dev.547 | 2020-05-18 10:47:54 -0700
* add packet fuzzer (Justin Azoff)
* Fix building fuzz targets on macOS (Jon Siwek, Corelight)
* Highwayhash: small build fix (Johanna Amann, Corelight)
Turns out that hh_neon should not be compiled on generic arm CPUs.
That one is only for aarch64.
3.2.0-dev.539 | 2020-05-15 19:47:55 +0000
* Replace bzero() with memset() (Noah Treuhaft)
* Change Coverity action to output build log (Jon Siwek, Corelight)
3.2.0-dev.530 | 2020-05-13 15:05:31 -0700
* GH-906: Fix the regex in url.zeek to better match for find_all_urls (Tim Wojtulewicz, Corelight)
3.2.0-dev.526 | 2020-05-13 13:49:29 -0700
* Hash unification: address PR feedback (Johanna Amann, Corelight)
* Update architecture checks for highwayhash
Now we use cmake to determine which compiler flags are set to
determine which implementation of highwayhash to compile. This should
be much more robust.
Also - fix missing include on old ubuntus. (Johanna Amann, Corelight)
* Highwayhas: unbreak 32 bit compiles (Johanna Amann, Corelight)
* Update external test suites for hash changes. (Johanna Amann, Corelight)
* Hashing: Remove unnecessary include (Johanna Amann, Corelight)
* Merge remote-tracking branch 'origin/master' into topic/johanna/hash-unification (Johanna Amann, Corelight)
* Add highwayhash in a more cmake-y way (Johanna Amann, Corelight)
* Remove highwayhash submodule.
It is moving to 3rdparty... (Johanna Amann, Corelight)
* Merge remote-tracking branch 'origin/master' into topic/johanna/hash-unification (Johanna Amann, Corelight)
* Switch file UID hashing from md5 to highwayhash.
This commit switches UID hashing from md5 to a highway hash. It also
moves the salt value out of the file plugin - and makes it
installation-specific instead - it is moved to the global namespace.
There now are digest hash functions to make "static"
installation-specific hashes that are stable over workers available to
everyone; hashes can be 64, 128 or 256 bits in size.
Due to the fact that we switch the file hashing algorithm, all file
hashes change.
The underlyigng algorithm that is used for hashing is highwayhash-128,
which is significantly faster than md5. (Johanna Amann, Corelight)
* Switch most internal md5 calls to digest calls.
The places that used md5 basically already used it as a digest
algorithm. Switching to a digest just means that the internal values
used to not change between runs - which is actually wanted in these
cases.
This commit also removes our special cmake subdirectory. We don't expose
highwayhash in headers anymore - so we can just treat it as an internal
implementation choice that is not directly exposed to plugins. (Johanna Amann, Corelight)
* Start refactoring hashing.
This commit moves some of the hash datastructures and code from
util.cc into Hash.cc - where it seems more appropriate.
It also starts to make more Keyed hash functions available - still
using siphash as the default 64 bit keyed hash, but also making
128 and 256 bit highway hashes available.
There already are a few other functions that are defined but not
yet implemented - these will be "static" keyed hashes - which use
an installation specific key. These will be used to, e.g., get
rid of md5 hashing for the generation of file UIDs. (Johanna Amann, Corelight)
* Introduce calculate_digest convenience function
This function just calculates the chosen digest and returns the result
in either the passed buffer, or in a static buffer. Basically a superset
to the surprisingly popular internal_md5. (Johanna Amann, Corelight)
* Remove the siphash->hmac-md5 switch after 36 bytes.
Currently, siphash is used for strings up to 36 bytes. hmac-md5 is used
for longer strings.
This switch-over is a remnant of the previous hash-function that was
used, which apparently was slower with longer input strings.
This change serves no purpose anymore. I performed a few performance tests
on strings of varying sizes:
For a 40 byte string with 10 million iterations:
siphash: 0.31 seconds
hmac-md5: 3.8 seconds
For a 1080 byte string with 10 million iterations:
siphash: 4.2 seconds
hmac-md5: 17 seconds
For a 18360 byte string with 10 million iterations:
siphash: 69 seconds