forked from apache/tika
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGES.txt
2802 lines (1945 loc) · 99.7 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Release 2.0.1 - ???
MAJOR CHANGES in 2.0.1
* Improved packaging for tika-parsers-extended. Use the tika-parser-scientific-package and
tika-parser-sqlite3-package artifacts if you want fat jars with dependencies. (TIKA-3510)
* Tika app writes UTF-8 when an encoding is not specified; the legacy behavior
was UTF-8 on Mac OS, but System default on other OSs (TIKA-3515).
* Change the default rendering strategy for PDFs from NO_TEXT to ALL (TIKA-3520).
Other changes:
* Fix markup ordering errors in xhtml output for ODT files (TIKA-2242).
* Fix serialization of embedded docs in OpenSearch emitter
and fix embedded documents not being indexed in some use
cases in the Solr emitter (TIKA-3490).
* Add pipesClientId system property to PipesServer so that each
forked process can log to its own logger (TIKA-3480).
* Add DateNormalizingMetadataFilter let users ensure that all dates
emitted to Solr/OpenSearch are in UTC. Users can configure which
timezone they'd like to use in cases where the file format does
not store a timezone (TIKA-3496).
* Breaking change in the Solr and OpenSearch emitters. To achieve
the SKIP or CONCATENATE attachment strategy, modify the
parseMode in the pipesiterators or in the FetchEmitTuple (TIKA-3494).
Release 2.0.0 - 07/07/2021
* Cleanup of fetcher integration with tika-server.
* Update dependencies.
Release 2.0.0-BETA - 05/19/2021
* Refactor pipes module for resilience
* Add transcribe capability (TIKA-94).
Release 2.0.0-ALPHA - 01/13/2021
BREAKING CHANGES in 2.0.0
* General
* OCR is now triggered automatically for PDFs if tesseract
is on the user's path see (https://cwiki.apache.org/confluence/display/TIKA/TikaOCR#TikaOCR-disable-ocr)
for how to disable OCR.
* We upgraded from log4j to log4j2 in tika-app, tika-server and anywhere else
we used to use log4j.
* By default, when rendering a page for OCR, the PDFParser does not render glyphs/text.
* Removed deprecated Metadata keys/properties (TIKA-1974).
* Removed deprecated PDFPreflightParser (TIKA-3437).
* Removed dangerous calls to read an inputstream or convert to bytes
without specifying a charset
* Parsers can be configured via tika-config.xml on instantiation.
We have moved away from configuration via .properties files because
of confusion among users. This affects the PDFParser, TesseractOCRParser
and the StringsParser.
* Changed namespaces of translator implementations (o.a.t.language.translate.impl) to avoid
split-package with tika-core
* tika-parsers
* The parser modules have been broken into three main modules:
tika-parsers-standard, tika-parsers-extended and tika-parsers-ml.
Users may now need to add tika-parsers-extended's
tika-parser-scientific-module or tika-parser-sqlite3-module to tika-app and
tika-server to include parsers that used to be included by default
(for example: envi, gdal, grib, isatab, netcdf, sqlite3).
* PDFParser -- a) see above on OCR. b) This parser no longer warns if the jpeg2000
dependency is not included. Tika now relies on PDFBox to log an error if a jpeg2000
image should be processed but can't because the required external dependency is
not available. See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
for the non-ASF-2.0-compatible jpeg2000 library.
* CompressorParser -- users must add the com.github.luben:zstd-jni dependency to
the classpath to process zstd files. This is an optional library that is no longer bundled
in tika-parsers-standard-package because it contains native libs.
* ChmParser was moved to org.apache.tika.parser.microsoft.chm
* RTFParser was moved to org.apache.tika.parser.microsoft.rtf
* We are now using non-shaded versions of xmpcore with namespaces com.adobe.internal.*
vs com.adobe.*.
* tika-app
* See above on default inclusion of only tika-parsers-standard.
* tika-server
* tika-server now by default forks a process to isolate the parsing
in the forked process (this was called the -spawnChild option
in tika-1.x). Clients must now expect that tika-server
will restart on OOM, timeouts, crashes or after parsing a
large number of files. When this happens tika-server will restand and not
receive connections for brief periods. The less robust, legacy behavior
of not forking a process is available with "-noFork"=
* Most of tika-server's legacy configuration via the commandline has been moved
into configuration via a tika-config.xml file.
* tika-server's "enableFileUrl" has been removed in favor of a FileSystemFetcher.
* tika-server's /metadata endpoint requires tika-server-standard to write XMP/rdf output.
This output is not available in tika-server-core.
* In tika-server, for those parsers that can be configured per parse via a config object
passed in through the ParseContext, the config object will only update those fields
that the user has modified. The config object will no longer
fully reset all settings to the default settings per parse.
This has a more intuitive "update the base/configured settings" with
what has been changed in the config object.
* tika-eval
* tika-eval's default profile and comparison reports no longer include tag reports.
Users can get the report configs that include tags (*-tags.xml):
https://github.com/apache/tika/tree/main/tika-eval/tika-eval-app/src/main/resources
Release 1.27 - 06/30/2021
* Migrate MP4 parsing to Drew Noakes' metadata-extractor (TIKA-3459).
To revert to legacy parser turn off NoakesMP4Parser and turn on MP4Parser
via tika-config.xml.
* Prevent rare infinite loop in tika-server's -spawnChild mode
when restart fails because of failure to bind to the port (TIKA-3441).
* Improve likelihood that tesseract will not be orphaned on
jvm restart in tika-server (TIKA-3441).
* Deprecate experimental PDFPreflightParser (TIKA-3437).
* Apply encoding detection to zip entry names via Ryan421 (TIKA-3374).
* Add json output for /tika endpoint in tika-server (TIKA-3352).
* Tika's PDFParser should use the underlying file if one is passed in
via a TikaInputStream (TIKA-3350)
Release 1.26 - 03/24/2021
* Fix thread safety bug in OpenOffice parser (TIKA-3334).
* The "writeLimit" header now pertains to the combined characters
written per container document (and embedded documents) in the /rmeta
endpoint in tika-server (TIKA-3325); it no longer functions only
per container or embedded document.
* Extract more embedded files in PDFs by recursively processing the
embedded file tree (TIKA-3332).
* Allow for case insensitive headers for configuration of the PDFParser
and the TesseractOCRParser in tika-server via Subhajit Das (TIKA-3320).
* Improve detection and parsing of XPS files (TIKA-3316).
* General dependency upgrades (TIKA-3244).
* Great optimization in ForkParser (TIKA-3237).
* Fix parsing of emails attached to other emails in PST files (TIKA-3004).
* MP3 parser should output the xmpDM:duration metadata as seconds not
milliseconds, consistent with the other Audio and Video parsers (TIKA-3318).
* MP4 parser check if any of the Compatible Brands match when identifying
the subtype (TIKA-3310).
Release 1.25 - 11/25/2020
* Fix inconsistent license in xmpcore (TIKA-3204).
* General upgrades including some dependencies with
recently found security vulnerabilities (TIKA-3119).
* Add detection and a parser for flat ODF files (TIKA-3159).
* Add extraction of macros from ODF files (TIKA-3161).
* Add mime detection for hprof and hprof text files (TIKA-3144).
* Add TextSignature and TextProfileSignature to tika-eval (TIKA-3145 and TIKA-3146)
* Create a metadata filter to trigger tika-eval stats post parsing (TIKA-3140)
* Add a configurable metadata-filter for the RecursiveParserWrapper (TIKA-3137)
* Parameterize writeLimit and maxEmbeddedResources for RecursiveParserWrapper
in tika-server (TIKA-3133)
* Add status endpoint to tika-server (TIKA-3129).
* Remove whitelist/blacklist terminology (TIKA-3120)
* Add detection for parquet files (TIKA-3115).
* Add detection and parsing for bplist (TIKA-3104).
* Enable metadata value filtering for RecursiveParserWrapper (TIKA-3137)
* Add a basic parser for plist files based on com.googlecode.plist:dd-plist (TIKA-3104).
* Read hyperlinked images from ODT files (TIKA-3156).
* Updated GrobidRESTParser to use new API location (TIKA-3191).
* Add FileProfiler to tika-eval (TIKA-3216).
* Add status endpoint to tika-server (TIKA-3129).
* Improved handling of zip files with STORED entries with
data descriptor (TIKA-3196).
* Add parsers for XLZ, IDML and MIF (TIKA-2976, TIKA-3188 and TIKA-3189).
* Add the beginnings of a format-aware fuzzing module (TIKA-3083).
* Add wrapper for Linux 'file' command for mime detection (TIKA-3215).
* Added ability to skip parsing of embedded files in Tika Server (TIKA-3227).
Release 1.24.1 - 4/17/2020
* Allow gzip compression of input and output streams for tika-server (TIKA-3073).
Release 1.24 - 3/11/2019
* Add scripts to run tika-server as a service via Eric Pugh,
and add these scripts and jar as a new artifact in the release (TIKA-3010).
* Upgrade Drew Noakes' metadata-extractor (TIKA-2952).
* Enable optional extraction of structural tags in PDFs (alpha-grade) (TIKA-3026).
* Tika app's --extract mode now outputs to STDOUT (TIKA-3035).
* Add an optional Preflight parser for PDFs (TIKA-3055).
* Improve detection of some zip-based formats (TIKA-3057).
* Upgrade metadata-extractor to 2.13.0 (TIKA-2952).
* Upgrade to POI 4.1.2 (TIKA-3047).
* Extract XMP from PSD files (TIKA-3050).
* Added XMLProfiler as an optional parser to profile XFA and XMP
in PDFs (TIKA-3045).
* Extract inline images that rely on the DCT filter from PDFs (TIKA-3041).
* Upgrade to PDFBox 2.0.19 (TIKA-3033).
* Fix bug in ASM parser configuration (TIKA-2992).
* Upgrade to java-libpst 0.9.3 (TIKA-2546).
* Fixed XLIFF12Parser failures with ToXMLHandler (TIKA-3014).
Release 1.23 - 12/02/2019
* NOTE: The PDFParser now relies on OCRDPI to render page images when
users configure OCR on rendered page images. This will have the effect
of increasing rendered image size (TIKA-2624).
* NOTE: tika-server no longer returns 415 for file types for which there
is no parser.
* Fix bug in AUTO OCR strategy in the PDFParser (TIKA-3002).
* Fix incorrect height and width metadata extraction from JPEG images (TIKA-2630).
* Upgrade to POI 4.1.1 (TIKA-2851).
* Upgrade to PDFBox 2.0.17 (TIKA-2951).
* Ensure that the PDFParser respects custom configuration of Tesseract
from tika-config.xml via Eric Pugh (TIKA-2970).
* Add parser for XLIFF v1.2 files (TIKA-2975).
* Add mime type detection support for WebAssembly (TIKA-2894),
HEIF / HEIC images (TIKA-2942), Digilite FDF (TIKA-2988);
and xml-root detection for XFDF (TIKA-2990) and XDP (TIKA-2989).
* Add an XLZ Parser (TIKA-2976).
* Fix deadlock with ForkParser when InputStream throws IOException (TIKA-2892).
Release 1.22 - 07/29/2019
* NOTE: tika-server no longer hard-codes the HtmlParser to handle
XML files (TIKA-2910). Users must now configure that behavior
via a tika-config.xml file.
* NOTE: Known regression: PDFBOX-4587 -- PDF passwords with codepoints
between 0xF000 and 0XF0000 will cause an exception.
* Add parser for HWP v5 files via SooMyung Lee (soomyung) and
JinSup Kim (ddoleye) (TIKA-2909).
* Fix order of closing streams to avoid "Failed to close temporary resource"
exception in TesseractOCRParser (TIKA-2908).
* Improve AutoDetectReader performance by caching encoding
detector (TIKA-1568).
* Prevent RTFParser from outputting illegal tag combinations (TIKA-2889).
* Fix RereadableInputStream to release all resources (TIKA-2903).
* Implement custom language identifier in the tika-eval module based on
OpenNLP's language detector; add 18 languages and add common words
lists for all 121 languages (TIKA-2790).
* Fix NPE in MimeTypesReader.releaseParser() via Eamonn Saunders (TIKA-2896).
* Fix RTFParser to extract more content (TIKA-2883).
* Add clientSubmitTime to the metadata extracted from PST files (TIKA-2898).
* Improve StreamingZipContainerDetector for xltx, xltm and
several other file formats (TIKA-2886).
Release 1.21 - 05/14/2019
* Add optional AUTO mode to OCR'ing of PDFs. If tesseract is installed
and on the path, and this option is selected programmatically
or via TikaConfig(), the PDFParser will use heuristics to decide
whether or not to run OCR per page on PDFs. (TIKA-2749)
* The ZipContainerDetector's default behavior was changed to run
streaming detection up to its markLimit. Users can get the
legacy behavior (spool-to-file/rely-on-underlying-file-in-TikaInputStream)
by setting markLimit=-1. The POIFSContainerDetector requires an underlying file;
it will try to spool the file to disk; if the file's length is > markLimit,
it will not attempt detection; set markLimit to -1 for legacy behavior (TIKA-2849).
* Upgrade PDFBox to 2.0.14 (TIKA-2834).
* Add CSV detection and replace TXTParser with TextAndCSVParser;
users can turn off CSV detection by excluding the TextAndCSVParser
and adding back the TXTParser via tika-config (TIKA-2833).
* Add a CSVParser. CSV detection is currently based solely on filename
and/or information conveyed via Metadata (TIKA-2826).
* General upgrades: asm, bouncycastle, commons-codec, commons-lang3, cxf,
guava, h2, httpcomponents, jackcess, junrar, Lucene, mime4j, opennlp, parso,
sqlite-jdbc (provided), zstd-jni (provided) (TIKA-2824)
* Bundle xerces2 with tika-parsers (TIKA-2802).
* Upgrade jaxb to 2.3.2 (TIKA-2819).
* Upgrade jackson to 2.9.8 (TIKA-2717).
* Update tika-eval's common tokens lists (TIKA-2822).
* Handle bad tags in tika-eval more robustly (TIKA-2810).
* Add reports for tags in tika-eval (TIKA-2809).
* Extract text from SDT element within textboxes in .docx files (TIKA-2807).
* Try to handle truncated OOXML files more robustly (TIKA-2765).
Release 1.20 - 12/17/2018
* Upgrade to POI 4.0.1 (TIKA-2751).
* Integrate/parameterize new angles handling in
PDFBox (TIKA-2779).
* Upgrade to PDFBox 2.0.13 (TIKA-2788).
* Prevent content within <style/> and <script/> elements
to be written in the ToTextContentHandler (TIKA-2550).
* Switch child to parent communication to a shared memory-mapped
file in tika-server's -spawnChild mode.
* Fix bug in tika-server when run in legacy mode (not -spawnChild)
that caused it to return 503 on documents submitted after
it hit an OutOfMemoryError (TIKA-2776).
* Upgrade jaxb-runtime and javax.activation (TIKA-2778).
* tika-app in batch mode now requires an interrupt or
kill signal to the parent process to stop the parent
and the child processes (TIKA-2780).
* Bulk upgrade of dependencies (TIKA-2775).
* Improve language id efficiency in tika-eval (TIKA-2777).
* Upgrade sqlite "provided" dependency to 3.25.2 (TIKA-2773).
* Remove duplication of notes in PPT slides (TIKA-2735)
* Use -javaHome or $JAVA_HOME (if they exist) when
spawning child in tika-server's -spawnChild mode.
* Fixed closing of styles around Hyperlinks in Word Parser
Contributed by Ronan O'Sullivan (TIKA-2599).
Release 1.19.1 - 10/4/2018
* Update PDFBox to 2.0.12, jempbox to 1.8.16
and jbig2 to 3.0.2 (TIKA-2745).
* Fix regression in parser for MP3 files (TIKA-2730).
* Updated Python Dependency Check for TesseractOCR (TIKA-2740).
* Improve SAXParser robustness (TIKA-2727).
* Remove dependency on slf4j-log4j12 by upgrading jmatio (TIKA-2742).
* Replace com.sun.xml.bind:jaxb-impl and jaxb-core with
org.glassfish.jaxb:jaxb-runtime and jaxb-core (TIKA-2743)
Release 1.19 - 9/14/2018
* Require Java 8 (TIKA-2679).
* Enable building with Java 11 (TIKA-2668)
* Add an option to make tika-server robust against infinite loops,
OOMs, and memory leaks (TIKA-2725).
* Allow configuration of the Tesseract parser via the standard
tika-config.xml options (TIKA-2705).
* Improve handling of empty cells across table-based
formats (TIKA-2479).
* Add a Standards compliant HTML encoding detector
via Gerard Bouchar (TIKA-2673).
* Improved XML parsing -- limited default entity expansions to 20.
To raise this limit, add -Djdk.xml.entityExpansionLimit=XXX to
your commandline.
* Mime magic improvements for Olympus RAW (TIKA-2658), interpreted
server-side languages via HTTP (TIKA-2648), MHTML (TIKA-2723)
* Add absolute timeout to ForkParser rather than testing
for active (TIKA-2656).
* Make the RecursiveParserWrapper work with the ForkParser (TIKA-2655).
* Allow the ForkParser to specify a directory containing tika-app.jar
for use by the ForkServer. This allows users to keep most of the
parser dependencies out of their code; and it allows for an easy
addition of optional jars for Parser dependencies,
such as the xerial sqlite jar (TIKA-2653).
* Use a pool for SAXParsers and DOMBuilders rather than creating
a new parser/builder for every parse.
For better performance, set XMLReaderUtils.setPoolSize() to the
number of threads you're using with Tika (TIKA-2645).
* Add the RecursiveParserWrapperHandler to improve the RecursiveParserWrapper
API slightly (TIKA-2644).
* Upgraded to Commons-Compress 1.18 (TIKA-2707).
* Upgraded to Apache POI 4.0.0 (TIKA-2552).
* Upgraded to Apache PDFBox 2.0.11 (TIKA-2681).
* Upgraded to deeplearning4j 1.0.0-beta2 (TIKA-2672).
* Upgraded jmatio to 1.4 (TIKA-2667)
* Upgraded Apache Lucene to 7.4.0 in tika-eval and tika-examples (TIKA-2695).
* Upgraded junrar to 1.0.1 (TIKA-2664).
* Numerous other upgrades (TIKA-2692).
* Excluded Spring as a transitive dependency (TIKA-2721).
Release 1.18 - 4/20/2018
* Upgrade jackson to 2.9.5 (TIKA-2634).
* Add support for brotli (TIKA-2621).
* Upgrade PDFBox to 2.0.9 and include new jbig2-imageio
from org.apache.pdfbox (TIKA-2579 and TIKA-2607).
* Support for TIFF images in PDF files (TIKA-2338)
* Detection of full encrypted 7z files (TIKA-2568)
* Various new mimes and typo fixes in tika-mimetypes.xml
via Andreas Meier (TIKA-2527).
* Revert to listenForAllRecords=false in ExcelExtractor
via Grigoriy Alekseev (TIKA-2590)
* Add workaround to identify TIFFs that might confuse
commons-compress's tar detection via Daniel Schmidt
(TIKA-2591)
* Ignore non-IANA supported charsets in HTML meta-headers
during charset detection in HTMLEncodingDetector
via Andreas Meier (TIKA-2592)
* Add detection and parsing of zstd (if user provides
com.github.luben:zstd-jni) via Andreas Meier (TIKA-2576)
* Allow for RFC822 detection for files starting with "dkim-"
and/or "x-" via Andreas Meier (TIKA-2578 and TIKA-2587)
* Extract xlsx files embedded in OLE objects within PPT and PPTX
via Brian McColgan (TIKA-2588).
* Extract files embedded in HTML and javascript inside HTML
that are stored in the Data URI scheme (TIKA-2563).
* Extract text from grouped text boxes in PPT (TIKA-2569).
* Extract language metadata item from PDF files via Matt Sheppard (TIKA-2559)
* RFC822 with multipart/mixed, first text element should be treated
as the main body of the email, not an attachment (TIKA-2547).
* Swap out com.tdunning:json for com.github.openjson:openjson to avoid
jar conflicts (TIKA-2556).
* No longer hardcode HtmlParser for XML files in tika-server (TIKA-2551).
* Require Java 8 (TIKA-2553).
* Add a parser for XPS (TIKA-2524).
* Mime magic for Dolby Digital AC3 and EAC3 files
* Fixed bug where TesseractOCRParser ignores configured ImageMagickPath,
and set rotation script to ignore Python warnings (TIKA-2509)
* Upgrade geo-apis to 3.0.1 (TIKA-2535)
* Mime definition and magic improvements for text-based programming
and config formats (TIKA-2554, TIKA-2567, TIKA-1141)
* Added local Docker image build using dockerfile-maven-plugin to allow
images to be built from source (TIKA-1518).
* Support for SAS7BDAT data files (TIKA-2462)
* Handle .epub files using .htm rather than .html extensions for the
embedded contents (TIKA-1288)
* Mime magic for ACES Images (TIKA-2628) and DPX Images (TIKA-2629)
* For sparse XLSX and XLSB files, always output missing cells to
the left of filled ones (matching XLS), and optionally output
missing rows on all 3 formats if requested via the
OfficeParserContext (TIKA-2479)
Release 1.17 - 12/8/2017
***NOTE: THIS IS THE LAST VERSION OF TIKA THAT WILL RUN
ON Java 7. The next versions will require Java 8***
* Fix thread-safety in ChmExtractor (TIKA-2519).
* Upgrade cxf to 3.0.16 (TIKA-2516).
* Allow users to configure maxMainMemoryBytes for PDFs via shrike (PR-213).
* Extract underline and strikethrough in docx (TIKA-2347 and TIKA-2512).
* Cache TikaConfig in EmbeddedDocumentUtil for better performance
in documents with large number of attachments (TIKA-2511).
* Extract media files from ooxml (TIKA-2510).
* Standardize the way the Image and Video captioning
dockers and extraction work (TIKA-2400, GitHub-208)
* Upgrade to xmpcore 5.1.3 (TIKA-2034).
* Upgrade to metadata-extractor 2.10.1 (TIKA-2486).
* Upgrade to OpenNLP 1.8.3 (TIKA-2502).
* Upgrade to Jackson 2.9.2 (TIKA-2501).
* Catch potential NPE in getting InputStream for attachments
in PST file (TIKA-2488).
* Upgrade to PDFBox 2.0.8 (TIKA-2489).
* Allow configuration of markLimit in EncodingDetectors
via tika-config.xml (TIKA-2485).
* RFC822Parser now selects the best alternative for
multipart/alternative body components. This aligns with the
behavior of the OutlookParser (TIKA-2478). Users can select
legacy behavior via the "extractAllAlternatives" parameter
in the RFC822 parser definition in tika-config.xml.
* Narrow mime detection for ms-owner files and add detection
for .nls files (TIKA-2469).
* Fix bug in CharsetDetector that led to different detected charsets
depending on whether user setText with a byte[] or an InputStream
via Sean Story (TIKA-2475).
* Remove JAXB for easier use with Java 9 via Robert Munteanu (TIKA-2466).
* Upgrade to POI 3.17 (TIKA-2429).
* Enabling extraction of standard references from text (TIKA-2449).
* Load external custom mimetypes XML from system property
tika.custom-mimetypes (TIKA-2460).
* Extract number of tiffs in a multi-page tiff (TIKA-2451).
* Fix detection of emails extracted from mbox (TIKA-2456).
* Add OverrideDetector and allow PSTParser to specify body content type
as text or html -- to avoid incorrect auto-detection of
rfc/mbox, etc. (TIKA-2454)
* AutoDetectParser throws ZeroByteFileException for zero-byte files after
detection on the file extension (TIKA-2450).
* Extract phonetic runs in docx with experimental SAX parser (TIKA-2448).
* Extract phonetic runs from xls and allow users to turn off extraction
of phonetic runs in both xls and xlsx (TIKA-2440).
* OOXML locale should be set by POI's LocaleUtil not Locale.getDefault().
Fix unit tests to be robust against different locales in OOXML
and ExcelParser (TIKA-2438).
* Upgrade to PDFBox 2.0.7 (TIKA-2431).
* Tika now has support for automatic image captioning, that
combines Computer Vision and Natural Language Processing to
automatically generate a readable caption for an image
(TIKA-2262, TIKA-2355, TIKA-2402, Gh-198, Gh-196, Gh-189).
* Add TestCorruptedFiles to allow devs to test parsers against
corrupted input files (TIKA-2430).
* Correct Mimetype definition for Windows batch files (CMD and BAT)
which are the same (TIKA-2445)
* PSDParser memory use improvements (TIKA-2447)
* Add underline extraction from Word documents (doc/docx) via Stuart Hendren
as well as strikethrough extraction in docx (TIKA-2347, GitHub-173)
* Corrected Tesseract OCR rotation.py script and made it a configurable
option via Peter Weiss (TIKA-2385)
Release 1.16 - 7/7/2017
* Exclude jj2000 from edu.ucar grip to avoid potential
license conflicts with ASL 2.0
* Add Age recognition using Ensemble model for Linear regression
and Apache OpenNLP Maximum Entropy. Tika can now detect age from
text (TIKA-1988).
* Add Tika Deep Learning support for the VGG16 model for
Very Deep Convolutional Networks for Large-Scale Image Recognition.
Now Tika supports both Inception v3/v4 and VGG16 based image
recognition (TIKA-2298).
* Extract macros from PPT (TIKA-2089).
* Extract absolute path for last saved location when available
in .xlsx and .xlsb (TIKA-2335).
* Rename SentimentParser to SentimentAnalysisParser to
prevent conflict with dependency (TIKA-2368).
* tika-app now extracts inline images in PDFs by
default, and it includes a warning to users that this is not the
default behavior elsewhere in Tika (TIKA-2374).
* Allow configurability of warnings for problems during
parser initialization (TIKA-2389).
* Upgrade to Jackcess 2.1.8 (TIKA-2380).
* Upgrade to POI 3.17-beta1 (TIKA-2336).
* Remove non-ASL-2.0-compatible org.json (TIKA-1804).
* Allow extraction of <script> elements in HTML as embedded "MACRO".
Users must turn this on via TikaConfig (TIKA-2391).
* Allow users to turn off extraction of headers and footers
from .doc, .docx, .xls, .xlsx, .xlsb (TIKA-2362)
* Extract text from charts in .docx, .pptx, .xlsx and .xlsb
(TIKA-2254).
* Extract text from diagrams in .docx, .pptx, .xlsx and .xlsb
(TIKA-1945).
* Fix bug in tika-server that led to an attempt to close the
input stream twice (TIKA-2384).
* Enable base32 encoding of digests and enable BouncyCastle implementations
of digest algorithms (TIKA-2386).
* Add snap builds to codebase (TIKA-2401)
* Canonical Mimetype of WAVE audio changed to match RFC 2361 defined
version, audio/vnd.wave, older audio/x-wav remains as an alias
* Upgrade "provided" xerial to 3.19.3 (TIKA-2412).
* Upgrade Gson to 2.8.1 (TIKA-2414).
* Upgrade mime4j to 0.8.1 (TIKA-2413).
* Mime magic improvements for GraphViz (TIKA-2422), HTML files which
claim to be XML but aren't quite valid XML (TIKA-2419) and QuickTime
/ MP4 (TIKA-2418)
Release 1.15 - 05/23/2017
* Tika now has a module for Deep Learning powered by the
DL4J toolkit. The initial included model is for InceptionV3
and so using this module, natively in Java, Tika can use
Deep learning for metadata/text extraction from Images using
the power of the Inception model (Github-165).
* A new parser for sentiment analysis using a categorical
(multi-class, anry, sad, neutral, like, love) and binary
(positive/negative) was added leveraging the USC data
science work (TIKA-2016).
* Tika now has the ability to automatically detect objects in videos,
using OpenCV and Tensorflow (TIKA-2322).
* Change default behavior to parse embedded documents even if the user
forgets to specify a Parser.class in the ParseContext (TIKA-2096).
Users who wish to parse only the container document should set
an EmptyParser as the Parser.class in the ParseContext.
* Change default behavior of Office Parsers to _not_ extract
Macros. User needs to setExtractMacros to "true" (TIKA-2302).
* Added tika-eval module (TIKA-1332).
* Unified logging across Tika: SLF4J as logging API, Apache Log4j as
implementation with JCL and JUL bridges in standalone tools like
tika-app, tika-batch and tika-server (TIKA-2245).
* Add parser for XLSB files (TIKA-1195).
* Add parsers for EMF/WMF files (TIKA-2246/TIKA-2247).
* Add parsers for WordPerfect and QuattroPro (.qpw) files.
Contributed by Pascal Essiembre (TIKA-1946 and TIKA-2228).
* Add experimental SAX parser for .pptx files. To select this parser,
set useSAXPptxExtractor(true) on OfficeParserConfig (TIKA-2210).
* Add experimental SAX parser for .docx files. To select this parser,
set useSAXDocxExtractor(true) on OfficeParserConfig (TIKA-1321, TIKA-2191).
* Add mime detection and parser for Word 2006ML format (TIKA-2179).
* Bug fix for WordPerfect via Pascal Essiembre (TIKA-2352).
* Added "text-main" equivalent option to tika-server via
/tika/main (TIKA-2343).
* Enabled configuration of the EncodingDetector used by
parsers that extend AbstractEncodingDetectorParser (TIKA-2273).
* Prevent easily preventable OOMs for both detection and parsing
of some compression formats (TIKA-2330).
* Extract images and thumbnails from ODT via Sam Bayer (TIKA-2295).
* Fix potential NPE in FeedParser via Julien Nioche (TIKA-2269).
* Official mime types for BMP, EMF and WMF have been registered with
IANA, so switch to these (image/bmp image/emf image/wmf) (TIKA-2250)
* Be more parsimonious with BufferedInputStreams via Josh Hight
(TIKA-2244).
* Enable handling of hyphenated language codes in TesseractOCRParser
via Graham Russell (TIKA-2231).
* Improve style tags in ODT (TIKA-2242).
* Add container detection for embedded MSEquation files (TIKA-2238).
* Add parsing of JBIG2 and extraction of JBIG2 from PDFs when
required dependencies are added to class path by user.
Contributed by Pascal Essiembre (TIKA-2232).
* Mime magic for the OneNote family (.one / .onetoc / .onepkg), no parser
(TIKA-2224).
* Add configurability of "preserve-interword-spacing" to
TesseractOCRParser (TIKA-2190).
* Upgrade to PDFBox 2.0.6 and JempBox 1.8.13 (TIKA-2209/TIKA-2236/TIKA-2361).
* Refactor MockParser to consolidate service loading
and mime types into tika-core/src/test (TIKA-2195).
* Enabled extraction of embedded objects from headers, footers,
footnotes, endnotes and comments in legacy .docx parser (TIKA-2192).
* Allow extraction of PDActions (including Javascript) from
PDFs (TIKA-2090). This is turned off by default. Users
must setExtractActions(true) on the PDFParserConfig.
* Change default behavior in experimental .docx parser to ignore
deleted text to align with .doc (TIKA-2187).
* Upgrade to POI 3.16 (TIKA-2116, TIKA-2181, TIKA-2329).
* Allow configuration of timeout for ForkParser (TIKA-2170).
* Add extraction of .jpx inline images from PDFs when required
dependencies are added by user to class path (TIKA-2175).
* Add .jpx, .jp2, .ppm to formats handled by Tesseract (TIKA-2174).
* Upgrade SQLite "provided" dependency to 3.16.1 (TIKA-2334).
* Update Apache CXF version to 3.0.12 (TIKA-2292).
* Add Lingo24 Language Detector (TIKA-2297).
* Further mime magic for WebVTT (TIKA-1772)
* Extend support for increased PSM options up to 13 for modern
versions of Tesseract (TIKA-2357).
* Prevent potential resource leak by closing TrueTypeFont
via Cameron Rollheiser (TIKA-2370).
Release 1.14 - 10/19/2016
* Extract all headers from MSG/RFC822 (TIKA-2122).
* Upgrade metadata-extractor to 2.9.1 (TIKA-2113).
* Extract PDF DocInfo metadata into separate keys to prevent
overwriting by XMP metadata (TIKA-2057).
* Re-enable fileUrl for tika-server (TIKA-2081). If you choose,
to use this feature, beware of the security vulnerabilities!
See: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3271
* Add Tesseract's hOCR output format as an option, via Eric Pugh
(TIKA-2093)
* Extract macros from MSOffice files (TIKA-2069).
* Maintain passed-in mime in TXTParser (TIKA-2047).
* Upgrade to POI.3-15 (TIKA-2013).
* Upgrade to PDFBox 2.0.3 (TIKA-2051).
* Fix hyperlinks with formatting in DOC and DOCX (TIKA-1255
and TIKA-2078)
* Tika now is integrated with the Tensorflow library from Google
and it can use its Inception v3 image classification model to
identify objects in images (TIKA-1993).
* Parser configuration is now type-safe and parameters for parsers
can have assigned types (TIKA-1508, TIKA-1986).
* Prevent OOM/permanent hang on some corrupt CHM files (TIKA-2040).
* Upgrade ICU4J charset detection components to fix multithreading
bug (TIKA-2041).
* Upgrade to Jackcess 2.1.4 (TIKA-2039).
* Maintain more significant digits in cells of "General" format
in XLS and XLSX (TIKA-2025).
* Avoid mark/reset issues when extracting or detecting embedded resources
in RFC822 emails (TIKA-2037).
* Improving accuracy of Tesseract for better extraction of numeric
and alphanumeric text from images (TIKA-2021, TIKA-2031).
* Improve extraction of embedded documents from PPT, PPTX and XLSX
(TIKA-2026).
* Add parser for applefile (AppleSingle) (TIKA-2022).
* Add mime types, mime magic and/or globs for:
* Endnote Import File (TIKA-2011)
* DJVU files (TIKA-2009)
* MS Owner File (TIKA-2008)
* Windows Media Metafile (TIKA-2004)
* iCal and vCalendar (TIKA-2006)
* MBOX (TIKA-2042)
* Stata DTA (TIKA-2064)
* Add configurable maximum threshold for number of events extracted
from the XMP Media Management Schema in JempboxExtractor (TIKA-1999).
* Integrate TesseractOCR with full page image rendering for PDFs (TIKA-1994).
* Add mime detection via Nick C and parser for DBF files (TIKA-1513).
* Add mime detection and parsers for MSOffice 2003 XML Word
and Excel formats (TIKA-1958).
* Extract hyperlinks from PPT, PPTX, XSLX (TIKA-1454).
* Upgrade to Commons Compress 1.12 (supports progress on TIKA-1358)
Release 1.13 - 05/08/2016
* Upgrade to PDFBox 2.0.1 (TIKA-1285/TIKA-1959).
MAJOR CHANGES in PDFParser:
* The classic sequential parser is no longer available.
* Tiff files are no longer extracted by default. See
https://pdfbox.apache.org/2.0/dependencies.html#optional-components
for optional components to process Tiff files.
* Some truncated/corrupted files that had some content extracted
with 1.8.x may have no content extracted in 2.0.x (see TIKA-1912).
* The MIT-NLP Information Extraction (MITIE) Named Entity
Recognition (NER) system is now supported in Tika
(TIKA-1913, GitHub-108).
* Tika now supports the use of the Yandex translation
service (TIKA-1943, GitHub-106).
* Tika now uses NER to extract scientific measurements
from text using either GROBID Quantities which uses
conditional random fields and NLTK which uses regular
expressesions (TIKA-1917, GitHub-104).
* Fixed JournalParser to handle null responses from
GROBID and to log a message (TIKA-1925).
* Refactored Language Detector into tika-landetect module,
added default N-Gram implementation, Optimaize Lang
Detector and MIT Text.jl implementation
(TIKA-1872, TIKA-1696, TIKA-1723).
* Extract metadata from MP4 videos whether or not the
PooledTimeSeries parser is available via Aditya Dhulipala
(TIKA-1844).
* Fix NPE when trying to get embedded image identifier in
WordParser (TIKA-1956).
* Improvements to MIME database for detection of Scientific
and other formats present in the TREC-DD-Polar dataset
(TIKA-1881, GitHub-85, TIKA-1883, TIKA-1884, TIKA-1886,
TIKA-1882).
* LinkContentHandler now extracts links from script tags
via Joseph Naegele (TIKA-1937).
* Handle per page IOExceptions more robustly in PDFParser (TIKA-1948).
* Upgrade commons-compress to 1.11 (TIKA-1949).
* Add detection for embedded MSChart.Graph files (TIKA-1033).
* Fix NPE in Sqlite parser from Nick C (TIKA-1927).
* Fix NPE in Open Document parser from Nick C (TIKA-1916).
* Upgrade mp4parser's isoparser to 1.1.7 (TIKA-1924 and TIKA-1931).
* Upgrade BouncyCastle to 1.54 (TIKA-1923).
* Upgrade Jackcess to 2.1.3 (TIKA-1922).
* Upgrade Drew Noakes' metadata-extractor to 2.8.1 (TIKA-1921).
* Upgrade Gson in tika-serialization to 2.6.2 (TIka-1920).
* Upgrade commons-cli in tika-batch to 1.3.1 (TIKA-1919).
* Add XMPMM support to PDFParser and JpegParser via Jempbox (TIKA-1894).
* Move serialization of TikaConfig to tika-core and enable dumping
of the config file via tika-app (TIKA-1657).
* Tika now incorporates the Natural Language Toolkit (NLTK) from the
Python community as an option for Named Entity Recognition (TIKA-1876).