Zippy

nimble install zippy

Zippy is an implementation of DEFLATE, ZLIB and GZIP data compression formats. Zippy can also create and open Tarballs (.tar, .tar.gz, .tgz, .taz) and ZIP archives.

The goal of this library is to be a pure Nim implementation that is small, performant and dependency-free.

Zippy can also be used at compile time. This is great for baking assets into executables in compressed form. Check out an example here.

To ensure Zippy is compatible with other implementations, tests/validate.nim can be run. This script verifies that data compressed by Zippy can be uncompressed by other implementations (and that other implementations can uncompress data compressed by Zippy).

This library works well using Nim's relatively new --gc:arc and --gc:orc as well as the default garbage collector. This library also works using both nim c and nim cpp, in addition to --cc:vcc on Windows.

I have also verified that Zippy builds with --experimental:strictFuncs on Nim 1.4.0.

Examples

Simple examples using Zippy can be found in the examples/ folder.

• HTTP client gzip
• HTTP server gzip
• Compress a dir into a tarball or zip archive
• Extract from a tarball or zip archive
• Compose a tarball or zip archive in code

Performance

Benchmarks can be run comparing different deflate implementations. My benchmarking shows this library performs very well, a bit faster than zlib in some cases and a bit slower in others. Check the performance yourself by running tests/benchmark.nim.

nim c -d:release -r .\tests\benchmark.nim

Compress

Each file is compressed 1000 times.

Default compression

https://github.com/guzba/zippy results:

File	Time	Size Reduction
alice29.txt	3.6858	63.32%
urls.10K	19.4039s	67.49%
rfctest3.gold	0.6802s	70.73%
randtest3.gold	0.1054s	0%
paper-100k.pdf	1.9085s	19.94%
geo.protodata	1.2939s	86.91%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

File	Time	Size Reduction
alice29.txt	6.8945s	64.23%
urls.10K	16.3272s	68.29%
rfctest3.gold	0.8147s	71.74%
randtest3.gold	0.1545s	0%
paper-100k.pdf	1.8938s	20.59%
geo.protodata	1.0743s	87.24%

Fastest compression

https://github.com/guzba/zippy results:

File	Time	Size Reduction
alice29.txt	1.5847s	55.32%
urls.10K	5.2101s	61.70%
rfctest3.gold	0.4295s	66.31%
randtest3.gold	0.0382s	0%
paper-100k.pdf	1.0918s	18.44%
geo.protodata	0.8353s	80.42%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

File	Time	Size Reduction
alice29.txt	1.7779s	57.17%
urls.10K	7.3260s	63.93%
rfctest3.gold	0.3270s	67.53%
randtest3.gold	0.1189s	0%
paper-100k.pdf	1.6632s	20.22%
geo.protodata	0.4888s	84.12%

Best compression

https://github.com/guzba/zippy results:

File	Time	Size Reduction
alice29.txt	4.5985s	63.75%
urls.10K	28.5602s	68.14%
rfctest3.gold	1.3401s	70.92%
randtest3.gold	0.1257s	0%
paper-100k.pdf	2.1010s	20.07%
geo.protodata	1.5293s	87.07%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

File	Time	Size Reduction
alice29.txt	9.9339s	64.38%
urls.10K	30.5398s	68.82%
rfctest3.gold	2.6180s	71.77%
randtest3.gold	0.1169s	0%
paper-100k.pdf	2.0639s	20.64%
geo.protodata	1.4266s	87.37%

Uncompress

Each file is uncompressed 1000 times:

https://github.com/guzba/zippy results:

File	Time
alice29.txt	0.4957s
urls.10K	2.2403s
rfctest3.gold	0.1240s
randtest3.gold	0.098s
paper-100k.pdf	0.4399s
geo.protodata	0.1905s

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

File	Time
alice29.txt	0.4806s
urls.10K	2.0093s
rfctest3.gold	0.1285s
randtest3.gold	0.0101s
paper-100k.pdf	0.3345s
geo.protodata	0.1866s

Testing

nimble test

To prevent Zippy from causing a crash or otherwise misbehaving on bad input data, a fuzzer has been run against it. You can do run the fuzzer any time by running nim c -r tests/fuzz.nim

API: zippy

import zippy

const NoCompression

NoCompression = 0

const BestSpeed

BestSpeed = 1

const BestCompression

BestCompression = 9

const DefaultCompression

DefaultCompression = -1

const HuffmanOnly

HuffmanOnly = -2

type CompressedDataFormat

Supported compressed data formats

CompressedDataFormat = enum
 dfDetect, dfZlib, dfGzip, dfDeflate

func compress

Compresses src and returns the compressed data.

func compress(src: seq[uint8]; level = DefaultCompression; dataFormat = dfGzip): seq[
 uint8] {.raises: [ZippyError].}

template compress

Helper for when preferring to work with strings.

template compress(src: string; level = DefaultCompression; dataFormat = dfGzip): string

func uncompress

Uncompresses src and returns the uncompressed data seq.

func uncompress(src: seq[uint8]; dataFormat = dfDetect): seq[uint8] {.raises: [ZippyError].}

template uncompress

Helper for when preferring to work with strings.

template uncompress(src: string; dataFormat = dfDetect): string

type ZippyError

Raised if an operation fails.

ZippyError = object of ValueError

API: zippy/tarballs

import zippy/tarballs

type EntryKind

EntryKind = enum
 ekNormalFile = 48, ekDirectory = 53

type TarballEntry

TarballEntry = object
 kind*: EntryKind
 contents*: string
 lastModified*: times.Time

type Tarball

Tarball = ref object
 contents*: OrderedTable[string, TarballEntry]

proc addDir

Recursively adds all of the files and directories inside dir to tarball.

proc addDir(tarball: Tarball; dir: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect].}

proc open

Opens the tarball file located at path and reads its contents into tarball.contents (clears any existing tarball.contents entries). Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc open(tarball: Tarball; path: string) {.raises: [IOError, ZippyError, ZippyError], tags: [ReadIOEffect].}

proc writeTarball

Writes tarball.contents to a tarball file at path. Uses the path's file extension to determine the tarball format. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc writeTarball(tarball: Tarball; path: string) {.raises: [ZippyError, IOError], tags: [WriteIOEffect].}

proc extractAll

Extracts the files stored in tarball to the destination directory. The path to the destination directory must exist. The destination directory itself must not exist (it is not overwitten).

proc extractAll(tarball: Tarball; dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadEnvEffect, ReadIOEffect, WriteDirEffect, WriteIOEffect].}

proc extractAll

Extracts the files in the tarball located at tarPath into the destination directory. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc extractAll(tarPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
 ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}

proc createTarball

Creates a tarball containing all of the files and directories inside source and writes the tarball file to dest. Uses the dest path's file extension to determine the tarball format. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc createTarball(source, dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect, WriteIOEffect].}

API: zippy/ziparchives

import zippy/ziparchives

type EntryKind

EntryKind = enum
 ekFile, ekDirectory

type ArchiveEntry

ArchiveEntry = object
 kind*: EntryKind
 contents*: string

type ZipArchive

ZipArchive = ref object
 contents*: OrderedTable[string, ArchiveEntry]

proc addDir

Recursively adds all of the files and directories inside dir to archive.

proc addDir(archive: ZipArchive; dir: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect].}

proc open

Opens the zip archive file located at path and reads its contents into archive.contents (clears any existing archive.contents entries).

proc open(archive: ZipArchive; path: string) {.raises: [IOError, ZippyError, ZippyError], tags: [ReadIOEffect].}

proc writeZipArchive

Writes archive.contents to a zip file at path.

proc writeZipArchive(archive: ZipArchive; path: string) {.raises: [ZippyError, ZippyError, IOError], tags: [WriteIOEffect].}

proc extractAll

Extracts the files stored in archive to the destination directory. The path to the destination directory must exist. The destination directory itself must not exist (it is not overwitten).

proc extractAll(archive: ZipArchive; dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadEnvEffect, ReadIOEffect, WriteDirEffect, WriteIOEffect].}

proc extractAll

Extracts the files in the archive located at zipPath into the destination directory.

proc extractAll(zipPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
 ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}

proc createZipArchive

Creates an archive containing all of the files and directories inside source and writes the zip file to dest.

proc createZipArchive(source, dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect, WriteIOEffect].}