DSQ

pytorch implementation of "Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks"

---

The Origin Paper : https://arxiv.org/abs/1908.05033

---

This repository follow the Algorithm 1 in the paper.

This repository uses the max value of int32 as the initial value.
It should not affect the value range (because the parameter of the deep model should not too large), and most of the edge device range is up to int32.

---

Training

Training with quantization. Scrip modified from https://github.com/pytorch/examples/tree/master/imagenet

Now support uniform/DSQ quantization
adding argments
-q : quantization type, default is None
-quantize_input : quantize input or not
--quan_bit : quantization bit num
--log_path : tensorboard log path to write, default folder is ./log.

Examples
Training DSQ with 8 bit (no quantiza input)

python train.py -a resnet18  -q DSQ --quan_bit 8

Training DSQ with 8 bit ( quantiza input)

python train.py -a resnet18  -q DSQ --quan_bit 8 -quantize_input

Experiments

model	QuanType	W/A bit	top1	top5
resnet18	UniformQuan	4/32	69.372	88.824
	DSQ	4/32
	UniformQuan	4/4
	DSQ	4/4

Update Note

20191218: Update uniform quantization results. It seems that the sgn function still need STE backward or the loss will becomes Nan.