Skip to content

Files

This branch is 2 commits behind dmlc/mshadow:master.

doc

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
Jul 4, 2015
Aug 24, 2017
Feb 14, 2015

MShadow Documentation

This is the documentation for mshadow: A Lightweight CPU/GPU Matrix/Tensor Template Library in C++/CUDA.

Links to Topics

Expression API

Expression is the key concept in mshadow, a common operation of mshadow is tensor = some code to construct expression

There are three major types of expression:

  • Mapper expression: only contain element-wise operations of Mapper expressions
    • Mapper expression can used as composition component of other operations.
    • Tensor, scalar are Mapper expressions
    • Example: weight = - eta * (grad + lambda * weight) is a Mapper expression.
    • Mapper expressions are translated using expression template code implemented by mshadow.
    • Assign safety: Element-wise mapping are assign safe, which means, we can write A = A * 2 + B, making lvalue appear in expression, the results are still correct.
  • Chainer expression: may contain element-wise operation such as reduction and broadcast
    • Example: dst = mirror(src) is a chainer expression
    • Assign safety: Most of the chainer extensions are not assignment safe, which means user should avoid putting target in source epression.
  • Complex expression: complex operations, need special translation rule to translate to specific implementations.
    • Complex expression can not be used as composition component of other operations.
    • Example: dot(lhs.T(), rhs), is complex expression, we can not write dst = 1.0 + dot(lhs.T(), rhs)
    • But limited syntax is supported depending on specification, for example, we do support dst += 2.0f * dot(lhs.T(), rhs)
    • Complex expressions are translated into specific implementations such as BLAS.

Element-wise Operations

The basic binary operators are overloaded to composite Mapper expressions, so we can write

weight = (-eta) * (grad + lambda * weight);

We can also use customized binary operators, and unary operators:

struct maximum {
  MSHADOW_XINLINE static float Map(float a, float b) {
    return a > b ? a : b;
  }
};
template<typename xpu>
void ExampleMaximum(Tensor<xpu, 2> out,
                    const Tensor<xpu, 2> &A,
                    const Tensor<xpu, 2> &B) {
  out= 10.0f * F<maximum>(A+1.0f, B); 
}
struct sigmoid {
  MSHADOW_XINLINE static float Map(float a) {
    return 1.0f/(1.0f+expf(-a));
  }
};
template<typename xpu>
void ExampleSigmoid(Tensor<xpu, 2> out, const Tensor<xpu, 2> &in) {
  // equivalent to out = sigmoid(in*2) + 1; 
  out = F<op::plus>(F<sigmoid>(in * 2.0f), ScalarExp(1.0f));
}

Matrix Multiplications

Matrix multiplications are supported by following syntax, with things brackets [] are optional

dst <sv> [scale*] dot(lhs [.T()] , rhs [.T()]), <sv> can be =,+=,-=

Example:

template<typename xpu>
void Backprop(Tensor<xpu, 2> gradin,
              const Tensor<xpu, 2> &gradout,
              const Tensor<xpu, 2> &netweight) {
  gradin = 2.0 * dot(gradout, netweight.T());
}

Introducing Expression Extensions

Naming conventions:

  • Tensor<xpu, dim> to refer to any Tensor with device any device and dimension.
  • xpu, dim, are implicit template parameters.
  • Expr<xpu, dim> will be used to refer to any mapper expression with type Tensor<xpu,dim>.

List of functions:

  • reshape: reshapes a tensor to another shape, number of content must be same
  • broadcast<?>: replicate a 1 dimension tensor in certain dimension
  • repmat, special case of broadcast<0>: repeat vector over rows to form a matrix
  • sumall_except_dim<?>: sum over all the dimensions, except the dimension specified in template parameter
  • sum_rows: special case of sumall_except_dim<0>, sum of rows in the matrix
  • unpack_patch2col: unpack local (overlap) patches of image to column of mat, can be used to implement convolution
  • pack_col2patch: reverse operation of unpack_patch2col, can be used to implement deconvolution
  • pool: do pooling on image
  • unpool: get gradient of pooling result
  • crop: crop the original image to a smaller size
  • mirror: get the mirrored result of input expression

======

reshape
  • reshape(Expr<xpu,dim> src, Shape<dimdst> oshape)
  • reshapes a tensor to another shape, total number of elements must be same
  • parameters:
    • src: input data
    • oshape: target shape
  • result expression type: Tensor<xpu, dimdst> with shape=oshape, is Mapper expression
void ExampleReshape(void) {
  Tensor<cpu, 2> dst = NewTensor<cpu>(Shape2(4, 5));
  Tensor<cpu, 1> src = NewTensor<cpu>(Shape1(20), 1.0f); 
  dst = reshape(src, dst.shape_);
  ...
}

======

broadcast
  • broadcast<dimcast>(Tensor<xpu,1> src, Shape<dimdst> oshape)
  • replicate a 1 dimension tensor certain dimension, specified by template parameter dimcast
  • parameters:
    • src: input 1 dimensional tensor
    • oshape: shape of output
  • return expression type: Tensor<xpu, dimdst>, shape = oshape, is Chainer expression
void ExampleBroadcast(void) {
  Tensor<cpu, 2> dst = NewTensor<cpu>(Shape2(2, 3));
  Tensor<cpu, 1> src = NewTensor<cpu>(Shape1(2), 1.0f);
  src[0] = 2.0f; src[1] = 1.0f;
  dst = broadcast<0>(src, dst.shape_);
  // dst[0][0] = 2, dst[0][1] = 2; dst[1][0]=1, dst[1][1] = 1
  ...
}

======

repmat
  • repmat(Tensor<xpu, 1> src, int nrows)
  • special case of broadcast, repeat 1d tensor over rows
  • input parameters:
    • src: input vector
    • nrows: number of rows in target
  • return expression type: Tensor<xpu, 2>, with shape=(nrows, src.size(0)), is Chainer expression
void ExampleRepmat(void) {
  Tensor<cpu,2> dst = NewTensor<cpu>(Shape2(3, 2));
  Tensor<cpu,1> src = NewTensor<cpu>(Shape1(2), 1.0f);
  src[0] = 2.0f; src[1] = 1.0f;
  dst = repmat(src, 3);
  // dst[0][0] = 2, dst[0][1] = 1; dst[1][0]=2, dst[1][1] = 1
  ...
}

======

sumall_except_dim
  • sumall_except_dim<dimkeep>(Expr<xpu,dim> src)
  • sum over all dimensions, except dimkeep
  • input parameters:
    • src: input mapper expression
  • return expression type: Tensor<xpu, 1>, with shape=(src.size(dimkeep)), is Complex expression
  • Syntax: ```dst [sv] [scale*] sumall_except_dim(src) , can be =, +=, -=, *=, /=````
void ExampleSumAllExceptDim(void) {
  Tensor<cpu,3> src = NewTensor<cpu>(Shape3(2, 3, 2), 1.0f);
  Tensor<cpu,1> dst = NewTensor<cpu>(Shape1(3), 1.0f);
  dst += sum_all_except<1>(src * 2.0f);
  // dst[0] = 1.0 + 4.0 *2.0 = 9.0
  ...
}

======

sum_rows
  • sum_rows(Expr<xpu, 2> src)
  • sum of rows in the matrix
  • input parameters:
    • src: input mapper expression
  • return expression type: Tensor<xpu,1>, with shape=(src.size(0)), is Complex expression
  • Syntax: ```dst [sv] [scale*] sum_rows(src) , can be =,+=,-=,*=,/=````
void ExampleSumRows(void) {
  Tensor<cpu, 2> src = NewTensor<cpu>(Shape2(3, 2), 1.0f);
  Tensor<cpu, 1> dst = NewTensor<cpu>(Shape1(2), 1.0f);
  dst += sum_rows(src + 1.0f);
  // dst[0] = 1.0 + 3.0 *(1.0+1.0) = 7.0
  ...
}

======

unpack_patch2col
  • unpack_patch2col(Expr<xpu,3> img, int psize_y, int p_size_x, int pstride)
  • unpack local (overlap) patches of image to column of mat, can be used to implement convolution, after getting unpacked mat, we can use: output = dot(weight, mat) to get covolved results, the relations:
    • weight; shape[0]: out_channel, shape[1]: ichannel * psize_y * psize_x
    • output; shape[0]: out_channel, shape[1]: out_height * out_width * num_of_images
    • out_height = (in_height - psize_y) / pstride + 1, this means we pad inperfect patch with 0
    • out_width = (in_width - psize_x) / pstride + 1
  • input parameters:
    • img: source image, can be expression; (in_channels, in_height, in_width)
    • psize_y height of each patch
    • psize_x width of each patch
    • pstride: stride of each patch
  • return expression type: Tensor<xpu, 2>, with shape=(in_channel*psize_x*psize_y, out_height*out_width), is Chainer expression
void ExampleCovolution(Tensor<cpu, 3> dst, Tensor<cpu, 3> src,
                       Tensor<cpu, 2> weight, int ksize, int stride) {
  int o_height = (src.size(1)- ksize) / stride + 1;
  int o_width  = (src.size(2)- ksize) / stride + 1;
  utils::Assert(weight.size(1) == src.size(0) * ksize * ksize);
  TensorContainer<cpu, 2> tmp_col(Shape2(src.size(0) * ksize * ksize,
                                         o_height * o_width)); 
  TensorContainer<cpu, 2> tmp_dst(Shape2(weight.size(0),
                                         o_height * o_width)); 
  tmp_col = unpack_patch2col(src, ksize, ksize, stride);
  tmp_dst = dot(weight, tmp_col);
  dst = reshape(tmp_dst, dst.shape_);
}

======

pack_col2patch
  • ```pack_col2patch(Tensor<xpu, 2> mat, Shape<3> imshape, int psize_y, int psize_x, int pstride) ````
  • reverse operation of unpack_patch2col, can be used to implement deconvolution
  • input parameters:
    • mat: source mat, same shape as output of unpack_patch2col
    • imshape: shape of target image
    • psize_y height of each patch
    • psize_x width of each patch
    • pstride: stride of each patch
  • return expression type: Tensor<xpu, 3>, with shape = imshape, is Chainer expression
void ExampleDecovolution(Tensor<cpu, 3> bottom, Tensor<cpu, 3> top,
                         Tensor<cpu, 2> weight, int ksize, int stride) {
  int o_height = (bottom.size(1)- ksize) / stride + 1;
  int o_width  = (bottom.size(2)- ksize) / stride + 1;
  utils::Assert(weight.size(1) == bottom.size(0) * ksize * ksize);
  TensorContainer<cpu, 2> tmp_col(Shape2(bottom.size(0) * ksize * ksize,
                                         o_height * o_width)); 
  TensorContainer<cpu, 2> tmp_dst(Shape2(weight.size(0), o_height*o_width)); 
  tmp_dst = reshape(top, tmp_dst.shape_);
  tmp_col = dot(weight.T(), tmp_dst);
  bottom = pack_col2patch(tmp_col, bottom.shape_, ksize, ksize, stride);
}

======

pool
  • pool<Reducer>(Expr<xpu, dim> img, [Shape<2> pshape,] int ksize_y, int ksize_x, int kstride)
  • Pooling on image with specify kernel size and stride, can be used to implement max pooilng and other pooling layer
  • input parameters:
    • Reducer: operation can be max or sum
    • img: source image, can be expression; (in_channels, in_height, in_width)
    • [optional] Shape<2> pshape, output shape
    • ksize_y height of each patch
    • ksize_x width of each patch
    • kstride: stride of each patch
  • return expression: Expr<xpu, dim>, with shape = (in_channel, (out_height - ksize) / kstride + 1, (out_width - ksize) / kstride + 1), or expression in pshape
    • Chainer expression
void ExampleMaxPooling(TensorContainer<cpu, 3> &data, int ksize, int stride) {
  TensorContainer<cpu, 3> pooled(Shape3(data.size(0),
                                        (data.size(2) - ksize) / kstride + 1), 
                                        (data.size(1) - ksize) / kstride + 1));
  pooled = pool<red::maximum>(data, ksize, ksize, stride);
}

======

unpool
  • unpool<Reducer>(Tensor<xpu, 4> data_src, Tensor<xpu, 4> data_pooled, Tensor<xpu, 4> grad_pooled, int ksize_y, int ksize_x, int kstride)
  • Unpooling on image with specify kernel size and stride, can be used to implement backprop of max pooilng and other pooling layer
  • input parameters:
    • Reducer: operation can be max or sum
    • data_src: source image batch.
    • data_pooled: pooled image batch.
    • grad_pooled: gradient of upper layer
    • ksize_y height of each patch
    • ksize_x width of each patch
    • kstride: stride of each patch
  • return: Expression, same shape to data_src
void ExampleMaxUnpooling(Tensor<cpu, 4> &data_src, Tensor<cpu, 4> &data_pooled, 
                         Tensor<cpu, 4> &grad_pooled, int ksize, int kstride) {
  TensorContainer<cpu, 4> grad(data_src.shape_);
  grad = unpool<red::maximum>(data_src, data_pooled,
                              grad_pooled, ksize, ksize, kstride);
}

======

crop
  • crop(Expr<xpu, dim> src, Shape<2> oshape, int start_height, int start_width)
  • input parameters:
  • src: input expression
  • oshape: output shape after crop
  • start_height: start height for cropping
  • start_width: start width for cropping
  • Can also be crop(Expr<xpu, dim> src, Shape<2> oshape) where the crop will happen in center.
  • return
  • cropped expression
void ExampleCrop(TensorContainer<cpu, 3> img, int start_height, int start_width) {
  TensorContainer<cpu> cropped(Shape3(img.size(0),
                                      img.size(1) - start_height,
                                      img.size(2) - start_width));
  cropped = crop(img, start_height, start_width);
}

======

mirror
  • mirrow(Expr<xpu, dim> src)
  • input:
    • src, source expression to be mirrored
  • output:
    • expression of mirrored result
void ExampleMirror(TensorContainer<cpu, 3> img) {
  TensorContainer<cpu> mirrored(img.shape_);
  mirrored = mirror(img);
}