[Diff since v0.12.7](FluxML/Flux.jl@v0.12.7...v0.12.8)

**Closed issues:**
- Coverage (FluxML#89)
- Flux.train! stops working after the first iteration without an error. (FluxML#1692)
- Update Zygote (FluxML#1728)
- additional arguments to loss function? (FluxML#1730)
- The Purpose and Goals of Flux.jl (FluxML#1734)
- FluxML's NumFOCUS Affiliate project application (FluxML#1740)
- ConvTranspose does not support groups (FluxML#1743)
- `deepcopy(nn::Chain)` does not deep copy with `CuArray` weights! (FluxML#1747)
- `InvalidIRError` when putting a model on the GPU  (FluxML#1754)

**Merged pull requests:**
- remove Manifest (FluxML#1725) (@CarloLucibello)
- add unbatch (FluxML#1726) (@CarloLucibello)
- Adds affine and track_stats params to BatchNorm docstring (FluxML#1729) (@Mottl)
- add some changes to the beginning of docs (FluxML#1736) (@DhairyaLGandhi)
- Fix doc string of Upsample (FluxML#1738) (@chunjiw)
- allow groups in ConvTranspose (FluxML#1744) (@jw3126)
- Fix Saving and loading model output example (FluxML#1746) (@logankilpatrick)
- Fix `train!` doc string 404 (FluxML#1748) (@logankilpatrick)
- Fix @ Functors 404's (FluxML#1749) (@logankilpatrick)
- fix CI build (FluxML#1750) (@DhairyaLGandhi)

[Diff since v0.12.6](FluxML/Flux.jl@v0.12.6...v0.12.7)

**Closed issues:**
- Poor performance relative to PyTorch (FluxML#886)
- Recur struct's fields are not type annotated, which is causing run–time dispatch and a significant slowdowns (FluxML#1092)
- Bug: lower degree polynomial substitute in gradient chain! (FluxML#1188)
- Very slow precompile (>50min) on julia 1.6.0 on Windows (FluxML#1554)
- Do not initialize CUDA during precompilation (FluxML#1597)
- GRU implementation details (FluxML#1671)
- `Parallel` layer doesn't need to be tied to array input (FluxML#1673)
- update! a scalar parameter (FluxML#1677)
- Support NamedTuples for Container Layers (FluxML#1680)
- Freezing layer parameters still computes all gradients (FluxML#1688)
- A demo is 1.5x faster in Flux than tensorflow, both use cpu; while 3.0x slower during using CUDA (FluxML#1694)
- Problems with a mixed CPU/GPU model (FluxML#1695)
- Flux tests with master fail with signal 11 (FluxML#1697)
- [Q] How does Flux.jl work on Apple Silicon (M1)? (FluxML#1701)
- Typos in documents (FluxML#1706)
- Fresh install of Flux giving errors in precompile (FluxML#1710)
- Flux.gradient returns dict of params and nothing (FluxML#1713)
- Weight matrix not updating with a user defined initial weight matrix (FluxML#1717)
- [Documentation] No `logsumexp` in NNlib page (FluxML#1718)
- Flattened data vs Flux.flatten layer in MNIST MLP in the model zoo (FluxML#1722)

**Merged pull requests:**
- Add WIP docstrings to CPU and GPU (FluxML#1632) (@logankilpatrick)
- Add section on Checking GPU Availability (FluxML#1633) (@logankilpatrick)
- fix README (FluxML#1668) (@DhairyaLGandhi)
- Generalise Parallel forwards pass (FluxML#1674) (@DhairyaLGandhi)
- Adding GRUv3 support. (FluxML#1675) (@mkschleg)
- Support NamedTuples for Chain + Parallel (FluxML#1681) (@mcabbott)
- Adding support for folding RNNs over 3d arrays (FluxML#1686) (@mkschleg)
- Update nnlib.md (FluxML#1689) (@CarloLucibello)
- fix typo (FluxML#1691) (@foldfelis)
- Typo fix (FluxML#1693) (@lukemerrick)
- Remove out of date dead code in Conv layers (FluxML#1702) (@ToucheSir)
- Gradient definitions for `cpu` & `gpu` (FluxML#1704) (@mcabbott)
- Fix FluxML#1706 (FluxML#1707) (@rongcuid)
- Add GPU Adaptor (FluxML#1708) (@DhairyaLGandhi)
- Initialize CUDA lazily. (FluxML#1711) (@maleadt)
- Update community.md to reflect help wanted != good first issue (FluxML#1712) (@logankilpatrick)
- Fix link in README (FluxML#1716) (@nilsmartel)
- Add logsumexp to docs (FluxML#1719) (@DhairyaLGandhi)

[Diff since v0.12.5](FluxML/Flux.jl@v0.12.5...v0.12.6)

**Merged pull requests:**
- Add grouped convolution (FluxML#1531) (@DhairyaLGandhi)
- fix deprecations of zeros (FluxML#1670) (@DhairyaLGandhi)
- Add GPU activation tests for grouped conv (FluxML#1672) (@DhairyaLGandhi)

[Diff since v0.12.4](FluxML/Flux.jl@v0.12.4...v0.12.5)

**Closed issues:**
- Hessian vector products (FluxML#129)
- Stopping criteria (FluxML#227)
- Flux + Julia ecosystem docs (FluxML#251)
- RNN unbroadcast on GPU not working (FluxML#421)
- Shouldn't gradcheck compares Jacobian? (FluxML#462)
- Transition examples in docs to doctests (FluxML#561)
- Batch-axis thread parallelism (FluxML#568)
- Add tests of ExpDecay (FluxML#684)
- Sudden memory leak when training on GPU over many epochs (FluxML#736)
- performance variance between macOS / Linux ? (FluxML#749)
- onehot ambiguous method (FluxML#777)
- Killed while training the model (FluxML#779)
- type Method has no field sparam_syms, while @save model (FluxML#783)
- Flux#zygote Error in phenomes... Mutating arrays is not supported (FluxML#819)
- Custom serialization pass for intermediate states (FluxML#845)
- OneHotMatrix does not support map (FluxML#958)
- CuArrays + huber_loss iterate(::nothing) error (FluxML#1128)
- Can't get Flux (v0.10.3) working for Custom Loss function  (FluxML#1153)
- Custom loss function on subset of parameters fails (FluxML#1371)
- Minimizing sum fails (FluxML#1510)
- `gpu` behaves differently from `cu` on a Char array (FluxML#1517)
- Warn different size inputs in loss functions (FluxML#1522)
- Recurrent docs need to be update for v0.12 (FluxML#1564)
- Computation of higher order derivatives for recurrent models results in strange errors (FluxML#1593)
- Why does `DataLoader` not throw an error when fed with a 1D vector for the target? (FluxML#1599)
- a small error in the documentation... (FluxML#1609)
- Slow unnecessary GPU copy of output of `gpu(::OffsetArray)` (FluxML#1610)
- "using Flux" makes type inference fail when there is a Ref{} (FluxML#1611)
- @epochs is missing a bracket (FluxML#1615)
- Flux Overview Documentation Out of Date (FluxML#1621)
- missing kernel for Base.unique (FluxML#1622)
- Compilation error on PPC (FluxML#1623)
- `_restructure` as part of the public API? (FluxML#1624)
- ERROR: setindex! not defined for Zygote.OneElement{...} (FluxML#1626)
- MethodError: Cannot `convert` an object of type Params to an object of type Float64 (FluxML#1629)
- MethodError: no method matching flatten(::Array{Float32,4}) (FluxML#1630)
- Where are the `cpu()` and `gpu()` functions?  (FluxML#1631)
- bug in RNN docs (FluxML#1638)
- Bug in the current overview documentation (FluxML#1642)
- How to tell Flux.jl not to use the GPU? (FluxML#1644)
- Missing docs for @functor (FluxML#1653)
- typo in the docs/overview section right at the beginning (FluxML#1663)

**Merged pull requests:**
- multiplication of {Transpose, Adjoint} of Array and OneHotVector (FluxML#1424) (@gxyd)
- show(::Chain) (FluxML#1467) (@mcabbott)
- Add test for show(io, ::OneHotArray) on GPU (FluxML#1550) (@darsnack)
- document Join and Split error (FluxML#1607) (@magicly)
- fix typo in models overview document (FluxML#1608) (@teamclouday)
- fix AdamW and improve decays docs (FluxML#1612) (@CarloLucibello)
- use ArrayInterface.restructure in update! (FluxML#1613) (@CarloLucibello)
- Warn on reconstruct length mismatch (FluxML#1616) (@ToucheSir)
- Forward map(f, ::OneHotLike) to broadcast (FluxML#1619) (@darsnack)
- Properly move isbits and numeric arrays to GPU (FluxML#1620) (@ToucheSir)
- Update "Composing Optimisers" docs (FluxML#1628) (@StevenWhitaker)
- Fixup `Dataloader`'s docstring (FluxML#1635) (@mcabbott)
- Add warnings for mismatched sizes in losses (FluxML#1636) (@mcabbott)
- updated recurrence.md which fixes FluxML#1564 (FluxML#1637) (@aditkumar72)
- fix recurrence docs (FluxML#1639) (@CarloLucibello)
- Update docstring for `Conv` to clarify feature dimensions (FluxML#1646) (@vivekkumar7089)
- Use correct eltype and rtol in CrossCor tests (FluxML#1650) (@ToucheSir)
- add Functors docs (FluxML#1654) (@DhairyaLGandhi)
- remove Manifest (FluxML#1657) (@CarloLucibello)
- Printing & docstrings for `onehot` / `onehotbatch` (FluxML#1660) (@mcabbott)
- Deprecate `Flux.zeros` (FluxML#1661) (@mcabbott)

[Diff since v0.12.3](FluxML/Flux.jl@v0.12.3...v0.12.4)

**Closed issues:**
- Unable to get gradients of "Dense" models when sparse arrays are involved (FluxML#965)
- Pullback within pullback throws error when using swish activation function (FluxML#1500)
- Stable docs are stuck on v0.11.2  (FluxML#1580)
- LSTM gradient calculation fails on GPU, works on CPU (FluxML#1586)
- BSON.@save model_path * ".bson" model ERROR: type Method has no field ambig (FluxML#1591)
- Too slow hcat of OneHotMatrix. (FluxML#1594)
- Fallback implementation convolution when using Duals (FluxML#1598)
- Bad printing for OneHot* (FluxML#1603)
- SamePad() with even kernel dimensions does not work (only in CUDA) (FluxML#1605)

**Merged pull requests:**
- Add AbstractOptimiser type (FluxML#1325) (@DhairyaLGandhi)
- Add early stopping utils (FluxML#1545) (@queensferryme)
- Add Flux Overview to basics.md (FluxML#1579) (@batate)
- [doc] fix Upsample docstring code block (FluxML#1587) (@johnnychen94)
- fix DataFrames.jl link (FluxML#1589) (@achuchmala)
- optimized hcat of onehot vectors and matrices (FluxML#1595) (@racinmat)
- Use limited array printing for OneHotArrays (FluxML#1604) (@darsnack)

tag to trigger Documenter for v0.12.3

## Flux v0.12.3

[Diff since v0.12.2](FluxML/Flux.jl@v0.12.2...v0.12.3)


**Closed issues:**
- Flux overrides cat behaviour and causes stack overflow (FluxML#1583)

**Merged pull requests:**
- fixes FluxML#1583 (FluxML#1584) (@DhairyaLGandhi)

## Flux v0.12.2

[Diff since v0.12.1](FluxML/Flux.jl@v0.12.1...v0.12.2)


**Closed issues:**
- Cosine_embedding_loss could be added to Flux.jl (FluxML#1094)
- Char RNN errors (FluxML#1215)
- Colab - MethodError: no method matching (::Flux.LSTMCell{... (FluxML#1563)
- Issue with Flux.jl installation (FluxML#1567)
- Issue with Flux.jl installation (FluxML#1568)
- Model no longer type stable when using destructure and restructure (FluxML#1569)

**Merged pull requests:**
- Cuda 3.0 support (FluxML#1571) (@DhairyaLGandhi)

## Flux v0.12.1

[Diff since v0.12.0](FluxML/Flux.jl@v0.12.0...v0.12.1)


**Closed issues:**
- Helper functions for choosing data types for bias and weight in Flux chains? (FluxML#1548)
- LSTM failed to return gradient (FluxML#1551)
- Flux.destructure gives MethodError when used with non-trainable parameters (FluxML#1553)
- Restructure on Dense no longer plays nicely with alternative types (FluxML#1556)

**Merged pull requests:**
- Add Julia 1.6 doc changes to CI (FluxML#1503) (@DhairyaLGandhi)
- Fix FluxML#1556 (FluxML#1557) (@DhairyaLGandhi)
- Minimal fix of FluxML#1556, remove eltype checks  (FluxML#1558) (@mcabbott)

## Flux v0.12.0

[Diff since v0.11.6](FluxML/Flux.jl@v0.11.6...v0.12.0)


**Closed issues:**
- RNN state dimension with batches (FluxML#121)
- Support for additional dimensions in Dense layer (FluxML#282)
- Error messages when CUDNN is not loaded. (FluxML#287)
- Easier way of switching models from cpu to gpu? (FluxML#298)
- how would I implement an echo state network in flux ? (FluxML#336)
- Pkg.update() in Julia 0.6.x gets you an incompatible version of Flux (FluxML#341)
- Indices not defined (FluxML#368)
- Regression with Flux (FluxML#386)
- LSTM sequence processing (FluxML#393)
- Checkpointing (FluxML#402)
- Allowing users to specify their default data folder (FluxML#436)
- elu not working with GPU (FluxML#477)
- Tied Weights (FluxML#488)
- rethinking Conv, and layer granularity in general (FluxML#502)
- σ.() on GPU not using CUDAnative (FluxML#519)
- Using tensorflow and pytorch layers (FluxML#521)
- Abstract layers (FluxML#525)
- Max norm regularisation (FluxML#541)
- Typical accuracy function using onecold with a OneHotMatrix fails to compile on GPU (FluxML#582)
- Export apply!, etc (FluxML#588)
- Better initialization support (FluxML#670)
- Deprecate initialiser keyword arguments (FluxML#671)
- backprop fails on min.(x1,x2) (FluxML#673)
- Adaptive pooling layers in Flux. (FluxML#677)
- CUDAnative (FluxML#682)
- accumulate gradient with the new gradient API? (FluxML#707)
- sigmoid: multiplicative identity only defined for non-square matrices (FluxML#730)
- 1D Conv Broken (FluxML#740)
- Layers and Params should support equality (FluxML#1012)
- InstanceNorm throws a scalar getindex disallowed error on GPU (FluxML#1195)
- Error with GroupNorm on GPU (FluxML#1247)
- Error with BatchNorm/InstanceNorm after Conv1D on GPU (FluxML#1280)
- How to apply L2 regularization to a subset of parameters? (FluxML#1284)
- define `modules` function (FluxML#1294)
- Misleading InstanceNorm documentation? (FluxML#1308)
- ConvTranspose on GPU fails with certain activation functions (FluxML#1350)
- Conv with non homogenous array eltypes gives confusing error message (FluxML#1421)
- Layers' docstrings and constructors inconsistencies (FluxML#1422)
- BatchNorm alters its sliding mean/standard deviation parameters even in testmode if Zygote is called (FluxML#1429)
- BatchNorm on CUDA accepts improper channel size argument and "works" in a possibly ill-defined way. Proper errors on CPU (FluxML#1430)
- Better handling for layers with multiple inputs w/ outputsize (FluxML#1466)
- Dense function does not support tensor? (FluxML#1480)
- Cannot load model saved with JLD  (FluxML#1482)
- RNN and GRU give mutation error; LSTM gives ArgumentError about number of fields (FluxML#1483)
- Moving OneHotMatrix to GPU triggers the slow scalar operations (FluxML#1494)
- Does gain do anything in kaiming_uniform? (FluxML#1498)
- Zeros has old behaviour on releases up to 0.11.6 (FluxML#1507)
- getting this -> ERROR: Mutating arrays is not supported (solved) (FluxML#1512)
- Moving multihead attention from transformers.jl into Flux.jl (FluxML#1514)
- Gradient cannot be got under testmode gpu net with Batchnorm (FluxML#1520)
- Development version document example on Dense layer's bias not working (FluxML#1523)
- how to use `flatten` layer? (it does not flatten arrays) (FluxML#1525)
- Ambiguity in recurrent neural network training (FluxML#1528)
- scalar indexing when showing OneHot gpu (FluxML#1532)
- Acitvation function relu terrible performance (FluxML#1537)
- Error on precompile (FluxML#1539)
- Flux.normalise vs standardise (FluxML#1541)
- Cudnn batchnorm causes errors when I disable BatchNorm when training  (FluxML#1542)
- DimensionMismatch("All data should contain same number of observations") (FluxML#1543)
- Softmax stucks the network (FluxML#1546)

**Merged pull requests:**
- Added Bilinear layer (FluxML#1009) (@bhvieira)
- Rework normalization layers (FluxML#1397) (@CarloLucibello)
- `Dense` keyword handling, and docstring (FluxML#1440) (@mcabbott)
- define modules function (FluxML#1444) (@CarloLucibello)
- Use fallback for reshape/cat OneHotArray (FluxML#1459) (@darsnack)
- add Upsample and PixelShuffle layers (FluxML#1468) (@CarloLucibello)
- Add activation tests for GPU layers (FluxML#1472) (@DhairyaLGandhi)
- CompatHelper: bump compat for "Functors" to "0.2" (FluxML#1474) (@github-actions[bot])
- reexport compat (FluxML#1478) (@DhairyaLGandhi)
- add FluxBot (FluxML#1484) (@DhairyaLGandhi)
- Make `outputsize` understand multiple inputs (FluxML#1486) (@mcabbott)
- Add training loop to docs (FluxML#1488) (@DhairyaLGandhi)
- Implementation of Focal loss (FluxML#1489) (@shikhargoswami)
- Make Dropout docstring clear w.r.t. N-D dropout (FluxML#1490) (@darsnack)
- Update ecosystem.md (FluxML#1491) (@churchofthought)
- Add Orthogonal initialization feature. (FluxML#1496) (@SomTambe)
- Fix docs syntax for Join/ Split layers (FluxML#1497) (@DhairyaLGandhi)
- Fix layer init functions kwargs getting overwritten (FluxML#1499) (@DevJac)
- updated DataLoader, added a optional keyword argument rng. (FluxML#1501) (@norci)
- Add Parallel GPU tests (FluxML#1505) (@darsnack)
- Add ParameterSchedulers.jl to docs (FluxML#1511) (@darsnack)
- Update for latest ParameterSchedulers.jl release (FluxML#1513) (@darsnack)
- Fixes to Recurrent models for informative type mismatch error & output Vector for Vector input (FluxML#1521) (@jeremiedb)
- Add identity_init (FluxML#1524) (@DrChainsaw)
- fix print layernorm (FluxML#1526) (@CarloLucibello)
- preparatory work for v0.12 (FluxML#1527) (@CarloLucibello)
- small refactor of Dense (FluxML#1533) (@CarloLucibello)
- Fix printing of OneHotArray in REPL when CUDA.allowscalar(false) (FluxML#1534) (@darsnack)
- add vgg16 performance script (FluxML#1535) (@CarloLucibello)
- fx norm deprecation (FluxML#1538) (@CarloLucibello)
- add news entry for end of deprecation cycle (FluxML#1540) (@CarloLucibello)