Maintenance on tests, CI and contributor docs (zarr-developers#468)

* upgrade latest numpy, add previous numpys to tox matrix

* simplify requirements using environment markers

* tweak environment markers

* no npy117 for py27

* upgrade pinnings

* try upgrading build image

* appveyor thrash

* numpy 1.16 on py27

* numpy 1.16 on py27

* test with less numpys

* downgrade to try if solves emulator compat

* tidy up tox [ci skip]

* release notes [ci skip]

* remove requirements.txt, not needed

* remove requirements.txt, not needed

* rework requirements so easier to run tests without services

* fix pyup config

* use doctestplus

* use pytest-remotedata

* pytest 5 not on py27

* deal with numpy argument change in unravel_index

* deal with packages not on py27

* rework running of service tests using environment variables

* use global variables for travis

* update pyup

* update pyup

.pyup.yml

@@ -4,21 +4,15 @@
 schedule: every month
 
 requirements:
-  - requirements.txt:
-      pin: False
-      update: False
-  - requirements_test.txt:
-      pin: False
-      update: False
-  - requirements_rtfd.txt:
-      pin: False
-      update: False
-  - requirements_dev.txt:
+  - requirements_dev_minimal.txt:
       pin: True
       update: all
-  - requirements_dev_npy.txt:
+  - requirements_dev_numpy.txt:
       pin: True
       update: all
   - requirements_dev_optional.txt:
       pin: True
       update: all
+  - requirements_rtfd.txt:
+      pin: False
+      update: False

.travis.yml

@@ -11,6 +11,12 @@ addons:
     packages:
     - libdb-dev
 
+env:
+  global:
+    - ZARR_TEST_ABS=1
+    - ZARR_TEST_MONGO=1
+    - ZARR_TEST_REDIS=1
+
 services:
   - docker
   - redis-server

appveyor.yml

@@ -14,6 +14,7 @@ environment:
     # See: http://stackoverflow.com/a/13751649/163740
     CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\build.cmd"
     EMULATOR_LOC: C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\Storage Emulator\\AzureStorageEmulator.exe
+    ZARR_TEST_ABS: 1
 
   matrix:
 
@@ -33,16 +34,15 @@ environment:
 install:
   - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
   - "%CMD_IN_ENV% python -m pip install -U pip setuptools wheel"
-  - "%CMD_IN_ENV% python -m pip install -rrequirements_test.txt"
-  - "%CMD_IN_ENV% python -m pip install -rrequirements_dev_npy.txt"
-  - "%CMD_IN_ENV% python -m pip install --no-binary=numcodecs -rrequirements_dev.txt"
+  - "%CMD_IN_ENV% python -m pip install -r requirements_dev_numpy.txt -r requirements_dev_minimal.txt -r requirements_dev_optional.txt"
   - "%CMD_IN_ENV% python setup.py install"
   - "%CMD_IN_ENV% python -m pip freeze"
 
 build: off
 
 before_test:
   - '"%EMULATOR_LOC%" start'
+  - '"%EMULATOR_LOC%" status'
 
 test_script:
   - "%CMD_IN_ENV% python -m pytest -v --pyargs zarr"

docs/contributing.rst

@@ -90,10 +90,9 @@ you have cloned the Zarr source code and your current working directory is the r
 the repository, you can do something like the following::
 
     $ mkdir -p ~/pyenv/zarr-dev
-    $ virtualenv --no-site-packages --python=/usr/bin/python3.6 ~/pyenv/zarr-dev
+    $ virtualenv --no-site-packages --python=/usr/bin/python3.7 ~/pyenv/zarr-dev
     $ source ~/pyenv/zarr-dev/bin/activate
-    $ pip install -r requirements_dev.txt
-    $ pip install -r requirements_dev_optional.txt
+    $ pip install -r requirements_dev_minimal.txt -r requirements_dev_numpy.txt
     $ pip install -e .
 
 To verify that your development environment is working, you can run the unit tests::
@@ -139,34 +138,44 @@ Again, any conflicts need to be resolved before submitting a pull request.
 Running the test suite
 ~~~~~~~~~~~~~~~~~~~~~~
 
-Zarr includes a suite of unit tests, as well as doctests included in function and class
-docstrings and in the tutorial and storage spec. The simplest way to run the unit tests
-is to invoke::
+Zarr includes a suite of unit tests, as well as doctests included in
+function and class docstrings and in the tutorial and storage
+spec. The simplest way to run the unit tests is to activate your
+development environment (see `creating a development environment`_ above)
+and invoke::
 
     $ pytest -v zarr
 
-To also run the doctests within docstrings, run::
-
-    $ pytest -v --doctest-modules zarr
+Some tests require optional dependencies to be installed, otherwise
+the tests will be skipped. To install all optional dependencies, run::
 
-To run the doctests within the tutorial and storage spec, run::
+    $ pip install -r requirements_dev_optional.txt
 
-    $ python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst docs/spec/v2.rst
+To also run the doctests within docstrings (requires optional
+depencies to be installed), run::
 
-Tests can be run under different Python versions using tox. E.g. (assuming you have the
-corresponding Python interpreters installed on your system)::
+    $ pytest -v --doctest-plus zarr
 
-    $ tox -e py27,py34,py35,py36
+To run the doctests within the tutorial and storage spec (requires
+optional dependencies to be installed), run::
 
-Zarr currently supports Python 2.7 and Python 3.4-3.6, so the above command must
-succeed before code can be accepted into the main code base. Note that only the py36
-tox environment runs the doctests, i.e., doctests only need to succeed under Python 3.6.
+    $ python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst docs/spec/v2.rst
 
-All tests are automatically run via Travis (Linux) and AppVeyor (Windows) continuous
-integration services for every pull request. Tests must pass under both services before
-code can be accepted. Test coverage is also collected automatically via the Coveralls
-service, and total coverage over all builds must be 100% (although individual builds
-may be lower due to Python 2/3 or other differences).
+Note that some tests also require storage services to be running
+locally. To run the Azure Blob Service storage tests, run an Azure
+storage emulator (e.g., azurite) and set the environment variable
+``ZARR_TEST_ABS=1``. To run the Mongo DB storage tests, run a Mongo
+server locally and set the environment variable ``ZARR_TEST_MONGO=1``.
+To run the Redis storage tests, run a Redis server locally on port
+6379 and set the environment variable ``ZARR_TEST_REDIS=1``.
+
+All tests are automatically run via Travis (Linux) and AppVeyor
+(Windows) continuous integration services for every pull
+request. Tests must pass under both Travis and Appveyor before code
+can be accepted. Test coverage is also collected automatically via the
+Coveralls service, and total coverage over all builds must be 100%
+(although individual builds may be lower due to Python 2/3 or other
+differences).
 
 Code standards
 ~~~~~~~~~~~~~~
@@ -177,8 +186,6 @@ Conformance can be checked by running::
 
     $ flake8 --max-line-length=100 zarr
 
-This is automatically run when invoking ``tox -e py36``.
-
 Test coverage
 ~~~~~~~~~~~~~
 
@@ -195,10 +202,15 @@ request. Coveralls coverage must also be 100% before code can be accepted.
 Documentation
 ~~~~~~~~~~~~~
 
-Docstrings for user-facing classes and functions should follow the `numpydoc
-<https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_ standard,
-including sections for Parameters and Examples. All examples should run and pass as doctests
-under Python 3.6 only.
+Docstrings for user-facing classes and functions should follow the
+`numpydoc
+<https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_
+standard, including sections for Parameters and Examples. All examples
+should run and pass as doctests under Python 3.7. To run doctests,
+activate your development environment, install optional requirements,
+and run::
+
+    $ pytest -v --doctest-plus zarr
 
 Zarr uses Sphinx for documentation, hosted on readthedocs.org. Documentation is
 written in the RestructuredText markup language (.rst files) in the ``docs`` folder.
@@ -208,7 +220,7 @@ folder. Any new features or important usage information should be included in th
 tutorial (``docs/tutorial.rst``). Any changes should also be included in the release
 notes (``docs/release.rst``).
 
-The documentation can be built by running::
+The documentation can be built locally by running::
 
     $ tox -e docs
 

docs/release.rst

@@ -6,11 +6,14 @@ Upcoming Release
 
 * Use uniform chunking for all dimensions when specifying ``chunks`` as an integer.
   Also adds support for specifying ``-1`` to chunk across an entire dimension.
-  By :user:`James Bourbeau <jrbourbeau>`; :issue:`456`
+  By :user:`James Bourbeau <jrbourbeau>`; :issue:`456`.
 
 * Rename ``DictStore`` to ``MemoryStore``.
-  By :user:`James Bourbeau <jrbourbeau>`; :issue:`455`
+  By :user:`James Bourbeau <jrbourbeau>`; :issue:`455`.
 
+* Upgrade dependencies in the test matrices and resolve a
+  compatibility issue with testing against the Azure Storage
+  Emulator. By :user:`alimanfoo`; :issue:`468`, :issue:`467`.
 
 .. _release_2.3.2:
 
@@ -21,19 +24,19 @@ Enhancements
 ~~~~~~~~~~~~
 
 * Use ``scandir`` in ``DirectoryStore``'s ``getsize`` method.
-  By :user:`John Kirkham <jakirkham>`; :issue:`431`
+  By :user:`John Kirkham <jakirkham>`; :issue:`431`.
 
 Bug fixes
 ~~~~~~~~~
 
 * Add and use utility functions to simplify reading and writing JSON.
-  By :user:`John Kirkham <jakirkham>`; :issue:`429`, :issue:`430`
+  By :user:`John Kirkham <jakirkham>`; :issue:`429`, :issue:`430`.
 
 * Fix ``collections``'s ``DeprecationWarning``\ s.
-  By :user:`John Kirkham <jakirkham>`; :issue:`432`
+  By :user:`John Kirkham <jakirkham>`; :issue:`432`.
 
 * Fix tests on big endian machines.
-  By :user:`Elliott Sales de Andrade <QuLogic>`; :issue:`427`
+  By :user:`Elliott Sales de Andrade <QuLogic>`; :issue:`427`.
 
 
 .. _release_2.3.1:

docs/tutorial.rst

@@ -806,10 +806,10 @@ The class is :class:`zarr.storage.ABSStore` (requires
  `azure-storage-blob <https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python>`_
 to be installed)::
 
-    >>> store = zarr.ABSStore(container='test', prefix='zarr-testing', blob_service_kwargs={'is_emulated': True})
-    >>> root = zarr.group(store=store, overwrite=True)
-    >>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
-    >>> z[:] = 42
+    >>> store = zarr.ABSStore(container='test', prefix='zarr-testing', blob_service_kwargs={'is_emulated': True})  # doctest: +SKIP
+    >>> root = zarr.group(store=store, overwrite=True)  # doctest: +SKIP
+    >>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')  # doctest: +SKIP
+    >>> z[:] = 42  # doctest: +SKIP
 
 When using an actual storage account, provide ``account_name`` and
 ``account_key`` arguments to :class:`zarr.storage.ABSStore`, the

requirements_dev_minimal.txt

@@ -0,0 +1,9 @@
+# library requirements
+asciitree==0.3.3
+fasteners==0.15
+numcodecs==0.6.3
+msgpack-python==0.5.6
+setuptools-scm==3.3.3
+# test requirements
+pytest==5.0.1; python_version > '3.0'
+pytest==4.6.5; python_version < '3.0'

requirements_dev_npy.txt → requirements_dev_numpy.txt

@@ -1,4 +1,5 @@
 # Break this out into a separate file to allow testing against
 # different versions of numpy. This file should pin to the latest
 # numpy version.
-numpy==1.15.4
+numpy==1.17.0; python_version > '3.0'
+numpy==1.16.4; python_version < '3.0'

requirements_dev_optional.txt

@@ -1,5 +1,17 @@
-# These packages are currently not available on Windows.
-bsddb3==6.2.6
-lmdb==0.94
-redis==3.0.1
-pymongo==3.7.1
+# optional library requirements
+bsddb3==6.2.6; sys_platform != 'win32'
+lmdb==0.97; sys_platform != 'win32'
+# optional library requirements for services
+azure-storage-blob==2.0.1
+redis==3.3.6
+pymongo==3.8.0
+# optional test requirements
+tox==3.13.2
+coverage==4.5.4
+coveralls==1.8.2
+flake8==3.7.8
+pytest-cov==2.7.1
+pytest-doctestplus==0.3.0
+pytest-remotedata==0.3.2
+h5py==2.9.0
+s3fs==0.3.3; python_version > '3.0'

tox.ini

@@ -4,9 +4,7 @@
 # and then run "tox" from this directory.
 
 [tox]
-# N.B., test different versions of numpy under py36 rather than py37
-# because wheels for npy113 not available for py37
-envlist = py27, py35, py36-npy{113,114,115}, py37, docs
+envlist = py27, py35, py36, py37-npy{115,116,latest}, docs
 
 [testenv]
 install_command = pip install --no-binary=numcodecs {opts} {packages}
@@ -15,35 +13,37 @@ setenv =
     # hooks for coverage exclusions based on Python major version
     py35,py36,py37: PY_MAJOR_VERSION = py3
     py27: PY_MAJOR_VERSION = py2
+passenv =
+    ZARR_TEST_ABS
+    ZARR_TEST_MONGO
+    ZARR_TEST_REDIS
 commands =
     # clear out any data files generated during tests
     python -c 'import glob; import shutil; import os; [(shutil.rmtree(d) if os.path.isdir(d) else os.remove(d) if os.path.isfile(d) else None) for d in glob.glob("./example*")]'
     # main unit test runner
-    # N.B., don't run npy113 or npy114 with coverage because it is run together npy115 on travis
-    py27,py35,py36-npy115: pytest -v --cov=zarr --cov-config=.coveragerc zarr
-    py36-npy113: pytest -v zarr
-    py36-npy114: pytest -v zarr
-    py37: pytest -v --cov=zarr --cov-config=.coveragerc --doctest-modules zarr
-    # generate a coverate report
-    py27,py35,py36-npy115,py37: coverage report -m
+    py27,py35,py36: pytest -v --cov=zarr --cov-config=.coveragerc zarr
+    # don't collect coverage when running older numpy versions
+    py37-{npy115,npy116}: pytest -v zarr
+    # collect coverage and run doctests under py37
+    py37-npylatest: pytest -v --cov=zarr --cov-config=.coveragerc --doctest-plus zarr --remote-data
+    # generate a coverage report
+    py27,py35,py36,py37-npylatest: coverage report -m
     # run doctests in the tutorial and spec
-    py37: python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst docs/spec/v2.rst
+    py37-npylatest: python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst docs/spec/v2.rst
     # pep8 checks
-    py37: flake8 zarr
+    py37-npylatest: flake8 zarr
     # print environment for debugging
     pip freeze
 deps =
     py27: backports.lzma
-    py36-npy113: numpy==1.13.3
-    py36-npy114: numpy==1.14.6
-    py27,py35,py36-npy115,py37: -rrequirements_dev_npy.txt
-    -rrequirements_test.txt
-    -rrequirements_dev.txt
-    # linux only
+    py37-npy115: numpy==1.15.4
+    py37-npy116: numpy==1.16.4
+    py27,py35,py36,py37-npylatest: -rrequirements_dev_numpy.txt
+    -rrequirements_dev_minimal.txt
     -rrequirements_dev_optional.txt
 
 [testenv:docs]
-basepython = python2.7
+basepython = python3.6
 changedir = docs
 deps =
     -rrequirements_rtfd.txt

zarr/indexing.py

@@ -693,7 +693,11 @@ def __init__(self, selection, array):
         self.chunk_rixs = np.nonzero(self.chunk_nitems)[0]
 
         # unravel chunk indices
-        self.chunk_mixs = np.unravel_index(self.chunk_rixs, dims=array._cdata_shape)
+        if tuple(map(int, np.__version__.split('.')[:2])) < (1, 16):
+            self.chunk_mixs = np.unravel_index(self.chunk_rixs, dims=array._cdata_shape)
+        else:
+            # deal with change dims->shape in arguments as of numpy 1.16
+            self.chunk_mixs = np.unravel_index(self.chunk_rixs, shape=array._cdata_shape)
 
     def __iter__(self):