Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating Array API submodule to the latest commit and minor fix in the array-api tests guide #28732

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs/overview/deep_dive/array_api_tests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ We have made the decision to import the test suite directly from this repository
This means that the test suite you see in the ivy source code cannot be modified in the usual way of pushing to the ivy main branch.
Instead, the change must be made to the array-api repository directly and then our submodule must be updated with the commands:

.. code-block:: none
.. code-block:: bash

# to initialise local config file and fetch + checkout submodule (not needed every time)
git submodule update --init --recursive
Expand All @@ -39,7 +39,7 @@ Instead, the change must be made to the array-api repository directly and then o
Sometimes you will face strange behaviour when staging changes from Ivy's main repo which includes submodule updates.
And this is being caused by your submodule being out of date because we update the submodule iteratively. You can get around this by running the following command:

.. code-block:: none
.. code-block:: bash

# Updating your submodule to the latest commit
git submodule update --remote
Expand All @@ -64,10 +64,10 @@ Using Terminal

Using the terminal, you can run all array-api tests in a given file for a certain backend using the bash file `scripts/shell/test_array_api.sh`_:

.. code-block:: none
.. code-block:: bash

# /ivy
/bin/bash -e ./scripts/shell/scripts/shell/test_array_api.sh jax test_linalg
/bin/bash -e ./scripts/shell/test_array_api.sh jax test_linalg

You can change the argument with any of our supported frameworks - tensorflow, numpy, torch, or jax - and the individual test function categories in :code:`ivy/ivy_tests/array_api_testing/test_array_api/array_api_tests`, e.g. *test_set_functions*, *test_signatures* etc.

Expand All @@ -76,20 +76,20 @@ To make this work, you should set the backend explicitly in the `_array_module.p
At the beginning of the file, you will see the following line of code :code:`array_module = None`.
You need to comment out that line and add the following:

.. code-block:: none
.. code-block:: bash

import ivy as array_module
array_module.set_backend("jax") # or numpy, tensorflow, torch

You should now be able to run the following commands via terminal:

.. code-block:: none
.. code-block:: bash

# run all tests in a file
pytest -vv ivy_tests/array_api_testing/test_array_api/array_api_tests/test_manipulation_functions.py

# run a single test
pytest -vv ivy_tests/array_api_testing/test_array_api/array_api_tests/test_manipulation_functions.py -k "test_concat"
pytest -vv ivy_tests/array_api_testing/test_array_api/array_api_tests/test_manipulation_functions.py::test_concat

Using the IDE
*************
Expand All @@ -108,7 +108,7 @@ However, unlike the ivy tests, the Array-API tests make liberal use of :code:`da
This means that failed tests cannot be re-run with the :code:`@example` decorator, as explained in the :ref:`final section <overview/deep_dive/ivy_tests:Re-Running Failed Ivy Tests>` of the ivy tests deep dive.
Fortunately, it is possible to regenerate test failures using a unique decorator that appears in the final line of the falsifying example in the error stack trace:

.. code-block:: none
.. code-block:: bash

=================================== FAILURES ===================================
______________________ test_remainder[remainder(x1, x2)] _______________________
Expand All @@ -135,7 +135,7 @@ Fortunately, it is possible to regenerate test failures using a unique decorator
Copy the :code:`@reproduce_failure` decorator and paste it after the usual decorators of `test_remainder`.
You may also need to include the hypothesis import of `reproduce_failure` as shown below.

.. code-block:: none
.. code-block:: python

from hypothesis import reproduce_failure

Expand Down
2 changes: 1 addition & 1 deletion ivy_tests/array_api_testing/test_array_api
Submodule test_array_api updated 47 files
+7 −5 .github/workflows/test.yml
+4 −0 .gitignore
+6 −18 README.md
+1 −1 array-api
+8 −0 array-api-strict-skips.txt
+29 −6 array_api_tests/__init__.py
+4 −34 array_api_tests/_array_module.py
+17 −5 array_api_tests/array_helpers.py
+61 −3 array_api_tests/dtype_helpers.py
+110 −62 array_api_tests/hypothesis_helpers.py
+68 −11 array_api_tests/pytest_helpers.py
+4 −2 array_api_tests/shape_helpers.py
+14 −7 array_api_tests/stubs.py
+13 −12 array_api_tests/test_array_object.py
+6 −3 array_api_tests/test_constants.py
+30 −43 array_api_tests/test_creation_functions.py
+7 −9 array_api_tests/test_data_type_functions.py
+328 −0 array_api_tests/test_fft.py
+5 −7 array_api_tests/test_has_names.py
+2 −3 array_api_tests/test_indexing_functions.py
+486 −150 array_api_tests/test_linalg.py
+17 −10 array_api_tests/test_manipulation_functions.py
+80 −63 array_api_tests/test_operators_and_elementwise_functions.py
+15 −11 array_api_tests/test_searching_functions.py
+5 −5 array_api_tests/test_set_functions.py
+1 −3 array_api_tests/test_signatures.py
+4 −4 array_api_tests/test_sorting_functions.py
+76 −130 array_api_tests/test_special_cases.py
+24 −76 array_api_tests/test_statistical_functions.py
+4 −4 array_api_tests/test_utility_functions.py
+53 −26 conftest.py
+1 −0 meta_tests/README.md
+0 −0 meta_tests/__init__.py
+2 −2 meta_tests/test_array_helpers.py
+1 −1 meta_tests/test_broadcasting.py
+1 −1 meta_tests/test_equality_mapping.py
+14 −16 meta_tests/test_hypothesis_helpers.py
+16 −0 meta_tests/test_linalg.py
+4 −4 meta_tests/test_partial_adopters.py
+12 −5 meta_tests/test_pytest_helpers.py
+1 −1 meta_tests/test_signatures.py
+1 −1 meta_tests/test_special_cases.py
+8 −8 meta_tests/test_utils.py
+0 −41 numpy-skips.txt
+0 −0 pytest.ini
+4 −4 reporting.py
+1 −1 requirements.txt
Loading