diff --git a/README.md b/README.md index dd762b692cb..df7e3ee557c 100644 --- a/README.md +++ b/README.md @@ -188,18 +188,37 @@ Run frontend tests: #### Cypress -Cypress integration tests are run via [Cypress.io](https://www.cypress.io/). +Cypress integration tests are implemented in [Cypress.io](https://www.cypress.io/). Launch Cypress test runner: ``` cd frontend oc login ... -yarn run test-cypress +yarn run test-cypress-console ``` -This will launch the Cypress test runner where you can run one or all cypress tests. +This will launch the Cypress Test Runner UI in the `console` package, where you can run one or all cypress tests. By default, it will look for Chrome in the system and use it, but if you want to use Firefox instead, set `BRIDGE_E2E_BROWSER_NAME` environment variable in your shell with the value `firefox`. +##### Execute Cypress in different packages + +An alternate way to execute cypress tests is via [test-cypress.sh](test-cypress.sh) which takes a `-p ` parameter to allow execution in different packages. It also can run Cypress tests in the Test Runner UI or in `-- headless` mode: +``` +console>./test-cypress.sh +Runs Cypress tests in Test Runner or headless mode +Usage: test-cypress [-p] [-s] [-h true] + '-p ' may be 'console, 'olm' or 'devconsole' + '-s ' is a file mask for spec test files, such as 'tests/monitoring/*'. Used only in headless mode when '-p' is specified. + '-h true' runs Cypress in headless mode. When omitted, launches Cypress Test Runner +Examples: + test-cypress.sh // displays this help text + test-cypress.sh -p console // opens Cypress Test Runner for console tests + test-cypress.sh -p olm // opens Cypress Test Runner for OLM tests + test-cypress.sh -h true // runs all packages in headless mode + test-cypress.sh -p olm -h true // runs OLM tests in headless mode + test-cypress.sh -p console -s 'tests/crud/*' -h true // runs console CRUD tests in headless mode +``` + [**_More information on Console's Cypress usage_**](frontend/packages/integration-tests-cypress/README.md) #### Protractor @@ -215,12 +234,12 @@ cd frontend && yarn run webdriver-update Run integration tests: ``` -yarn run test-gui +yarn run test-protractor ``` Run integration tests on an OpenShift cluster: ``` -yarn run test-gui-openshift +yarn run test-protractor-openshift ``` This will include the normal k8s CRUD tests and CRUD tests for OpenShift resources. @@ -239,55 +258,31 @@ For macOS, you can use: ``` yarn run webdriver-update-macos ``` -#### How the Integration Tests Run in CI - -The end-to-end tests run against pull requests using [ci-operator](https://github.com/openshift/ci-operator/). -The tests are defined in [this manifest](https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/console/openshift-console-master-presubmits.yaml) -in the [openshift/release](https://github.com/openshift/release) repo and were generated with [ci-operator-prowgen](https://github.com/openshift/ci-operator-prowgen). - -CI runs the [test-prow-e2e.sh](test-prow-e2e.sh) script, which uses the `e2e` suite defined in [protractor.conf.ts](frontend/integration-tests/protractor.conf.ts). - -You can simulate an e2e run against an existing 4.0 cluster with the following commands (replace `/path/to/install-dir` with your OpenShift 4.0 install directory): - -``` -$ oc apply -f ./frontend/integration-tests/data/htpasswd-secret.yaml -$ oc patch oauths cluster --patch "$(cat ./frontend/integration-tests/data/patch-htpasswd.yaml)" --type=merge -$ export BRIDGE_BASE_ADDRESS="$(oc get consoles.config.openshift.io cluster -o jsonpath='{.status.consoleURL}')" -$ export BRIDGE_KUBEADMIN_PASSWORD=$(cat "/path/to/install-dir/auth/kubeadmin-password") -$ ./test-gui.sh e2e -``` - -If you don't want to run the entire e2e tests, you can use a different suite from [protractor.conf.ts](frontend/integration-tests/protractor.conf.ts). For instance, - -``` -$ ./test-gui.sh -``` - ##### Hacking Protractor Tests To see what the tests are actually doing, it is posible to run in none `headless` mode by setting the `NO_HEADLESS` environment variable: ``` -$ NO_HEADLESS=true ./test-gui.sh +$ NO_HEADLESS=true ./test-protractor.sh ``` To use a specific binary version of chrome, it is posible to set the `CHROME_BINARY_PATH` environment variable: ``` -$ CHROME_BINARY_PATH="/usr/bin/chromium-browser" ./test-gui.sh +$ CHROME_BINARY_PATH="/usr/bin/chromium-browser" ./test-protractor.sh ``` To avoid skipping remaining portion of tests upon encountering the first failure, `NO_FAILFAST` environment variable can be used: ``` -$ NO_FAILFAST=true ./test-gui.sh +$ NO_FAILFAST=true ./test-protractor.sh ``` ##### Debugging Protractor Tests 1. `cd frontend; yarn run build` 2. Add `debugger;` statements to any e2e test -3. `yarn run debug-test-suite --suite ` +3. `yarn run debug-protractor-suite --suite ` 4. Chrome browser URL: 'chrome://inspect/#devices', click on the 'inspect' link in **Target (v10...)** section. 5. Launches chrome-dev tools, click Resume button to continue 6. Will break on any `debugger;` statements @@ -299,22 +294,30 @@ The end-to-end tests run against pull requests using [ci-operator](https://githu The tests are defined in [this manifest](https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/console/openshift-console-master-presubmits.yaml) in the [openshift/release](https://github.com/openshift/release) repo and were generated with [ci-operator-prowgen](https://github.com/openshift/ci-operator-prowgen). -CI runs the [test-prow-e2e.sh](test-prow-e2e.sh) script, which runs the cypress tests and the protractor `e2e` test suite defined in [protractor.conf.ts](frontend/integration-tests/protractor.conf.ts). +CI runs the [test-prow-e2e.sh](test-prow-e2e.sh) script, which runs [test-cypress.sh](test-cypress.sh) and ['test-protractor.sh e2e'](test-protractor.sh), which runs the protractor `e2e` test suite. + +##### Cypress in CI +The CI executes [test-cypress.sh](test-cypress.sh) to run all Cypress tests, in all 'packages' (console, olm, and devconsole), in `-- headless` mode via: + +`test-cypress.sh -h true` -You can simulate an e2e run against an existing 4.0 cluster with the following commands (replace `/path/to/install-dir` with your OpenShift 4.0 install directory): +For more information on `test-cypress.sh` usage please see [Execute Cypress in different packages](#execute-cypress-in-different-packages) +##### Protractor in CI +['test-protractor.sh e2e'](test-protractor.sh) runs the protractor `e2e` test suite defined in [protractor.conf.ts](frontend/integration-tests/protractor.conf.ts) +You can simulate an e2e run against an existing cluster with the following commands (replace `/path/to/install-dir` with your OpenShift install directory): ``` $ oc apply -f ./frontend/integration-tests/data/htpasswd-secret.yaml $ oc patch oauths cluster --patch "$(cat ./frontend/integration-tests/data/patch-htpasswd.yaml)" --type=merge $ export BRIDGE_BASE_ADDRESS="$(oc get consoles.config.openshift.io cluster -o jsonpath='{.status.consoleURL}')" $ export BRIDGE_KUBEADMIN_PASSWORD=$(cat "/path/to/install-dir/auth/kubeadmin-password") -$ ./test-gui.sh e2e +$ ./test-protractor.sh e2e ``` If you don't want to run the entire e2e tests, you can use a different suite from [protractor.conf.ts](frontend/integration-tests/protractor.conf.ts). For instance, ``` -$ ./test-gui.sh +$ ./test-protractor.sh ``` ### Deploying a Custom Image to an OpenShift Cluster diff --git a/frontend/integration-tests/protractor.conf.ts b/frontend/integration-tests/protractor.conf.ts index 4f4dfb36379..c3215f45625 100644 --- a/frontend/integration-tests/protractor.conf.ts +++ b/frontend/integration-tests/protractor.conf.ts @@ -242,7 +242,7 @@ export const config = { params: { // Set to 'true' to enable OpenShift resources in the crud scenario. // Use a string rather than boolean so it can be specified on the command line: - // $ yarn test-gui --params.openshift true + // $ yarn test-protractor --params.openshift true openshift: 'false', // Set to 'true' to enable Service Catalog resources in the crud scenario. servicecatalog: 'false', diff --git a/frontend/package.json b/frontend/package.json index 258a7fa3842..9e12cea2226 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -22,10 +22,11 @@ "webdriver-update": "webdriver-manager update", "webdriver-update-macos": "CHROME_VERSION=$(/Applications/Google\\ Chrome.app/Contents/MacOS/Google\\ Chrome --version) && yarn webdriver-update --versions.chrome=\"${CHROME_VERSION}\"", "webdriver-update-fedora": "CHROME_VERSION=$(/usr/bin/google-chrome-stable --version) && yarn webdriver-update --versions.chrome=\"${CHROME_VERSION}\"", - "test-gui-tap": "TAP=true yarn test-gui", - "test-gui-openshift": "yarn test-suite --suite crud --params.openshift true", - "test-gui": "yarn test-suite --suite all", - "test-suite": "yarn ts-node ./node_modules/.bin/protractor integration-tests/protractor.conf.ts", + "test-protractor-tap": "TAP=true yarn test-protractor", + "test-protractor-openshift": "yarn test-protractor-suite --suite crud --params.openshift true", + "test-protractor": "yarn test-protractor-suite --suite all", + "test-protractor-suite": "yarn ts-node ./node_modules/.bin/protractor integration-tests/protractor.conf.ts", + "debug-protractor-suite": "TS_NODE_COMPILER_OPTIONS='{\"module\":\"commonjs\"}' node -r ts-node/register --inspect-brk ./node_modules/.bin/protractor integration-tests/protractor.conf.ts", "test-cypress-console": "cd packages/integration-tests-cypress && cypress open --env openshift=true", "test-cypress-olm": "cd packages/operator-lifecycle-manager/integration-tests-cypress && ../../../node_modules/.bin/cypress open --config-file cypress-olm.json --env openshift=true", "test-cypress-console-headless": "cd packages/integration-tests-cypress && node --max-old-space-size=4096 ../../node_modules/.bin/cypress run --env openshift=true --browser ${BRIDGE_E2E_BROWSER_NAME:=chrome} --headless", @@ -34,7 +35,6 @@ "cypress-generate": "marge -o ./gui_test_screenshots/ -f cypress-report -t 'OpenShift Console Cypress Test Results' -p 'OpenShift Cypress Test Results' --showPassed false --assetsDir ./gui_test_screenshots/cypress/assets ./gui_test_screenshots/cypress.json", "cypress-a11y-report": "echo '\nA11y Test Results:' && mv packages/integration-tests-cypress/cypress-a11y-report.json ./gui_test_screenshots/ && node -e \"console.table(JSON.parse(require('fs').readFileSync(process.argv[1])));\" ./gui_test_screenshots/cypress-a11y-report.json", "cypress-postreport": "yarn cypress-merge && yarn cypress-generate && yarn cypress-a11y-report", - "debug-test-suite": "TS_NODE_COMPILER_OPTIONS='{\"module\":\"commonjs\"}' node -r ts-node/register --inspect-brk ./node_modules/.bin/protractor integration-tests/protractor.conf.ts", "analyze": "NODE_ENV=production NODE_OPTIONS=--max-old-space-size=4096 yarn ts-node ./node_modules/.bin/webpack --mode=production --profile --json | sed '0,/^{/s/^[^{].*//g' > public/dist/stats.json && yarn ts-node ./node_modules/.bin/webpack-bundle-analyzer --mode static -r public/dist/report.html public/dist/stats.json", "prettier-all": "prettier --write '**/*.{js,jsx,ts,tsx,json}'", "ts-node": "ts-node -O '{\"module\":\"commonjs\"}'", diff --git a/frontend/packages/operator-lifecycle-manager/src/components/descriptors/README.md b/frontend/packages/operator-lifecycle-manager/src/components/descriptors/README.md index dd8a48a28b5..17b1eeb5001 100644 --- a/frontend/packages/operator-lifecycle-manager/src/components/descriptors/README.md +++ b/frontend/packages/operator-lifecycle-manager/src/components/descriptors/README.md @@ -120,5 +120,5 @@ To sufficiently test your component, modify `descriptors.scenario.ts` and add a Run the OLM end-to-end tests against a cluster with OLM installed: ```shell -$ yarn test-suite --suite olmFull +$ yarn test-protractor-suite --suite olmFull ``` diff --git a/test-cypress.sh b/test-cypress.sh index f9023d44286..b0b0f7fcaaf 100755 --- a/test-cypress.sh +++ b/test-cypress.sh @@ -35,9 +35,9 @@ if [ $# -eq 0 ]; then fi if [ -n "${headless-}" ] && [ -z "${pkg-}" ]; then - echo "yarn run test-cypress-console-headless" - echo "yarn run test-cypress-olm-headless" - echo "yarn run test-cypress-devconsole-headless" + yarn run test-cypress-console-headless + yarn run test-cypress-olm-headless + # yarn run test-cypress-devconsole-headless exit; fi diff --git a/test-gui.sh b/test-protractor.sh similarity index 75% rename from test-gui.sh rename to test-protractor.sh index bf8b8961085..bffaaa1b457 100755 --- a/test-gui.sh +++ b/test-protractor.sh @@ -16,7 +16,7 @@ else fi if [ $# -gt 0 ] && [ -n "$1" ]; then - yarn run test-suite --suite "$1" --params.openshift true + yarn run test-protractor-suite --suite "$1" --params.openshift true else - yarn run test-gui --params.openshift true + yarn run test-protractor --params.openshift true fi diff --git a/test-prow-e2e.sh b/test-prow-e2e.sh index a5493668771..c725341a3d2 100755 --- a/test-prow-e2e.sh +++ b/test-prow-e2e.sh @@ -35,7 +35,7 @@ export DBUS_SESSION_BUS_ADDRESS SCENARIO="${1:-e2e}" if [ "$SCENARIO" != "login" ]; then - CHROME_VERSION=$(google-chrome --version) ./test-gui.sh "$SCENARIO" + CHROME_VERSION=$(google-chrome --version) ./test-protractor.sh "$SCENARIO" fi if [ "$SCENARIO" == "e2e" ] || [ "$SCENARIO" == "release" ]; then