This document describes automated tests and how to write an automated test.。
├─.github
│ ├─actions #stores all test scenarios.
│ │ ├─common #common directory for storing test scenarios.
│ │ │ ├─dubbo #dubbo common procedure in test scenarios
│ │ │ ├─entry #common procedure for action entry
│ │ │ ├─exit #common procedure for action exit
│ │ │ ├─plugin-change-check #action for check plugin dir change
│ │ │ └─spring #spring common procedure in test scenarios
│ │ └─scenarios #define specific test scenarios.
│ │ ├─dubbo #dubbo test scenarios
│ │ │ ├─dubbo-common #dubbo common test scenarios
│ │ │ └─router #dubbo router test scenarios
│ │ └─spring #spring test scenarios
│ │ ├─dynamic-config-nacos #spring dynacos config for nacos test scenarios
│ │ ├─dynamic-config-zk #spring dynacos config for zk test scenarios
│ │ ├─graceful #spring graceful online/offline test scenario
│ │ ├─router #spring router test scenarios
│ │ │ ├─spring-router #spring router test scenario
│ │ │ └─spring-tag-router #spring router test scenario for springboot
│ │ └─spring-common #spring common test scenarios
│ ├─ISSUE_TEMPLATE
│ └─workflows #auto test workflow
└─sermant-integration-tests #storing automated tests demo
├─dubbo-test #dubbo automated tests demo
│ ├─dubbo-2-6-integration-consumer
│ ├─dubbo-2-6-integration-controller
│ ├─dubbo-2-6-integration-provider
│ ├─dubbo-2-7-integration-consumer
│ ├─dubbo-2-7-integration-controller
│ ├─dubbo-2-7-integration-provider
│ ├─dubbo-integration-api
│ └─dubbo-integration-test
├─scripts #auto test script
└─spring-test #spring automated tests demo
├─spring-common
├─spring-common-demos #spring common demos
│ ├─spring-common-feign #common feign demo， 2.x
│ ├─spring-common-feign-1.5.x #common feign demo, 1.5.x
│ ├─spring-common-gateway #common gateway for demo-gateway
│ ├─spring-common-resttemplate #common Resttemplate test demo
│ └─spring-common-zuul #common gateway for demo-zuul
├─spring-intergration-test #auto test ut(Junit)
├─spring-nacos-config #nacos dynamic config demo
└─spring-zookeeper-config #zk dynamic config demoThe preceding figure shows the directory structure of the current automatic test, which consists of the following three parts:
- workflow, this is an automatic test entry. You need to add actions to be executed in the workflow (test scenario).
- actions，the specific end-to-end test procedure is defined here.
- sermant-integration-tests， this part stores the test demo required by the action and compiles the junit test cases that require the test logic.
You need to define the automatic test entry in the workflow. Currently, the workflow has test entries based on the Dubbo and Spring frameworks. The entry file is as follows:
dubbo_integration_test.yml, Dubbo test scenario entryspring_integration_test_1,spring_integration_test_2, Spring test scenario entry. The latter is mainly used to test the label routing capability, while the former is used to test the capabilities of other plug-ins, including flow control, registration, graceful login and logout, and load balancing. The reason why the GitHub workflow is split is to make full use of the concurrent test of the GitHub workflow to improve the test efficiency.
Currently, the automatic test is driven based on the version matrix of the test framework and depends on the GitHub action matrix. The version matrix is set at the workflow entry. Based on the specified matrix, all actions (specific test scenarios) configured at the entry are tested to achieve the test purpose.
test-for-spring:
name: Test for spring
runs-on: ubuntu-latest
needs: [build-agent-and-cache, download-midwares-and-cache]
strategy:
matrix:
include:
- springBootVersion: "1.5.0.RELEASE"
springCloudVersion: "Edgware.SR2"
nacosVersion: "1.5.0.RELEASE"
- springBootVersion: "2.0.2.RELEASE"
springCloudVersion: "Finchley.RELEASE"
nacosVersion: "2.0.0.RELEASE"
fail-fast: false
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 100
- name: common operations
uses: ./.github/actions/common/spring
- name: (graceful) test for springboot=${{ matrix.springBootVersion }} springCloudVersion=${{ matrix.springCloudVersion }}
if: env.enableGraceful == 'true'
uses: ./.github/actions/scenarios/spring/graceful
- name: (zk dynamic config) test for springboot=${{ matrix.springBootVersion }} springCloudVersion=${{ matrix.springCloudVersion }}
if: env.enableDynamicConfig == 'true'
uses: ./.github/actions/scenarios/spring/dynamic-config-zkAs shown in the preceding figure, two matrix versions are defined. Take springBootVersion as an example. There are 1.5.0.RELEASE and 2.0.2.RELEASE. During the actual test, the actions(configured graceful and zk dynamic config) of the two versions are tested at the same time (if the concurrency is not limited).The execution does not affect each other.
The action path is specified by the uses configuration file.
The automatic test is specific to a specified scenario. Take graceful in the preceding section as an example. The action.yml file is defined in the ./.github/actions/scenarios/spring/graceful directory. This file simulates the end-to-end test process and simulates the manual graceful login and logout test process. Reproduce it during automated testing.
Common capabilities are provided to simplify test scenario compilation and decouple service test scenarios from common capabilities. The common capabilities are as follows:
- Middleware caching/downloading/starting, agent packaging caching/downloading
- Action public entry processing (only logs currently) and action public exit processing (only processes are terminated and environment variables are cleared currently)
- Code directory change check and judgment
The related component cache is defined at each test entry. The following uses Spring graceful as an example. Two public jobs are defined at the entry:
download-midwares-and-cache:
name: download midwares and cache
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: cache local cse
uses: actions/cache@v3
with:
path: Local-CSE-2.1.3-linux-amd64.zip
key: ${{ runner.os }}-local-cse
restore-keys: |
${{ runner.os }}-local-cse
- name: download cse
run: |
export ROOT_PATH=$(pwd)
bash ./sermant-integration-tests/scripts/tryDownloadMidware.sh cse
- name: cache zookeeper
uses: actions/cache@v3
with:
path: apache-zookeeper-3.6.3-bin.tar.gz
key: ${{ runner.os }}-apache-zookeeper-3.6.3
restore-keys: |
${{ runner.os }}-apache-zookeeper-3.6.3
- name: download zookeeper
run: |
export ROOT_PATH=$(pwd)
bash ./sermant-integration-tests/scripts/tryDownloadMidware.sh zk
- name: cache nacos server
uses: actions/cache@v3
with:
path: nacos-server-1.4.2.tar.gz
key: ${{ runner.os }}-nacos-server-1.4.2
restore-keys: |
${{ runner.os }}-nacos-server-1.4.2
- name: download nacos
run: |
export ROOT_PATH=$(pwd)
bash ./sermant-integration-tests/scripts/tryDownloadMidware.sh nacos
build-agent-and-cache:
name: build agent and cache
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up JDK 8
uses: actions/setup-java@v3
with:
java-version: '8'
distribution: 'adopt'
cache: maven
- name: cache agent
uses: actions/cache@v3
with:
path: sermant-agent-*/
key: ${{ runner.os }}-agent-${{ github.run_id }}
- name: package agent
run: |
sed -i '/sermant-backend/d' pom.xml
sed -i '/sermant-backend-lite/d' pom.xml
sed -i '/sermant-integration-tests/d' pom.xml
sed -i '/sermant-injector/d' pom.xml
mvn package -DskipTests -Ptest --file pom.xmlUse the needs action of GitHub action to ensure that specific action tests are processed after caching.
test-for-spring:
name: Test for spring
runs-on: ubuntu-latest
needs: [build-agent-and-cache, download-midwares-and-cache] #Depending on the front jobDownload and start related components in the unified action. For details, see the file ./.github/actions/common/spring/action.yml
The entry and exit of each action are unified. The steps must be performed before and after each action is processed.
-
Entry, it's path is
./.github/actions/common/entry/action.ymlname: "Entrance operations" description: "do something entrance operations for all test" inputs: log-dir: description: 'Log path for cur workflow' required: true default: './logs/default' runs: using: "composite" steps: - name: create log dir shell: bash run: | log_dir=${{ inputs.log-dir }} mkdir -p ${log_dir} echo "logDir=${log_dir}" >> $GITHUB_ENV echo "=======check log dir======" ls -l ${log_dir}
The log output path is unified to prevent each action from occupying the same file after the demo application is started. The input parameter
log-diris required. The user needs to specify the path. In addition, when the demo needs to output logs, add the log path prefix environment variable (${{env.logDir}}). -
Exit, it's path is
./.github/actions/common/exit/action.ymlname: "Exit operations" description: "do something exit operations for all test" inputs: processor-keyword: description: 'The keyword associate to processor that you want to kill, if you have one more key word, you can separate with |,such as rest|feign' required: false default: '' runs: using: "composite" steps: - name: reset shell: bash run: | echo "tailVersion=" >> $GITHUB_ENV echo "healthApi=" >> $GITHUB_ENV echo "logDir=" >> $GITHUB_ENV keyword='${{ inputs.processor-keyword }}' if [ $keyword != '' ];then jps -l | grep -E "${keyword}" | awk '{print $1}' | xargs -n 1 kill -9 echo "========check process========" jps -l fi
The exit action is used to clear environment variables and terminate processes based on the jps command. Process keywords need to be transferred. Multiple keywords are separated by vertical bars (
|). The keywords are obtained from the name of the JAR package of the demo.
To improve the efficiency and accuracy of the automatic test, determine whether to execute the corresponding scenario based on the code submission path.
For example, the graceful code path is sermant-plugins/sermant-service-registry, . If the submitted code modifies the file in the path, the action test is triggered. If the submitted code is another plug-in, the scenario will not be triggered.
The implementation of this function depends on the open-source action has-changed-path. This component can determine the change of a specified path force based on the submitted code. An independent action is extracted to detect the code change. The path is ./.github/actions/common/plugin-change-check/action.yml.
The following figure shows the graceful path detection process.
name: "Plugin change check"
description: "check file change for all plugins"
runs:
using: "composite"
steps:
# 1. Check whether change files exist in the sermant-plugins/sermant-service-registry directory.
- uses: marceloprado/[email protected]
id: changed-sermant-service-registry
with:
paths: sermant-plugins/sermant-service-registry
- name: env sermant-sermant-service-registry
shell: bash
run: |
echo "sermantServiceRegistryChanged=${{ steps.changed-sermant-service-registry.outputs.changed }}" >> $GITHUB_ENV
# 2. Check whether the pipeline file is changed.
- uses: marceloprado/[email protected]
id: changed-workflow-or-test
with:
paths: ./.github/actions ./.github/workflows sermant-integration-tests
# 3. Check whether the event that triggered is push
- name: check push event
shell: bash
run: |
eventName=${{ github.event_name }}
if [ $eventName == 'push' ];then
echo "triggerPushEvent=true" >> $GITHUB_ENV
else
echo "triggerPushEvent=false" >> $GITHUB_ENV
fi
- name: statistic scenarios change env
shell: bash
run: |
# ==========graceful is needed to test?==========
if [ ${{ env.sermantAgentCoreChanged }} == 'true' -o ${{ env.sermantServiceRegistryChanged }} == 'true' ];then
# 4. If the file of the agentcore or graceful plugin changes, add the environment variable enableGraceful to the environment variable.
echo "enableGraceful=true" >> $GITHUB_ENV
fi
# all workflow will trigger while workflow changed
if [ ${{ steps.changed-workflow-or-test.outputs.changed }} == 'true' ];then
# 5. If the workflow changes, add it to the environment variable.
echo "enableGraceful=true" >> $GITHUB_ENV
fiIn the preceding command, paths indicates the specified change path. After the preceding action is specified, the environment variable env.enableGraceful is stored. The environment variable is used in the workflow, as shown in the following figure:
- name: (graceful) test for springboot=${{ matrix.springBootVersion }} springCloudVersion=${{ matrix.springCloudVersion }}
if: env.enableGraceful == 'true'
uses: ./.github/actions/scenarios/spring/gracefulThe graceful test case is executed only when env.enableGraceful is true.
Notices:
In addition to the preceding path judgment, the entire workflow path judgment takes effect for event pull_request. The configuration is as follows:
on:
push:
pull_request:
branches:
- main
- develop
paths:
- 'sermant-agentcore/**'
- 'sermant-integration-tests/**'
- 'sermant-plugins/sermant-dynamic-config/**'
- 'sermant-plugins/sermant-flowcontrol/**'
- 'sermant-plugins/sermant-loadbalancer/**'
- 'sermant-plugins/sermant-service-registry/**'
- 'sermant-plugins/sermant-springboot-registry/**'
- '.github/workflows/spring_integration*.yaml'
- '.github/actions/**'You can find the set corresponding to the paths configuration. If the workflow takes effect, the submitted code path must be included.
Therefore, if a plugin or directory is added, you must add the corresponding directory in paths.
How to compile an automated test? The steps are as follows:
- Check whether the current middleware meets the requirements. (Currently, only ZooKeeper, Nacos, and LocalCse are supported.)
- (Optional) Add common middleware support.
- Coding test demos and corresponding Junit test cases for specific scenarios.
- Create the test action, simulate the manual test process, and reproduce the problem in the action.
- (Optional) Adding code path detection
- Add actions to the automatic test entry, that is, workflow, based on the framework type and version matrix.
- (Optional) Modify the paths of the working ingress.
- Submit the workflow for testing
(1) To add the cache to the middleware, you need to add the cache steps in the corresponding automatic test entry. The steps are as follows:
download-midwares-and-cache:
name: download midwares and cache
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: cache zookeeper
uses: actions/cache@v3
with:
path: apache-zookeeper-3.6.3-bin.tar.gz
key: ${{ runner.os }}-apache-zookeeper-3.6.3
restore-keys: |
${{ runner.os }}-apache-zookeeper-3.6.3
- name: download zookeeper
run: |
export ROOT_PATH=$(pwd)
bash ./sermant-integration-tests/scripts/tryDownloadMidware.sh zkAn example. Use actions/cache@v3 to cache data, and then perform the download step. When cache-hit is triggered, the cached ZooKeeper is directly used next time.
(2) Add Startup Steps for Mideware
Check whether the middleware is universal. If the middleware is universal, it is recommended that the middleware be included in the common action. Otherwise, the life cycle of the middleware is managed only in the current action.
a) Add the startup step to the common action.
If the application is Dubbo, add the startup steps to./.github/actions/common/dubbo/action.yml
If the application is a Spring application, add the startup step to./.github/actions/common/spring/action.yml
b) Add a startup step to the current action.
In this case, you need to add the middleware startup step at the beginning of the action, and add the middleware termination step at the end of the action.
Find the sermant-integration-tests directory and check whether the current test scenario is applicable to the existing demo application. If yes, add related test interfaces. Otherwise, you need to compile an independent test demo.
Codeing the Junit test cases in the sermant-integration-tests/spring-test/spring-integration-test or sermant-integration-tests/dubbo-test/dubbo-integration-test directory.
Create a directory in the current scenario in ./.github/actions/scenarios. For example, the content of ./.github/actions/scenarios/test/action.yml, is as follows:
name: "{Scenarios} Test"
description: "Auto test for {your scenarios}"
runs:
using: "composite"
steps:
- name: entry
uses: ./.github/actions/common/entry
with:
log-dir: ./logs/test
- name: package common demos
shell: bash
run: |
xxx
- name: start provider that has closed graceful ability
shell: bash
run: |
xxx
- name: start applications that has opened graceful ability
shell: bash
env:
grace.rule.enableSpring: true
run: |
xxx
- name: waiting for services start
shell: bash
run: |
ps -ef | grep java
bash ./sermant-integration-tests/scripts/checkService.sh http://127.0.0.1:8015/graceful/testGraceful 120
- name: test graceful up
shell: bash
run: mvn test -Dsermant.integration.test.type=GRACEFUL -Dgraceful.test.type=up --file sermant-integration-tests/spring-test/pom.xml
- name: exit
if: always()
uses: ./.github/actions/common/exit
with:
processor-keyword: feign|rest
- name: if failure then upload error log
uses: actions/upload-artifact@v3
if: ${{ failure() || cancelled() }}
with:
name: (${{ github.job }})-graceful-(${{ matrix.springBootVersion }}-${{ matrix.springCloudVersion }})-logs
path: |
./*.log
./logs/**
if-no-files-found: warn
retention-days: 2
As shown above, there are the following steps:
- entry
- package demo
- start
- test
- exit
- if failure then upload error log （Upload error logs when errors occur）
Perform the preceding steps. Steps 2, 3, and 4 are used to simulate the end-to-end test.
Referring to Code Directory Change Check. If the plugin type is a new plug-in type, add the corresponding plugin path and trigger the action test using environment variables at the entrance of the automatic test.
For details about the entry, see section entry. Add the current action as follows (for details, see Graceful Online and Offline):
steps:
- name: (graceful) test for springboot=${{ matrix.springBootVersion }} springCloudVersion=${{ matrix.springCloudVersion }}
if: env.enableGraceful == 'true'
uses: ./.github/actions/scenarios/spring/gracefulIf the matrix of the current entry is seriously inconsistent with the test scenario, you can add an entry or add a job based on the workflow of the entry to compile your own version matrix. (For details, see the test-for-spring job task.) Certainly, steps may be reused herein, for example, middleware caching and common processing steps.
on:
push:
pull_request:
branches:
- main
- develop
paths:
- 'sermant-agentcore/**'
- 'sermant-plugins/sermant-dynamic-config/**'
- 'sermant-plugins/sermant-flowcontrol/**'
- 'sermant-plugins/sermant-loadbalancer/**'
- 'sermant-plugins/sermant-service-registry/**'
- 'sermant-plugins/sermant-springboot-registry/**'
- '.github/workflows/spring_integration*.yaml'
- '.github/actions/**'The preceding figure shows the triggering event of the work ingress. The workflow is triggered only when branches and paths meet the requirements at the same time. In this example, paths are used for path detection. The workflow is triggered only when the path is matched with the code in the paths list. Therefore, if the new code is in another directory, you must add the path to paths.
Currently, the automatic test version is unified and defined in the workflow entry, as shown in the following figure:
name: Spring Integration Test1
env:
sermantVersion: 1.0.0The following demo startup commands must use the environment variable ${{env.sermantVersion}} as the agent version path. For example:
-javaagent:sermant-agent-${{ env.sermantVersion }}/agent/sermant-agent.jar=appName=default
Currently, a concurrent group is defined for each working inbound interface. The configuration is as follows:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}-${{ github.head_ref }}
cancel-in-progress: true${{ github.workflow }} indicates the name of the current workflow
${{ github.event.pull_request.number }} indicates the ID of the pull request
${{ github.ref }} indicates the target branch of the pull request; When the event is a push event, the value is the current branch
${{ github.head_ref }} indicates the source branch of the pull request; When the event is a push event, the value is null.
If two tasks have the same group, the former will cancel the workflow.
The triggering of a specific action (specific test scenario) depends on the triggering of a workflow. The triggering of a workflow depends on the push and pull_request events. Therefore, the automatic test workflow mainly depends on the preceding two events. In addition, the workflow triggering and action triggering must be preferred. Only when the workflow is triggered, the action layer can be triggered. The following describes workflow triggering and action triggering based on priorities.
(1) Trigger Workflow
-
pushevent, the commits. This event triggers all workflows by default, regardless of branches and paths. -
pull_request, the PR. This event takes into account the branch name and code path. Take chestnuts, as follows:on: pull_request: branches: - main - develop paths: - 'sermant-agentcore/**' - 'sermant-integration-tests/**' - 'sermant-plugins/sermant-dynamic-config/**'
The preceding trigger conditions are as follows:
-
- The merged branch is the
mainordevelopbranch. - The submitted code path must be in one of the
sermant-agentcore,sermant-integration-tests,sermant-plugins/sermant-dynamic-config.
It will take effect only when both of the preceding conditions are met.
Therefore, if you add a plugin test, you must add a new directory to
paths. - The merged branch is the
（2）Trigger Action（Actions in specific test scenarios and non-public actions)
An action can be triggered only after a workflow is triggered.
The triggering of an action is more about the code path, but a specific path or event will enable full automated testing. The following will describe by using a specific event.
pushevent. This event enables all action tests by default. You can find the action file for path judgment , refer to Code Directory Change Check. If the event ispushevent, all action switches are enabled.pull_requestevent is used to determine the code change path. For details, see Code Directory Change Check. Different test scenarios are determined based on different paths. For example, if thesermant-plugins/sermant-routerfile is changed, the test can be enabled for all associated route scenarios. In addition, the corresponding switch is added to the automatic test entry to control the execution of the automatic test.