Connectivity tests

Test Scenarios

These are test scenarios to be covered in Connectivity test

1. Messaging Test
   1. Module-to-Module
   2. Module-to-IoT Hub
2. Twin Property Update Test
   1. Desired property
   2. Reported property
3. Direct Method Test
   1. Cloud-to-module
   2. Ping method in EdgeAgent
4. EdgeAgent Deployment Test
   1. Deployment change

General flow

1. Currently Connectivity test is supported only to run on Linux AMD64.
2. Connectivity test can be run by calling Connectivity Test script.
3. Each test runs using AMQP and MQTT TCP protocol.
4. For each run, a new tracking id (Guid) is generated by the test script.
5. Test starts after pre-defined warm-up period (default 2 minutes), which assumes EdgeHub and other modules containers are up and running.
6. A test duration is configurable and is set to 1 hour by default.
7. A test module is responsible for providing the Test Result Coordinator with a result of its operation.
8. An operation result is persistently stored by the Test Result Coordinator after validating result type is supported.
9. The Test Result Coordinator consumes events from IoT hub's built-in event hub endpoint for messaging scenario verification.
10. The Network Controller simulates a different test network condition, e.g. offline or packet loss. The Network Controller, like any other test modules, is responsible to report a result of its action to The Test Result Coordinator.
11. After the test duration is over, the test waits for a brief period of time to ensure that all of the actions propagated through the system. Then, the final test report is generated and uploaded to Log Analytics. The buffer is 15 minutes by default, which assumes all messages are received on IoT hub and are pulled for verification.
12. After test duration, test modules should stop calling operations, e.g. sending messages, making Direct method calls, etc.

Offline Scenario

1. The Network Controller controls the virtual network "Azure-IoT-Edge" to simulate offline and packet-loss scenarios.
2. When generating test result reports, network status is taken into consideration based on result reported by Network controller.

Potential Issues

1. If there are many tests running at the same time using devices in the same IoT hub, there may be too many event messages received by Test Result Coordinator; which slows down test result verification and may cause failed result.
   1. Try to reduce number of test devices in a test IoT hub.
   2. Introduce Azure Stream Analytics (ASA) to process messages, which should provide a higher throughput.
2. Network Controller implementation may support on Linux only, but not Windows platform.

Tests

Messaging Tests

1. loadGen starts sending messages after pre-defined warm-up period.
2. loadGen sends module-to-module messages to Relayer through Edge hub (#1 -> Edge hub -> #2).
3. After sending each message, loadGen reports result to Test Result Coordinator (#3).
4. Relayer receives the message and reports result to Test Result Coordinator (#2 -> #4).
5. After receiving in Relayer, messages send forward to IoT hub through Edge hub (#5 -> Edge hub -> #7).
6. After forwading each message in relayer, it reports result to Test Result Coordinator (#6).
7. All messages should contain tracking id, batch id and sequence number. Sequence number should always start at 1 in loadGen.

Network offline/online should not affect this test result since Edge hub stores all messages when offline and resumes back once get back online.