Skip to content

Commit 0b3493c

Browse files
evelyn-ysevelyn-ys
authored
{Doc} Add telemetry doc (Azure#22689)
* add telemetry folder * properties' explanation * type * format * format and typo
1 parent 1602622 commit 0b3493c

File tree

3 files changed

+216
-0
lines changed

3 files changed

+216
-0
lines changed

‎doc/telemetry/README.md

+85
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
#Telemetry Documentation
2+
3+
> Two types of telemetry are used to monitor and analyze execution of Azure CLI commands. One called ARM telemetry is recorded basing on HTTP traffic by ARM, and another is client side telemetry sent by Azure CLI.
4+
5+
## ARM Telemetry
6+
7+
ARM telemetry tracks all HTTP requests and responses through ARM endpoint. As far as we know, below cases don't have ARM telemetry record.
8+
- Command doesn't create request successfully, for instance, parameter cannot pass validation, request or payload cannot be constructed.
9+
- Command calls data plane service API.
10+
- Network is inaccessible.
11+
- No request is needed during execution.
12+
13+
Kusto Cluster and Database: https://dataexplorer.azure.com/clusters/armprod/databases/ARMProd
14+
15+
16+
## CLI Client Telemetry
17+
18+
Client side telemetry is sent at the end of Azure CLI command execution. It covers all commands, no matter if it has http requests or just has local operations.
19+
Sanitized data is stored in Kusto cluster which is managed by DevDiv Data team.
20+
21+
Kusto Cluster and Database: https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli
22+
23+
All Azure CLI data is stored in a large json named `Properties` in table `RawEventsAzCli`. Some properties are flatten, some are not. Here are some useful fields:
24+
> The telemetry has different schema pre Azure CLI 2.0.28. All the fields explained below are for new schema, in other words, CLI version > 2.0.28.
25+
26+
- `EventName`: `azurecli/command` or `azurecli/fault` or `azurecli/extension`
27+
- `azurecli/command` means this is common event record with general `Properties` field.
28+
- `azurecli/fault` means this is additional event record with extra exception info in `Properties` field.
29+
- `azurecli/extension` means this is additional event record with customized info in `Properties` field.
30+
- Additional event record can be joined with common event record using `CorrelationId`.
31+
- `CorrelationId`: GUID to join additional event records with common event record
32+
- `azurecli/extension` event records have the same `CorrelationId` with related `azurecli/command` event record.
33+
- `azurecli/fault` event has `Properties['reserved.datamodel.correlation.1']` field. The field value is `{correlationId},UserTask,` where `{correlationId}` is `CorrelationId` field of related `azurecli/command` record.
34+
- `EntityType`: `UserTask`/`Fault`
35+
- For `EventName == 'azurecli/command'`, the `EntityType` is `UserTask`.
36+
- For `EventName == 'azurecli/fault'`, the `EntityType` is `Fault`.
37+
- `EventTimeStamp`: time when the telemetry record is sent
38+
- `ProductVersion`: CLI core version in the format of `azurecli@{version}`
39+
- `CoreVersion`: CLI core version
40+
- `ExeVersion`: `{cli_core_version}@{module_version}`. In the new schema (CLI version > 2.0.28), all module versions are `none`. Hence this field is `{cli_core_version}@none`
41+
- `OsType`: OS system, eg. linux, windows
42+
- `OsVersion`: OS platform version, eg. 10.0.14942
43+
- `PythonVersion`: platform python version
44+
- `ShellType`: cmd/bash/ksh/zsh/cloud-shell/... Note: may not be accurate.
45+
- `MacAddressHash`: SHA256 hashed MAC address
46+
- `MachineId`: GUID coming from the first 128bit of MacAddressHash
47+
- `UserId`: CLI installation id. Each CLI client installed locally will have a GUID as installation id. It's stored in $homeFolder/.azure/azureProfile.json
48+
- `SessionId`: SHA256 hashed result of CLI installation id, parent process (terminal session) creation time and parent process (terminal session) id. Note: may not be accurate.
49+
- `RawCommand`: CLI command name
50+
- `Params`: CLI command arguments (without argument value)
51+
- `AzureSubscriptionId`: current subscription id
52+
- `ClientRequestId`: GUID which is set on HTTP header
53+
- `StartTime`: time when the command begins executing
54+
- `EndTime`: time when the command exits
55+
- `ActionResult`:
56+
- For `EntityType == 'UserTask'`, it could be `Success`/`Failure`/`UserFault`/`None`. All others besides `Success` means failure.
57+
- For `EntityType == 'Fault'`, it's empty.
58+
- `ResultSummary`: details of result, may be suppressed to meet security & privacy requirements.
59+
- `ExceptionMessage`: details of exception, may be suppressed to meet security & privacy requirements.
60+
- `Properties`: large json with all constructed fields. Below is to explain some unflattened fields not introduced before.
61+
- `reserved.datamodel.entityname`: CLI command name with hyphens
62+
- `reserved.datamodel.correlation.1`: Additional field when `EventName == 'azurecli/fault'`. It's in the format of `{correlationId},UserTask,` where `{correlationId}` is `CorrelationId` field of related `EventName == 'azurecli/command'` record.
63+
- `reserved.datamodel.fault.typestring`: Additional field when `EventName == 'azurecli/fault'`. It logs the exception class.
64+
- `reserved.datamodel.fault.description`: Additional field when `EventName == 'azurecli/fault'`. It logs exception description or fault type.
65+
- `context.default.vs.core.os.platform`: OS platform
66+
- `context.default.azurecli.source`: `az`/`completer`. It's `completer` if we found argument auto complete settings in os environment variable.
67+
- `context.default.azurecli.environmentvariables`: It logs customer's environment variables starting with `AZURE_CLI`
68+
- `context.default.azurecli.extensionname`: It logs the extension name and version in the format of `{extension_name}@{extension_version}` if the command is from CLI extension.
69+
- `context.default.azurecli.installer`: value of os environment variable `AZ_INSTALLER`
70+
- `context.default.azurecli.error_type`: It logs the exception class name.
71+
- `context.default.azurecli.exception_name`: A supplementation for `context.default.azurecli.error_type`
72+
73+
## Accessing Client Telemetry
74+
75+
Kusto Cluster and Database: https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli
76+
77+
To ensure you have a smooth experience using our Data Tools and Data, you have to take the required trainings and join a security group.
78+
Please follow instruction [Accessing DevDiv Data](https://devdiv.visualstudio.com/DevDiv/_wiki/wikis/DevDiv.wiki/9768/Accessing-DevDiv-Data) to get access permission.
79+
80+
81+
## Doc Sections
82+
83+
- [Kusto Examples](kusto_examples.md) - Samples for kusto query
84+
85+
- [FAQ](faq.md) - Commonly asked questions

‎doc/telemetry/faq.md

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#FAQ
2+
3+
## What's the relationship of CLI telemetry and ARM telemetry?
4+
5+
- CLI telemetry is client telemetry. It logs os, platform, command, parameter, result and other client info.
6+
- ARM telemetry is server telemetry. It tracks all HTTP requests and responses through ARM endpoint from different clients, including CLI, Powershell, SDK...
7+
- They share the same `clientRequestId` which you can leverage to join `HttpIncomingRequests` (ARM telemetry table) with `RawEventsAzCli` (CLI telemetry table)
8+
9+
10+
## How can I filter CLI requests from ARM telemetry?
11+
12+
[Execute in Web](https://dataexplorer.azure.com/clusters/armprod/databases/ARMProd?query=H4sIAAAAAAAAA/MoKSnwzEvOz83MSw9KLSxNLS4p5qpRKM9ILUpVCPH0dQ0OcfQNULBTSEzP1zDM0ITLlRanFjmmp+aVKCTn55UkZuYVK6g7VpUWpTr7eOob6akDFZYkZqcqGBoAAPoAVdxjAAAA)
13+
```
14+
HttpIncomingRequests
15+
| where TIMESTAMP > ago(1h)
16+
| where userAgent contains 'AzureCLI/2.'
17+
| take 10
18+
```
19+
20+
## How can I collect customized properties into CLI telemetry?
21+
22+
You can utilize `add_extension_event` [function](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli-core/azure/cli/core/telemetry.py#L418-L420) to collect properties for your extension.
23+
24+
When customers run command, in additional to general CLI record whose `EventName` is `azurecli/command`, there will be another record whose `EventName` is `azurecli/extension` recorded in CLI telemetry.
25+
26+
And you can join the general `azurecli/command` record with `azurecli/extension` record on `CorrelationId` field.
27+
28+
29+
## How can customer disable telemetry collection?
30+
31+
We have an switch for customer to disable/enable telemetry collection. Try `az config set core.collect_telemetry=true` for enabling and `az config set core.collect_telemetry=false` for disabling.

‎doc/telemetry/kusto_examples.md

+100
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
#Samples for kusto query
2+
3+
## Query for new schema
4+
CLI telemetry has different schema after version 2.0.28
5+
6+
[Execute in Web](https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli?query=H4sIAAAAAAAAAwtKLHctS80rKXascs7J5KpRKM9ILUpVKC7IySzRCCjKTylNLglLLSrOzM/TUVDXU9eMNohVsLVVUE+sKi1KTc7JdDBSh+sqyc/MK9HAo9cwVlPBTsFAIb+IsFojsFojC6DpJYnZqQqmAIl8AharAAAA)
7+
```
8+
RawEventsAzCli
9+
| where split(ProductVersion, '.')[0] == 'azurecli@2'
10+
| where toint(split(ProductVersion, '.')[1]) > 0 or toint(split(ProductVersion, '.')[2]) > 28
11+
| take 5
12+
```
13+
14+
## Query for specific command
15+
e.g. `az account show` command
16+
17+
[Execute in Web](https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli?query=H4sIAAAAAAAAAwtKLHctS80rKXascs7J5KpRKM9ILUpVAIuFZOamFpck5hYo2CkkpudrGGZowhUEJZY75+fmJualKNjaKqgnJifnl+aVKBRn5JerAxWVJGanKhgaAABH9KmYXgAAAA==)
18+
```
19+
RawEventsAzCli
20+
| where EventTimestamp > ago(1h)
21+
| where RawCommand == 'account show'
22+
| take 10
23+
```
24+
25+
## Query for specific command group
26+
e.g. `az storage account` command group
27+
28+
[Execute in Web](https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli?query=H4sIAAAAAAAAAz3KQQqAIBAAwHuvWDzVLT8QhPQB6QOLLSqlhm4J0eOLDl2H0VinkyKX8VKbb26ojjLBZ7MPVBjDDgOgTa103R80VpVCwLjAWzKX6tmBKJwyWgI0Jh2RxfsZVwLZP6LV9udpAAAA)
29+
```
30+
RawEventsAzCli
31+
| where EventTimestamp > ago(1h)
32+
| where RawCommand startswith "storage account"
33+
| take 10
34+
```
35+
36+
## Query for specific command with specific cli version
37+
e.g. `az account show` command with CLI version `2.35.0`
38+
39+
[Execute in Web](https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli?query=H4sIAAAAAAAAAz3KsQqDMBAA0L1fcZt1Ea04Wlqku0hxP+LRhJpcSS4GpB9ftOD6eAOmx0JOwn3tZnP6QtLkCXZ7GktB0H7gCvjic6XzIwyYOrYW3QRtCxkqxdEJBM0pO1LveYpKRvLBsPvHNXpSs7ldiropyu0Kvgmq8gc6Rz0AigAAAA==)
40+
```
41+
RawEventsAzCli
42+
| where EventTimestamp > ago(1h)
43+
| where RawCommand == 'account show'
44+
| where ProductVersion == '[email protected]'
45+
| take 10
46+
```
47+
48+
## Query for specific command with specific cli extension version
49+
e.g. `az connectedk8s connect` command with version `1.2.8` of extension `connectedk8s`
50+
51+
[Execute in Web](https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli?query=H4sIAAAAAAAAA12OvQ6CQBCEe59iQwM0F7GiwWgIrTHEzlhsYMUL3B25W8QQH97D+BOtJjv5ZmdKHIsraXbbKe/k4g7jhSzB0ztIRY5R9bAGbEyUXGIP0I1J11DM4qTRO1QEGbBxbKVuor01PVmW5I5BZTR7XtR0xqFjgdNgqeqkoHda+3Rwij/FJY65UQp9Q5ZB6B9oqpjqNnXwOsLvyt8Nf/wmESuRzjRjS5AsH7YF5vDsAAAA)
52+
```
53+
RawEventsAzCli
54+
| where EventTimestamp > ago(1h)
55+
| extend ExtensionName = tostring(Properties["context.default.azurecli.extensionname"])
56+
| where RawCommand == 'connectedk8s connect'
57+
| where ExtensionName == '[email protected]'
58+
| take 10
59+
```
60+
61+
## Count specific command calls by date
62+
e.g. `az account show` usage by date
63+
64+
[Execute in Web](https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli?query=H4sIAAAAAAAAA1WMwQrCMBAF737FuzUFL548RZDiDxR/YJssNmASyW4NFj/eEkHwOjPMSPXy5KRyXod72L1RZy6Mxq4hsijFB06gWzZH3/+CkeqQY6TkYS06ci4vSSFzrt0WybK5ElaGS2qbMz2mF1TsFJL5/+9xaOtcPJdvBc/iPgod8yqdAAAA)
65+
```
66+
RawEventsAzCli
67+
| where EventTimestamp > ago(7d)
68+
| where RawCommand == 'account show'
69+
| summarize cnt=count() by ts=bin(EventTimestamp, 1d)
70+
| order by ts desc
71+
```
72+
73+
## Calculate success rate for specific command
74+
e.g. `az group create` command
75+
76+
[Execute in Web](https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli?query=H4sIAAAAAAAAA1XMwQrCMBAE0Ltfsbe2J4+eKpTi1UPwB9Z0qYEkWzYbg8GPb60geBuGN2OwXJ4UNQ119O7whvIgIdi7mwuUFMMCZ8CZ29PU/YDBMnIIGCfoe2hm4byAFUKl5v/lioF2gzULWe+O9jv8uJS3KK4SWM5R2w7uLxisOo6GUva6AvX30C+gAAAA)
77+
```
78+
RawEventsAzCli
79+
| where EventTimestamp > ago(7d)
80+
| where RawCommand == 'group create'
81+
| where EventName == 'azurecli/command'
82+
| summarize count() by ActionResult
83+
```
84+
Note: `where EventName == 'azurecli/command'` is necessary because in some cases one record will have additional records whose `EventName` could be `azurecli/extension` or `azurecli/fault`. If you count these additional records, some calls might be calculated twice or more times.
85+
86+
## Query failure details for specific command
87+
e.g. `az group create` command
88+
89+
[Execute in Web](https://dataexplorer.azure.com/clusters/ddazureclients/databases/AzureCli?query=H4sIAAAAAAAAA52Ry2rDMBBF9/0KdeUEhKEf4EJI3V1LcLIrJQzS1FGxHozGcRL68ZVtGseUbroV554Z3amgK4/oOK4u68bcfYnugIRieNsZi5HBBvEooPaLB728AhV0a28tOC2KQmQ1+TYIRQiM2RVaKTbeVRjbhsV9wratUhhjT+CJMYVFSeRpdw4oCsE+MhlXLzbkAxIbjG+Z8o4TnGv8gOTJ4dISqsbk2Cf3nKLZ+3IylieFoZ/7CvYf0p/03qX4TPzck08YFZmB+MtNGJGOqHMNqTyvscnHIXrKjuJA/hMVj23368qbXqXYAIGNclajnAqT86/KXwsm25DZtklI5xv+JR0B6hSZ9v4GVmNP5AkCAAA=)
90+
```
91+
RawEventsAzCli
92+
| where EventTimestamp > ago(1d)
93+
| where RawCommand == 'group create'
94+
| where ActionResult != 'Success'
95+
| extend ErrorType = tostring(Properties['context.default.azurecli.error_type'])
96+
| extend ExceptionName = tostring(Properties['context.default.azurecli.exception_name'])
97+
| extend FaultDescription = tostring(Properties['reserved.datamodel.fault.description'])
98+
| project EventName, RawCommand, Params, ActionResult, ErrorType, ExceptionName, FaultDescription, ResultSummary, ExceptionMessage, Properties
99+
```
100+
Notes: `ResultSummary` and `ExceptionMessage` might be suppressed to meet security & privacy requirements.

0 commit comments

Comments
 (0)