Help with HiveMQ cloud export MQTT using docker compose file #283

Devendra-7 · 2024-06-07T04:08:59Z

Devendra-7
Jun 7, 2024

Hello All,

Method 1: mqtt-export using the docker

I have followed the getting started Edgex with docker(https://docs.edgexfoundry.org/3.1/getting-started/quick-start/) for starting the edgex services to export the data to the HiveMQ broker by using mqtt-export method.
Edgex services are started without security, but still getting the error related to the secret file creation.

Method 2: mqtt-export wihout docker(in Developer mode)
Downloaded the Edgex services, services are built and started without security, services are starting fine without any issue but still not getting any data from the mqtt-export method.
Referred the link below:
https://docs.edgexfoundry.org/3.1/getting-started/native/Ch-BuildRunOnLinuxDistro/
Didn't see any prints related to function pipe line is executing like.

Configuration file for mqtt-export:

Writable:
  LogLevel: "INFO"
  StoreAndForward:
    Enabled: false

  Pipeline:
    ExecutionOrder: "FilterByDeviceName, Transform, MQTTExport"

    Functions:
      FilterByDeviceName:
        Parameters:
          DeviceNames: "Random-Integer-Device"
          FilterOut: "false"
      Transform:
        Parameters:
          Type: "json"
      Compress:
        Parameters:
          Algorithm: "gzip" # can be "gzip" or "zlib"
      Encrypt:
        Parameters:
          Algorithm: "aes256" 
          SecretName: "aes"
          SecretValueKey: "key"
      Batch:
        Parameters:
          Mode: "bytimecount" # can be "bycount", "bytime" or "bytimecount"
          BatchThreshold: "30"
          TimeInterval: "60s"
          MergeOnSend: "false"
      AddTags:
        Parameters:
          Tags: ""
      MQTTExport:
        Parameters:
          BrokerAddress: "192.168.1.238:1883"
          #BrokerAddress:"tcp://broker.mqttdashboard.com:1883"  # this is also didn't work
          # Topic can have place holders in the form "{key-name}" that are replaced with matching Context values. Error occurs if no value match place holder(s).
          Topic: "mytopic"
          SecretName: "mqtt"
          ClientId: "myclientid"
          QOS: "0"
          AutoReconnect: "false"
          KeepAlive: "" # Empty indicates to use default value
          ConnectTimeout: "" # Empty indicates to use default value
          Retain: "false"
          SkipVerify: "false"
          AuthMode: "none"
          PersistOnError: "false"
          WillEnabled: "false"
          WillPayload: ""
          WillQos: "0"
          WillRetained: "false"
          WillTopic: ""

  InsecureSecrets:
    mqtt:
      SecretName: "mqtt"
      SecretData:
        username: ""
        password: ""
        cacert: ""
        clientcert: ""
        clientkey: ""
    aes:
      SecretName: "aes"
      SecretData:
        key: "217A24432646294A404E635266556A586E3272357538782F413F442A472D4B6150645367566B59703373367639792442264529482B4D6251655468576D5A7134"

Service:
  Host: "localhost"
  Port: 59703
  StartupMsg: "app-mqtt-export has Started"

MessageBus:
  Disabled: false # Set to true if not using edgex-messagebus Trigger below and Metrics are not enabled
  Optional:
    ClientId: "app-mqtt-export"

# Using default Trigger config from common config

I have configured the pipeline execution order: ExecutionOrder: "FilterByDeviceName, Transform, MQTTExport". Just want to confirm once app service and device-virtual are started function pipeline will be triggered or we need to generate the trigger manual by generating the Random integer number with help POST commands ?

Is my configuration need to be changed ?

Thanks in Advance...!

cloudxxx8 · 2024-06-10T08:20:47Z

cloudxxx8
Jun 10, 2024
Maintainer

you need to setup the know secret for your app service
https://docs.edgexfoundry.org/3.1/security/Ch-Configuring-Add-On-Services/#optional-configure-known-secrets-for-add-on-services

also, set up the proper volume mapping for the folder to store the secret store token
https://github.com/edgexfoundry/edgex-compose/blob/8445870debbb756528338dc9018d3dca8d43cf8d/docker-compose.yml#L98

4 replies

Devendra-7 Jun 10, 2024
Author

I am running the Edgex services in Non-Secure mode(https://raw.githubusercontent.com/edgexfoundry/edgex-compose/v3.1/docker-compose-no-secty.yml ). In this case also it's required to configure the secret store keys for the services ?.
I have configured the pipeline execution order: ExecutionOrder: "FilterByDeviceName, Transform, MQTTExport". Just want to confirm once app service and device-virtual are started function pipeline will be triggered automatically or we need to generate the trigger manual by generating the Random integer number with help POST commands ?

cloudxxx8 Jun 11, 2024
Maintainer

no, non-security mode doesn't require secret store.
the default app service trigger is the edgex event message bus, so it will be triggered by device-virtual events automatically.

You should set the service key for app-service. It may also help your security mode.
https://docs.edgexfoundry.org/3.1/microservices/application/details/EnvironmentVariables/#edgex_service_key

Devendra-7 Jun 11, 2024
Author

My final config look like this for the cloud export mqtt, i am using mqtt web client as a broker. I have selected the random numbers from the device-virtual service as a trigger for the pipeline.

app-service(mqtt-export) private configuration:

Writable:
  LogLevel: "INFO"
  StoreAndForward:
    Enabled: false

  Pipeline:
    ExecutionOrder: "FilterByDeviceName, Transform, MQTTExport"

    Functions:
      FilterByDeviceName:
        Parameters:
          DeviceNames: "Random-Integer-Device"
          FilterOut: "false"
      Transform:
        Parameters:
          Type: "json"
      Compress:
        Parameters:
          Algorithm: "gzip" # can be "gzip" or "zlib"
      Encrypt:
        Parameters:
          Algorithm: "aes256" 
          SecretName: "aes"
          SecretValueKey: "key"
      Batch:
        Parameters:
          Mode: "bytimecount" # can be "bycount", "bytime" or "bytimecount"
          BatchThreshold: "30"
          TimeInterval: "60s"
          MergeOnSend: "false"
      AddTags:
        Parameters:
          Tags: ""
      MQTTExport:
        Parameters:
          BrokerAddress: "192.168.1.238:1883"
          #BrokerAddress:"tcp://broker.mqttdashboard.com:1883"  # this is also didn't work
          # Topic can have place holders in the form "{key-name}" that are replaced with matching Context values. Error occurs if no value match place holder(s).
          Topic: "mytopic"
          SecretName: "mqtt"
          ClientId: "myclientid"
          QOS: "0"
          AutoReconnect: "false"
          KeepAlive: "" # Empty indicates to use default value
          ConnectTimeout: "" # Empty indicates to use default value
          Retain: "false"
          SkipVerify: "false"
          AuthMode: "none"
          PersistOnError: "false"
          WillEnabled: "false"
          WillPayload: ""
          WillQos: "0"
          WillRetained: "false"
          WillTopic: ""

  InsecureSecrets:
    mqtt:
      SecretName: "mqtt"
      SecretData:
        username: "user-dev"
        password: "1234567"
        cacert: ""
        clientcert: ""
        clientkey: ""

Service:
  Host: "localhost"
  Port: 59703
  StartupMsg: "app-mqtt-export has Started"

MessageBus:
  Disabled: false # Set to true if not using edgex-messagebus Trigger below and Metrics are not enabled
  Optional:
    ClientId: "app-mqtt-export"

# Using default Trigger config from common config

Devendra-7 Jun 11, 2024
Author

The final common config look like this. with this config i am not able to validate the cloud mqtt export.

all-services:
  Writable:
    InsecureSecrets:
      DB:
        SecretName: "redisdb"
        SecretData:
          username: ""
          password: ""
    
    Telemetry:
      Interval: "30s"
      Metrics:
        # Common Security Service Metrics
        SecuritySecretsRequested: false
        SecuritySecretsStored: false
        SecurityConsulTokensRequested: false
        SecurityConsulTokenDuration: false
        SecurityRuntimeSecretTokenDuration: false
        SecurityGetSecretDuration: false
#     Tags: # Contains the service level tags to be attached to all the service's metrics
      #  Gateway: "my-iot-gateway" # Tag must be added here or via Consul Env Override can only change existing value, not added new ones.
      
  Service:
    HealthCheckInterval: "10s"
    ServerBindAddr: "" # Leave blank so default to Host value unless different value is needed.
    MaxResultCount: 1024
    MaxRequestSize: 0 # Not currently used. Defines the maximum size of http request body in bytes
    RequestTimeout: "5s"
    EnableNameFieldEscape: false # The name field escape could allow the system to use special or Chinese characters in the different name fields, including device, profile, and so on.  If the EnableNameFieldEscape is false, some special characters might cause system error.  TODO: This is set to false by default to avoid breaking change and will be removed in EdgeX 4.0
    CORSConfiguration:
      EnableCORS: false
      CORSAllowCredentials: false
      CORSAllowedOrigin: "https://localhost"
      CORSAllowedMethods: "GET, POST, PUT, PATCH, DELETE"
      CORSAllowedHeaders: "Authorization, Accept, Accept-Language, Content-Language, Content-Type, X-Correlation-ID"
      CORSExposeHeaders: "Cache-Control, Content-Language, Content-Length, Content-Type, Expires, Last-Modified, Pragma, X-Correlation-ID"
      CORSMaxAge: 3600

  Registry:
    Host: "localhost"
    Port: 8500
    Type: "consul"
  
  Database:
    Host: "localhost"
    Port: 6379
    Timeout: "5s"
    Type: "redisdb"
  
  MessageBus:
    Protocol: "redis"
    Host: "localhost"
    Port: 6379
    Type: "redis"
    AuthMode: "usernamepassword"  # required for redis MessageBus (secure or insecure).
    SecretName: "redisdb"
    BaseTopicPrefix: "edgex" # prepended to all topics as "edgex/<additional topic levels>
    Optional:
      # Default MQTT Specific options that need to be here to enable environment variable overrides of them
      Qos:  "0" # Quality of Service values are 0 (At most once), 1 (At least once) or 2 (Exactly once)
      KeepAlive: "10" # Seconds (must be 2 or greater)
      Retained: "false"
      AutoReconnect: "true"
      ConnectTimeout: "5" # Seconds
      SkipCertVerify: "false"
      # Additional Default NATS Specific options that need to be here to enable environment variable overrides of them
      Format: "nats"
      RetryOnFailedConnect: "true"
      QueueGroup: ""
      Durable: ""
      AutoProvision: "true"
      Deliver: "new"
      DefaultPubRetryAttempts: "2"
      Subject: "edgex/#" # Required for NATS JetStream only for stream auto-provisioning

app-services:
  Writable:
    StoreAndForward:
      Enabled: false
      RetryInterval: "5m"
      MaxRetryCount: 10
    Telemetry:
      Metrics:
        MessagesReceived: false
        InvalidMessagesReceived: false
        PipelineMessagesProcessed: false # Pipeline IDs are added as the tag for this metric for each pipeline defined
        PipelineMessageProcessingTime: false # Pipeline IDs are added as the tag for this metric for each pipeline defined
        PipelineProcessingErrors: false # Pipeline IDs are added as the tag for this metric for each pipeline defined
        HttpExportSize: false #  Url is added as tag for this metric for each HTTP export defined
        HttpExportErrors: false # Url is added as tag for this metric for each HTTP export defined
        MqttExportSize: false # BrokerAddress and Topic are added as the tag for this metric for each MQTT export defined
        MqttExportErrors: false # BrokerAddress and Topic are added as the tag for this metric for each MQTT export defined
  Clients:
    core-metadata:
      Protocol: "http"
      Host: "localhost"
      Port: 59881
  Trigger:
    Type: "edgex-messagebus"
    SubscribeTopics: "events/device-virtual/Random-Integer-Device/Random-Integer-Device/#" # Base topic is prepended to this topic when using edgex-messagebus

device-services:
  MaxEventSize: 0 # value 0 represents unlimited  maximum event size that can be sent to message bus or core-data
  Writable:
    Reading:
      ReadingUnits: true
    Telemetry:
      Metrics:
        EventsSent: false
        ReadingsSent: false
        LastConnected: false
  Clients:
    core-metadata:
      Protocol: "http"
      Host: "localhost"
      Port: 59881
  Device:
    DataTransform: true
    MaxCmdOps: 128
    MaxCmdValueLen: 256
    ProfilesDir: "./res/profiles"
    DevicesDir: "./res/devices"
    # ProvisionWatchersDir is omitted here since most Device Services don't use it.
    # Those that do will have it in their private config
    EnableAsyncReadings: true
    AsyncBufferSize: 16
    Labels: []
    Discovery:
      Enabled: false
      Interval: "30s"

cloudxxx8 · 2024-06-11T03:31:21Z

cloudxxx8
Jun 11, 2024
Maintainer

please provide the error log and the steps you set up your environment
as I mentioned, there are some environment variable or flag should be set up, not just config

3 replies

Devendra-7 Jun 11, 2024
Author

Basically i want to replicate the this
(https://docs.edgexfoundry.org/3.1/getting-started/quick-start/).

Downloaded the docker compose file from the below link.

started the containers in non secure mode using docker compose up -d command
After starting the containers tried to get the random numbers from device-virtual service getting the error msg.

     devendra@devendra-Precision-3581:~/EdgeX$ curl http://localhost:59880/api/v3/event/device/name/Random-Integer-Device
curl: (56) Recv failure: Connection reset by peer
devendra@devendra-Precision-3581:~/EdgeX$ curl -X GET localhost:59882/api/v3/device/name/Random-Integer-Device/Int8
curl: (56) Recv failure: Connection reset by peer

To debug the issue further started container manually, set the environment to non secure mode.

5. start the docker containers in non-secure mode

6. After staring containers manually still getting error msg related to the secret store

  level=INFO ts=2024-06-11T04:30:37.362615836Z app=core-metadata source=secret.go:69 msg="Creating SecretClient"
level=INFO ts=2024-06-11T04:30:37.362942675Z app=core-metadata source=secret.go:172 msg="SecretStore information created with 0 overrides applied"
level=INFO ts=2024-06-11T04:30:37.362958556Z app=core-metadata source=secret.go:79 msg="Reading secret store configuration and authentication token"
level=INFO ts=2024-06-11T04:30:37.362995623Z app=core-metadata source=secret.go:216 msg="load token from file"
level=WARN ts=2024-06-11T04:30:37.363056423Z app=core-metadata source=secret.go:132 msg="Retryable failure while creating SecretClient: open /tmp/edgex/secrets/core-metadata/secrets-token.json: no such file or directory"
level=INFO ts=2024-06-11T04:30:38.36323514Z app=core-metadata source=secret.go:79 msg="Reading secret store configuration and authentication token"
level=INFO ts=2024-06-11T04:30:38.363371802Z app=core-metadata source=secret.go:216 msg="load token from file"
level=WARN ts=2024-06-11T04:30:38.363410006Z app=core-metadata source=secret.go:132 msg="Retryable failure while creating SecretClient: open /tmp/edgex/secrets/core-metadata/secrets-token.json: no such file or directory"
level=INFO ts=2024-06-11T04:30:39.363734189Z app=core-metadata source=secret.go:79 msg="Reading secret store configuration and authentication token"
level=INFO ts=2024-06-11T04:30:39.363926062Z app=core-metadata source=secret.go:216 msg="load token from file"
level=WARN ts=2024-06-11T04:30:39.364000911Z app=core-metadata source=secret.go:132 msg="Retryable failure while creating SecretClient: open /tmp/edgex/secrets/core-metadata/secrets-token.json: no such file or directory"

cloudxxx8 Jun 11, 2024
Maintainer

to run non-security mode, please use https://github.com/edgexfoundry/edgex-compose/blob/f316e07dd5b9873035b9e330a1aaaa9f6df5aa6b/docker-compose-no-secty.yml

Devendra-7 Jun 11, 2024
Author

yes, i am using same yml file.
Referred below user guide :(https://docs.edgexfoundry.org/3.1/getting-started/quick-start/) in linux environment i.e. ubuntu22.04

Please confirm the steps mentioned in (https://docs.edgexfoundry.org/3.1/getting-started/quick-start/)) is sufficient to replicate the cloud export mqtt mechanism or any extra changes are required.

Devendra-7 · 2024-06-17T08:40:44Z

Devendra-7
Jun 17, 2024
Author

consul is not starting due to that, getting some unexpected errors, after resolving the consul issues. Able to get the export data

0 replies

cloudxxx8 · 2024-06-25T12:20:57Z

cloudxxx8
Jun 25, 2024
Maintainer

The new HiveMQ testing broker address is broker.hivemq.com
https://www.hivemq.com/mqtt/public-mqtt-broker/

0 replies

Erickrk · 2024-06-27T12:07:20Z

Erickrk
Jun 27, 2024

Configuration file for mqtt-export:
...

Hey @Devendra-7 I'm going through a similar issue, but currently my Consul is up and running. I wonder where you've added this "Configuration file for mqtt-export" since I've only seen editions to docker-compose in the tutorial.
Thank you in advance for your help!

5 replies

Erickrk Jun 27, 2024

I also wonder where to add username and password in case I want to auth with a broker.

Erickrk Jun 30, 2024

Just found that the files were the App Service Configurable and we could try to add username and password through the docker-compose.yml with something like the "WRITABLE_PIPELINE_FUNCTIONS_MQTTEXPORT_PARAMETERS_BROKERADDRESS" env variable but couldn't find docs on that. I guess I'll just move to Developer mode.

Devendra-7 Jul 1, 2024
Author

I also wonder where to add username and password in case I want to auth with a broker.

Hi @Erickrk ,

Pls refer the below configuration file for mqtt-export, which is present in //res/mqtt-export/configuratiom.yaml

Writable:
  LogLevel: "INFO"
  StoreAndForward:
    Enabled: false

  Pipeline:
    ExecutionOrder: "FilterByDeviceName, Transform, MQTTExport"

    Functions:
      FilterByDeviceName:
        Parameters:
          DeviceNames: "Random-Integer-Device"
          FilterOut: "false"
      Transform:
        Parameters:
          Type: "json"
      Batch:
        Parameters:
          Mode: "bytimecount" # can be "bycount", "bytime" or "bytimecount"
          BatchThreshold: "30"
          TimeInterval: "60s"
          MergeOnSend: "false"
      AddTags:
        Parameters:
          Tags: ""
      MQTTExport:
        Parameters:
          BrokerAddress: "tcp://10.217.110.172:1883"
            #BrokerAddress: "tcp://broker.mqttdashboard.com:1883"
          # Topic can have place holders in the form "{key-name}" that are replaced with matching Context values. Error occurs if no value match place holder(s).
          Topic: "edgex-events"
          SecretName: "mqtt"
          ClientId: "myclientid"
          QOS: "0"
          AutoReconnect: "false"
          KeepAlive: "" # Empty indicates to use default value
          ConnectTimeout: "" # Empty indicates to use default value
          Retain: "false"
          SkipVerify: "false"
          AuthMode: "none"
          PersistOnError: "false"
          WillEnabled: "false"
          WillPayload: ""
          WillQos: "0"
          WillRetained: "false"
          WillTopic: ""

  InsecureSecrets:
    mqtt:
      SecretName: "mqtt"
      SecretData:
        **username: "jagan"
        password: "app@12345"**
        cacert: ""
        clientcert: ""
        clientkey: ""

Service:
  Host: "localhost"
  Port: 59703
  StartupMsg: "app-mqtt-export has Started"

MessageBus:
  Disabled: false # Set to true if not using edgex-messagebus Trigger below and Metrics are not enabled
  Optional:
    ClientId: "app-mqtt-export"

# Using default Trigger config from common config

Devendra-7 Jul 1, 2024
Author

WRITABLE_PIPELINE_FUNCTIONS_MQTTEXPORT_PARAMETERS_BROKERADDRESS

Hi @Erickrk ,
Macros are not available for client user name and password, switch to Developer mode and run in secure mode(secrets are stored in secrete store vault), non- secure mode without client user name and password.

Erickrk Jul 1, 2024

Thank you!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EdgeX Foundry Project

Help with HiveMQ cloud export MQTT using docker compose file #283

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 12 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

EdgeX Foundry Project

Help with HiveMQ cloud export MQTT using docker compose file #283

Devendra-7 Jun 7, 2024

Replies: 5 comments · 12 replies

cloudxxx8 Jun 10, 2024 Maintainer

Devendra-7 Jun 10, 2024 Author

cloudxxx8 Jun 11, 2024 Maintainer

Devendra-7 Jun 11, 2024 Author

Devendra-7 Jun 11, 2024 Author

cloudxxx8 Jun 11, 2024 Maintainer

Devendra-7 Jun 11, 2024 Author

cloudxxx8 Jun 11, 2024 Maintainer

Devendra-7 Jun 11, 2024 Author

Devendra-7 Jun 17, 2024 Author

cloudxxx8 Jun 25, 2024 Maintainer

Erickrk Jun 27, 2024

Erickrk Jun 27, 2024

Erickrk Jun 30, 2024

Devendra-7 Jul 1, 2024 Author

Devendra-7 Jul 1, 2024 Author

Erickrk Jul 1, 2024

Devendra-7
Jun 7, 2024

Replies: 5 comments 12 replies

cloudxxx8
Jun 10, 2024
Maintainer

Devendra-7 Jun 10, 2024
Author

cloudxxx8 Jun 11, 2024
Maintainer

Devendra-7 Jun 11, 2024
Author

Devendra-7 Jun 11, 2024
Author

cloudxxx8
Jun 11, 2024
Maintainer

Devendra-7 Jun 11, 2024
Author

cloudxxx8 Jun 11, 2024
Maintainer

Devendra-7 Jun 11, 2024
Author

Devendra-7
Jun 17, 2024
Author

cloudxxx8
Jun 25, 2024
Maintainer

Erickrk
Jun 27, 2024

Devendra-7 Jul 1, 2024
Author

Devendra-7 Jul 1, 2024
Author