avatar  

Log out

Recently viewed tickets

Simulated data collector

This Article explains the use and options of JSON formatted config file for Data Collection Module in Sitemanager.

Its intended as a reference sheet for the config file creator / maintainer.






Content

General information about the config file

The Collectors

  • Samplepoint items
  • Simulated samplepoint
  • Aggreation


The DataServers

  • Amazon parameters
  • Microsoft Azure parameters
  • DCC by Secomea parameters
  • Cumulocity parameters

Additional information

  • Simulator Functions
  • RPN expressions



General information about the config file.



In the Sitemanager, an Agent is controlling one sample-device. With each such agent a (part of) the DCM configuration in JSON is associated. The agent is associated with a device to be sampled and therefore has the following configuration parameters:

  • IP address
  • Portnumber
  • DCM collector name ( CollectorName )

Names (and namerefs) in the configuration file are in UTF-8, but cannot not start with a digit (0-9+-) and cannot contain (space), ,(comma), : (colon), / (slash), or \ (backslash) and cannot be any of the reserved words or arithmetric operators listed for the rpnexpr field. The name also cannot be empty.

There is only one DCM configuration file per SiteManager. It contains 2 major sections:

The Collectors

This section is a list of collectors from which data is collected.

The DataServers

This section is a list of servers to receive the collected data.



For general management there are also 4 configuration file related fields:

  • ConfigName (string, mandatory ). Unique name for this confguration file. Follows the rules for names.
  • ConfigDescription (string, optional) text describing this configuration.
  • CheckpointInterval (integer == 0 or >= 60, optional). The interval with which all data from the internal datasample cache is saved to disk. This more unstable (power) the SiteManager, the more often data should be saved. Saving it often, however will also cost performance and wear on the harddisk/flash. A value of 0 means, never checkpoint, otherwise it means interval in seconds between saves.
  • AlarmsSaved (integer >= 4, optional). The number of alarms saved in the store-and-forward database. If this values is not specified, it will default to 32.

There is a concept of a sampleref, which is a string uniquely identifying a sample. The format is CollectorName:SampleName, where the CollectorName: part is optional and the CollectorName used will be the one given by the context. Some contexts does not provide any CollectorName, so here it is mandatory.

Partial example:

{
"ConfigName": "DCC_SIM",
"ConfigDescription": "Test configuration for DCC with SIMs only",
"CheckpointInterval": 600,
"AlarmsSaved": 10,
"Collectors": [
],

"DataServers": [
]
}






The Collectors



(array of objects, mandatory ) list elements contains the following fields:

  • CollectorName (string, mandatory ) which is unique name (within this confguration file) of a device to collect data from. (This is the name used in the agent configuration). Follows the normal rules for names.
  • CollectorDesription (string, optional) description of this collector.
  • CollectorIPAddr (string, optional) provides a default IP address (or DNS name) for the device to be polled. The agent Target IP address will take precedence over this.
  • CollectorPortno (integer > 0 and < 65535, optional) provides the default port number for the device to be polled. The agent Target Port number will take precedence over this. If neither CollectorPortno , nor agent Target Port Number is specified, the appropriate port will be used, depending on the Protocol parameter value.
  • Alias (string, optional). This is the CollectorName of another collector, from which, all the parameters (expect CollectorName , CollectorDescription , CollectorIPAddr and CollectorPortno ) are copied from. An alias cannot refer to another alias.
  • Protocol (string-enum, mandatory ) the protocol used to collected data. Currently 3 possible values exists: "OPC-UA/TCP", "Modbus/TCP", "Http/GET" and "Simulator".
  • ConnectRetryMin (integer > 0, mandatory ) Minimum number of seconds before trying to reestablish a connection to the device being sampled. An exponential backoff algorithm is used to retry connection establishment, starting after ConnectRetryMin and increasing to ConnectRetryMax .
  • ConnectRetryMax (integer > 0, mandatory ) Maxumin number of seconds before trying to reestablish a connection to the device being sampled.
  • SamplePoints (Array of objects, mandatory ) List of samples to be collected, possibly aggregated and stored in the Store'n'Forward database.


Partial example:

{

"CollectorName": "TVOsim",
"CollectorDescription": "DCM Internal Simulation Server",
"Protocol": "Simulator",



Samplepoint items

The SamplePoints objects contains the following elements:

  • SampleName (string, mandatory ) Name of this sampled value (sometimes called a tagname). Follows the normal rules for names.
  • SampleDescription (string, optional) is a text description of this sample value.
  • SampleUnit (string, optional) is a description of the unit of messurement for this sample.
  • SampleGroup (string, optional) designates (in some datasrvs) that all samplepoints in that group should be handled (displayed) together. The DCM system itself places no meaning on this group affiliation.
  • ' SamplesSaved (integer > 0, mandatory) is the number of values collected that is saved in the Store'n'Forward database.
  • OnlySaveOnChanged (bool, optional) control is new sample values should only be saved to the store-and-forward database if the values have changed since last sample. If not set, it is false.
  • SampleDataType (string-enum, mandatory ) is the type of data collected. The possible values are: "bool" (1 bit), "sbyte" (8 bit signed), "byte" (8 bit unsigned), "int16" (16 bit signed), "uint16" (16 bit unsigned), "int32" (32 signed), "uint32" (32 bit unsigned), "int64" (64 bit signed), "uint64" (64 bit unsigned), "float" (32 bit), "double" (64 bit), "string" (variable text up to 24575 bytes. Includes terminating NUL), "data" (variable variable data up to 24575 bytes).
  • SampleLowerLimit (integer/double, optional) Is the lower limit for the collected data. If the collected data value is less than SampleLowerLimit it is discarded.
  • SampleUpperLimit (integer/double, optional) Is the upper limit for the collected data. If the collected data value is more than SampleUpperLimit it is discarded.
  • SampleLimitDiscard (bool, maybe-optional). Controls if a sample limit violation results in the sample value being discarded. Mandatory if either SampleLowerLimit or SampleUpperLimit is present.
  • SampleLimitAlarm (bool, maybe-optional). Controls if a sample limit violation generates an alarm. Mandatory if either SampleLowerLimit or SampleUpperLimit is present.
  • SimulatorData (Object, maybe-optional) is a description of the simulation behaviour of this sample point. Mandatory if the protocol type is "Simulator".
  • Aggregation (object, optional) is a description of how to create a new sample value by performing an aggregation function of other sample values, either physically sample values or other aggregation created sample values.


Partial example:

{
"SampleDataType": "uint32",
"SampleDescription": "DCM simulation of sawtooth",
"SampleName": "sawtooth",
"SamplesSaved": 300,
"SimulatorData": {



Simulated samplepoint

The SimulatorData object contains the following elements:

  • SimulatorFunction (string-enum, optional). This is the simulation function to be performed. The currently possible values are: "counter", "sawtooth", "oscillator", and "sine". If none is specified "counter" will be used.
  • SimulatorInitValue (double, mandatory ). The initial value of the simulation data.
  • SimulatorIncValue (double, mandatory ). The value added to the simulation data every sample interval.
  • SimulatorTimescale (double, optional). The timescale (in seconds) the simulation data is being generated. If not specified, the value defaults to same SimulatorSampleInterval value unless it 0, then the value default to 1. If you use the same value for SimulatorTimescale and SimulatorSampleInterval , the functions appear "perfect".
  • SimulatorSampleInterval (integer >= 0, mandatory ). The number of seconds between each sample of this Simulator object. If the interval is 0, only eventdriven polling is used.

Partial example:

{
"SampleDataType": "uint32",
"SampleDescription": "DCM simulation of oscillator",
"SampleName": "oscillator",
"SamplesSaved": 300,
"SimulatorData": {
"SimulatorFunction": "oscillator",
"SimulatorIncValue": 1,
"SimulatorInitValue": 5,
"SimulatorSampleInterval": 10
}

Aggreation

The Aggregation object contains the following fields:

  • Function (string-enum, mandatory ). This is the aggregation function to be performed. The currently possible values are: "max", "min", "avg", "mavg", "derived", "compute" and "eventcompute".
  • Values (array is strings, maybe-optional). This is a list of sampleref references, whose values will be arguments to the aggregation function. Mandatory for all values of Function except "compute".
  • TriggerSample : (string, mandatory ). This is a sampleref to the sample that will trigger the evaluation of the aggregation function. Only when the sample is updated (either by sampling from a device or from another aggregation), will this aggregation function evaluate and create a new sample value itself.
  • Expression (string, maybe-optional). This is an arithmetic expression using Reverse Polish Notation (RPN) with sampleref , numeric constants and operators separated by commas. See the section on RPN expressions. This field is mandatory if Function is "compute" or "eventcompute".
  • TriggerNewInterval (integer > 0, optional). This is a new sample interval (in secs) for the TriggerSample that should be in effect when the event is triggered. When the event is not triggered, the TriggerSample s normal interval is used. Only used when the Function is "eventcompute". If more than one "eventcompute" event forces a new sample interval for the same sample, the lowest new interval is used.

Partial example:

{
"SampleName": "Test1_REG108_DIV",
"SampleDescription": "Register 9 value divided by 100",
"SamplesSaved": 100,
"SampleDataType": "double",
"SampleUnit" : "Volt DC",
"Aggregation" : {
"Function" : "compute",
"Expression" : "Test1_REG108, 100, /",
"TriggerSample" : "Test1_REG108"
}
},






The DataServers

(array of objects, mandatory ) list elements contains the following fields:

  • DatasrvName (string, mandatory ). Unique (within this configuration file) name for each dataserver, this Sitemanager delivers data to. Follows the normal rules for names.
  • DatasrvDesription (string, optional) description of this data server.
  • SampleList (array of strings, optional) This array contains the list of sampleref , this dataserver will deliver data for. If it is not present, all sample values in this configuration will be delivered. Since this object is not a Collectors object, the CollectorName part of the sampleref field is mandatory. If the SampleName part of the sampleref is left blank, all samples from that collector are delivered.
  • DatasrvProtocol (string-enum, mandatory ). Designates the protocol used to deliver data from the Store'n'Forward database to data servers. Currently 5 possible values exists: "AWS\/MQTT", "AZURE\/IOTHUB", "SCI\/SPD" or "C8Y/REST".
  • ConnectRetryMin (integer > 0, mandatory ) Minimum number of seconds before trying to reestablish a connection to the dataserver for data delivery. An exponential backoff algorithm is used to retry connection establishment, starting after ConnectRetryMin and increasing to ConnectRetryMax .
  • ConnectRetryMax (integer > 0, mandatory ) Maxumin number of seconds before trying to reestablish a connection to the dataserver for data delivery.
  • MQTTAWSParams (object, maybe-optional). Contains all the parameters needed to deliver (publish) data to Amazon AWS MQTT cloud service. Mandatory if DatasrvProtocol has the value "AWS\/MQTT".
  • AZUREIotHubParams (object, maybe-optional). Contains all the parameters needed to deliver data to a Azure IoTHub client service. Mandatory if DatasrvProtocol has the value "AZURE\/IOTHUB".
  • SCISPDParams (object, maybe-optional). Contains all the parameters needed to deliver (publish) data to Secomea cloud service, CSI. Mandatory if DatasrvProtocol has the value "SCI\/SPD".
  • C8YParams (object, maybe-optional). Contains all the parameters needed to deliver (publish) data to the Cumulocity Cloud service. Mandatory if DatasrvProtocol has the value "C8Y/REST".

Amazon parameters

The MQTTAWSParams object contains the following fields:

  • BrokerAddr (string, mandatory ) The DNS name/IP address of the server to deliver data to from the Store'n'Forward database.
  • BrokerPort (integer > 0 and < 65535, mandatory ) The TCP port to use for the connection from the SiteManager to the MQTT server.
  • RootCA (string, optional). The certname of the RootCA to use for the MQTT over SSL TCP connection. If not specified, the first AWS ROOT-CA found in the cert repository will be used.
  • DeviceCert (string, optional). The certname of the Device Certificate to use for the MQTT over SSL TCP connection. If not specified, the first AWS Device Certificate found in the cert repository will be used.
  • DevicePrivKey (string, optional). The certname of the Device Private Key to use for the MQTT over SSL TCP connection. If not specified, the first AWS Device Private Key found in the cert repository will be used.
  • PublishQos (string-enum, optional). The QOS (Quality Of Service) used to publish messages to AWS. Currently "qos0" and "qos1" is supported. If not specified, the value "qos0" is used.
  • PublishInterval (integer, mandatory ). The interval in seconds between each delivery of new data from the store'n'forward database.
  • PayloadFormat (string, optional). This string describes the format of the payload reported over MQTT. The format is printf-like with % formats as follows:

'm' - SiteManager MAC address.

'V' - Configuration version ID.

'd' - Collector name.

's' - Sample name.

'u' - Sample unit. If undefined, an empty string.

'g' - Sample group name. If undefined, the sample name will be used.

't' - Timestamp value as an integer string.

'v' - Sample value as a string. (binary data as hex byte string)

'b' - Multiple values are delivered (in bulk) as an array, described in the BulkElementFormat .

'%' - A percent character.

The format can be prefixed with 'j', which will perform escape sequence to data according to the RFC 7159 (JSON) standard. If not specified, the format is "{ "v" : [ %jb ] }".

* BulkElementFormat (string, optional). This string describes the format of the element used by the "%b" format used in the PayloadFormat . The format is printf-like with the same as for PayloadFormat except "%b" is not allowed. If not specified the format is { "ts" : %t, "%jd:%js" : %jv },.

* TopicFormat (string, optional). This string describes the format of the topic used over MQTT. The format is printf-like the same formats as PayloadFormat . If not specified, the format is "%m".

Example:

{
"DatasrvName" : "aws_iot",
"DatasrvProtocol" : "AWS\/MQTT",
"ConnectRetryMin" : 2,
"ConnectRetryMax" : 240,
"MQTTAWSParams" : {
"BrokerAddr" : "a1cn4k05gzdtcl.iot.us-east-2.amazonaws.com",
"BrokerPort" : 8883,
"RootCA" : "awsrootca",
"DeviceCert" : "awsdevcert",
"DevicePrivKey" : "awsdevpkey",
"PublishQos" : "qos1",
"PublishInterval" : 0,
"TopicFormat" : "iotgateway/%d",
"PayloadFormat" : "{ \"%js\" : %jv, \"timestamp\" : %t }"
}
}



Microsoft Azure parameters

The AZUREIotHubParams object contains the following fields:

  • PublishInterval (integer, mandatory ). The interval in seconds between each delivery of new data from the store'n'forward database.
  • ConnectName (string, optional). The certname of the connectstring to use for the Azure connection. If not specified, the first connectstring found in the cert repository will be used.
  • PayloadFormat (string, optional). This string describes the format of the payload reported to Azure IotHub. The format is as for the MQTTAWSParams PayloadFormat field.
  • BulkElementFormat (string, optional). This string describes the format of the element used by the "%b" format used in the PayloadFormat . The format is the same as for MQTTAWSParams PayloadFormat . If not specified the format is { "ts" : %t, "%jd:%js" : %jv },.
  • ContentType (string, optional). The contenttype of the values being delivered. If not specified it is "application/json".
  • ContentEncoding (string, optional). The content encoding of the values being delivered. If not specified it is "utf-8".

Example:

{
"DatasrvName": "azure",
"DatasrvProtocol": "AZURE/IOTHUB",
"ConnectRetryMin": 2,
"ConnectRetryMax": 240,
"AZUREIotHubParams": {
"PublishInterval": 6,
"ConnectName": "azureccstr",
"PayloadFormat": "{ \"v\" : [ %jb ] }",
"BulkElementFormat": "{ \"ts\" : %t, \"%jd:%js\" : %jv }"
}
}



Cumulocity parameters

The C8YParams object contains the following fields:

  • C8YRegisterURL (string, mandatory ). The URL of the Cumulocity Cloud for registering new devices.
  • C8YRootCAName (string, mandatory ). The certname of the Cumulosity Root-CA certificate. If the certname cannot be found only unsecure HTTP server connections can be used.
  • RegName (string, optional). The certname of the Cumulosity Device Registration Information for this connection. If not specified, the first Cumulosity Device Registration Information found in the cert repository will be used.
  • PushInterval (integer >= 0, mandatory ). The interval in seconds between each delivery of new data from the store'n'forward database. The value 0 means that data is delivered as fast as possible.

Example:

{
"DatasrvName" : "secomea_c8y",
"DatasrvProtocol" : "C8Y\/REST",
"ConnectRetryMin" : 2,
"ConnectRetryMax" : 10,
"C8YParams" : {
"C8YRegisterURL" : "https://SecomeaPresales.cumulocity.com",
"RegName" : "c8yreg",
"C8YRootCAName" : "cumulocityROOTCA",
"PushInterval" : 30
}






Additional information

Simulator Functions

The functions in the simulator collector have the following functions:

  • "counter" which is a simple counter function with the initial value SimulatorInitValue and increasing with each sample by SimulatorIncValue .
  • "sawtooth" which is counter function with the initial value SimulatorInitValue and increasing with each sample by SimulatorIncValue . However, when the value has increased 10 times it will change back to SimulatorInitValue and start incrementing again.
  • "oscillator" which is a oscillating function that alternates between SimulatorInitValue and SimulatorInitValue + SimulatorIncValue with each sample.
  • "sine" which generates a sine curve in SimulatorIncValue steps over 2*PI and is scaled by SimulatorInitValue .


RPN expressions

The aggregation functions compute and eventcompute both have a string with an arithmetric expression, that is being evaluated, whenever the TriggerSample value is updated. This expression is represented in RPN (Reverse Polish Notation), which operates on a stack (max 32 values). All calculation are done in either signed 64bit integer, unsigned 64bit integer or double floating point, depending on the datatype of the aggregation sample. Once the result is stored, any overflow is ignored. The RPN is a list of operators separated by commas. The operators are:

  • sampleref which pushes the newest value of the designated value onto the stack.
  • Constant, which pushes the constant onto the stack.
  • "+" (addition), which pops the top 2 values off the stack, adds them and pushes the result onto the stack.
  • "-" (subtraction), which pops the top 2 values off the stack, subtracts them and pushes the result onto the stack.
  • "*" (multiplication), which pops the top 2 values off the stack, multiplies them and pushes the result onto the stack.
  • "/" (division), which pops the top 2 values off the stack, divides them and pushes the result onto the stack.
  • "%" (modulo), which pops the top 2 values off the stack, calculates the modulo of them and pushes the result onto the stack.
  • "^" (exponential), which pops the top 2 values off the stack, calculates the exponential of them and pushes the result onto the stack.
  • "sqrt" (squareroot), which pops the top 1 value off the stack, calculates the squareroot of it and pushes the result onto the stack.
  • "abs" (absolute), which pops the top 1 value off the stack, calculates the absolute value of it and pushes the result onto the stack.
  • "max" (maximum), which pops the top 2 values off the stack, calculates the maximum of them and pushes the result onto the stack.
  • "min" (minimum), which pops the top 2 values off the stack, calculates the minimum of them and pushes the result onto the stack.
  • ">" (greater than), which pops the top 2 values off the stack, compares them and pushes 1 onto the stack if greater than and 0 otherwise.
  • ">=" (greater than or equal to), which pops the top 2 values off the stack, compares them and pushes 1 onto the stack if greater than or equal to and 0 otherwise.
  • "<" (less than), which pops the top 2 values off the stack, compares them and pushes 1 onto the stack if less than and 0 otherwise.
  • "<=" (less than or equal to), which pops the top 2 values off the stack, compares them and pushes 1 onto the stack if less than or equal to and 0 otherwise.
  • "==" (equal to), which pops the top 2 values off the stack, compares them and pushes 1 onto the stack if they are equal and 0 otherwise.
  • "!=" (not equal to), which pops the top 2 values off the stack, compares them and pushes 1 onto the stack if they are not equal and 0 otherwise.
  • "if" (3 values), which pops pops the top 3 values off the stack, pushes top-1 value if top value is non-zero, otherwise top-2 value is pushed.

When the RPN expression ends, the top value is the new value of the aggregation sample. If the aggregation is an "eventcompute" aggregation and the new values is non-zero, the Values list will be collected. This is known as event driven data collection.

If an expression results in a runtime error, (compilation error, divide by zero, stack overflow, insufficient operands for operator or empty stack on completion), the execution is suspended until DCM is restarted.

Creation date: 11/12/2019 14:08 (skr@secomea.com)      Updated: 18/03/2020 08:32 (tvo@secomea.com)