Collector http/rest

The purpose of this article is to be a reference for the Modbus collector part of the full Data Collection Module documentation.

Prerequisite

A Starter package (SiteManager + accounts)
A webserver capable device

Config main

The Config main (array of objects, mandatory) contain the following fields:

Function	Type	Importance	Description
ConfigName	string	mandatory	Unique name for this configuration file. Follows the rules for names.
ConfigDescription	string	optional	text describing this configuration.
CheckpointInterval	integer == 0 or >= 60	optional	The interval with which all data from the internal datasample cache is saved to disk. This more unstable (power) the SiteManager, the more often data should be saved. Saving it often, however will also cost performance and wear on the hard-disk/flash. A value of 0 means, never checkpoint, otherwise it means interval in seconds between saves. If not specified the value 15 * 60 (15 minutes) is used.
AlarmsSaved	integer >= 4	optional	The number of alarms saved in the store-and-forward database. If this values is not specified, it will default to 32.
WaitForNTP	boolean	optional	Synchronize DCM data collection with NTP time. If this parameter is true, all DCM collectors will not begin collecting data until NTP is enabled and configured and NTP has synchronized date/time. If not specified, the value is true.

Since the JSON format itself doesn't define a comment notation, DCM accepts a fieldname comment with any type of value in any section.

Example of comment

"Comment": "Example of a comment for use in any section.":

Collector main

The Collectors (array of objects, mandatory) list elements contain the following fields:

Function	Type	Importance	Description
CollectorName	string	mandatory	Unique name (within this configuration file) of a device to collect data from (this is the name used in the agent configuration). Follows the normal rules for names.
CollectorDescription	string	optional	Description of this collector. For your use, no system function.
CollectorIPAddr	string	optional	Default IP address (or DNS name) for the device to be polled. The agent Target IP address will take precedence over this.
CollectorPortno	integer > 0 and < 65535	optional	Default port number for the device to be polled. The agent Target Port Number will take precedence over this. If neither CollectorPortno , nor agent Target Port Number, is specified, the appropriate port will be used, depending on the Protocol parameter value.
Alias	string	optional	This is the CollectorName of another collector from which all the parameters (expect CollectorName, CollectorDescription, CollectorIPAddr and CollectorPortno) are copied. An alias cannot refer to another alias.
Protocol	string-enum	mandatory	The protocol used to collect data. Currently 3 possible values exists: "OPC-UA/TCP", "Modbus/TCP", "Http/GET" and "Simulator".
ConnectRetryMin	integer > 0	mandatory	The minimum number of seconds before trying to reestablish a connection to the device being sampled. An exponential backoff algorithm is used to retry connection establishment, starting after ConnectRetryMin and increasing to ConnectRetryMax.
ConnectRetryMax	integer > 0	mandatory	Maximum number of seconds before trying to reestablish a connection to the device being sampled.
HTTPGETAccess	object	optional	collection of parameters needed to access a given HTTP server via GET.
SamplePoints	array of objects	mandatory	List of samples to be collected, possibly aggregated and stored in the Store'n'Forward database.

HTTPGETAccess

The HTTPGETAccess objects contain the following elements:

Function	Type	Importance	Description
HTTPGETs	array of objects	mandatory	List of HTTP GET URLs to be performed during collection of data if the Protocol is "Http/GET".
HTTPGETUsername	string	optional	Username for login to HTTP server.
HTTPGETPassword	string	optional	Password for login to HTTP server.
HTTPGETUrlRootCertificate	string	optional	Name of ROOT certificate to use for verifying the HTTPS/FTPS server used in the URL.
HTTPGETTimeout	integer > 0	optional	Timeout (in millisecond) for all Http/Get requests. If not present, the value 500 is used.

HTTPGETs

The HTTPGETs objects contain the following elements:

Function	Type	Importance	Description
HTTPGETID	integer >= 0	mandatory	is a unique ID (within this Collectors element) for this entry. (Used by the SamplePoints entries.)
HTTPGETUrl	string	mandatory	The URL used for this access. It must adhere to the syntax `proto://[user[:password]@]host[:port]][/path]`. The supported "proto" values are "http". "https", "ftp" and "ftps". Note that the host, port, user and password of this URL will be replaced by CollectorIPAddr, CollectorPortno, HTTPGETUrlUsername and HTTPGETUrlPassword.
HTTPGETUrlUsername	string	optional	Username for login to HTTP server for this specific URL. If not present, the value of HTTPGETUsername references by HTTPGETID is used.
HTTPGETUrlPassword	string	optional	Password for login to HTTP server.
HTTPGETUrlRootCertificate	string	optional	Root certificate name for the HTTP server accessed. If not present, the peer validate of the server is not performed.
HTTPGETSampleInterval	integer >= 0	mandatory	Number of seconds between each collection of data using this URL. If the interval is 0, only event-driven polling is used.
HTTPGETUrlTimeout	integer > 0	optional	Timeout (in milliseconds) for these Modbus FC requests. If not present, the value is that of ModbusTimeout referenced by the ModbusFCID.
HTTPGETFormat	string-enum	mandatory	The expected type of content of the response from the device. Currently 3 values exists: "json", "xml", "raw". Note that there is no check for the HTML content-type, only an attempt to run a "json" or "xml" parser on the returned data.

Samplepoint items

The SamplePoints objects contain the following elements:

Function	Type	Importance	Description
SampleName	string	mandatory	Name of this sampled value (sometimes called a tagname). Follows the normal rules for names.
SampleDescription	string	optional	Is a text description of this sample value.
SampleUnit	string	optional	Is a description of the unit of messurement for this sample.
SampleGroup	string	optional	Designates (in some datasrvs) that all samplepoints in that group should be handled (displayed) together. The DCM system itself places no meaning on this group affiliation.
SamplesSaved	integer > 0	mandatory	Is the number of values collected that is saved as minimum in the Store'n'Forward database. If set to 0 the samplepoint isn't stored in the S'n'F database and thus not delivered to cloud, but can be used for aggregations.
OnlySaveOnChanged	bool	optional	Control if new sample values should only be saved to the Store'n'Forward database if the values have changed since last sample. If not set, it is false.
ChangeLimit	integer/double	optional	Only used if OnlySaveOnChanged is true. This describes a limit for what "changed" means. If a new sample value is this amount less or more than the previous sample value, it is considered the "same" value. This is a simple hysteresis to filter out inaccurate (jitter) measurements.
SampleDataType	string-enum	mandatory	The type of data collected. The possible values are: "bool" (1 bit), "sbyte" (8 bit signed), "byte" (8 bit unsigned), "int16" (16 bit signed), "uint16" (16 bit unsigned), "int32" (32 signed), "uint32" (32 bit unsigned), "int64" (64 bit signed), "uint64" (64 bit unsigned), "float" (32 bit), "double" (64 bit), "string" (variable text up to 24575 bytes. Includes terminating NULL), "data" (variable data up to 24575 bytes). Note that not all collectors can handle all data types.
SampleLowerLimit	integer/double	optional	The lower limit for the collected data. If the collected data value is less than SampleLowerLimit, it is evaluated in accordance with SampleLimitDiscard and SampleLimitAlarm.
SampleUpperLimit	integer/double	optional	The upper limit for the collected data. If the collected data value is more than SampleUpperLimit, it is evaluated in accordance with SampleLimitDiscard and SampleLimitAlarm.
SampleLimitDiscard	bool	maybe-optional	Controls if a sample limit violation results in the sample value being discarded. Mandatory if either SampleLowerLimit or SampleUpperLimit is present.
SampleLimitAlarm	bool	maybe-optional	Controls if a sample limit violation generates an alarm. Mandatory if either SampleLowerLimit or SampleUpperLimit is present.
HTTPGETValue	object	maybe-optional	Description of how to address a specific HTTPGET value, when collecting values from it. Mandatory if the protocol type is "Http/GET".

HTTPGETValue

The HTTPGETValue object contains the following elements:


HTTPGETREF	integer >= 0	mandatory	This is a reference to the HTTPGETs element with the same value of it's field HTTPGETID.
Select	string	maybe-optional	This expression selects element from which, the value is to be extracted. Mandatory if HTTPGETFormat has the value "json" or "xml". When the format is "json", the string must adhere to RFC6901 JSON pointer notation. When it is "xml", it must adhere to XPath 1.0 (W3C Recommendation) notation.

http Collector examples

{
            "comment": "Collecting from a simple JSON formatted webserver",
            "CollectorDescription": "Production simulation",
            "CollectorName": "fake",
            "ConnectRetryMax": 240,
            "ConnectRetryMin": 20,
            "HTTPGETAccess": {
                "HTTPGETTimeout": 2000,
                "HTTPGETs": [
                    {
                        "HTTPGETFormat": "json",
                        "HTTPGETID": 1,
                        "HTTPGETSampleInterval": 2,
                        "HTTPGETUrl": "http://10.0.0.249/fake.html"
                    }
                ]
            },
            "Protocol": "Http/GET",
            "SamplePoints": [
                {
                    "HTTPGETValue": {
                        "HTTPGETREF": 1,
                        "Select": "/status"
                    },
                    "SampleDataType": "string",
                    "SampleDescription": "PLC status",
                    "SampleName": "PLCStatus",
                    "SamplesSaved": 30
                }
            ]
        }

{
            "comment": "Collecting from a simple webserver with raw", 
            "CollectorDescription": "Production simulation 2",
            "CollectorName": "fake2",
            "ConnectRetryMax": 240,
            "ConnectRetryMin": 20,
            "HTTPGETAccess": {
                "HTTPGETTimeout": 2000,
                "HTTPGETs": [
                    {
                        "HTTPGETFormat": "raw",
                        "HTTPGETID": 1,
                        "HTTPGETSampleInterval": 2,
                        "HTTPGETUrl": "http://10.0.0.249/fake2.html"
                    }
                ]
            },
            "Protocol": "Http/GET",
            "SamplePoints": [
                {
                    "HTTPGETValue": {
                        "HTTPGETREF": 1,
                        "Select": "/Heading"
                    },
                    "SampleDataType": "string",
                    "SampleDescription": "PLC status",
                    "SampleName": "PLCStatusSTRING",
                    "SamplesSaved": 30
                },
{
                    "HTTPGETValue": {
                        "HTTPGETREF": 1,
                        "Select": "/Heading"
                    },
                    "SampleDataType": "data",
                    "SampleDescription": "PLC status",
                    "SampleName": "PLCStatusDATA",
                    "SamplesSaved": 30
                }
            ]
        }

{
            "comment": "Collecting from a XML formatted webserver",
            "CollectorDescription": "Production simulation",
            "CollectorName": "fakeXML",
            "ConnectRetryMax": 240,
            "ConnectRetryMin": 20,
            "HTTPGETAccess": {
                "HTTPGETTimeout": 2000,
                "HTTPGETs": [
                    {
                        "HTTPGETFormat": "xml",
                        "HTTPGETID": 1,
                        "HTTPGETSampleInterval": 2,
                        "HTTPGETUrl": "http://10.0.0.249/fakeXML.html"
                    }
                ]
            },
            "Protocol": "Http/GET",
            "SamplePoints": [
{
                    "HTTPGETValue": {
                        "HTTPGETREF": 1,
                        "Select": "/value1"
                    },
                    "SampleDataType": "string",
                    "SampleDescription": "version",
                    "SampleName": "version",
                    "SamplesSaved": 30
                },
{
                    "HTTPGETValue": {
                        "HTTPGETREF": 1,
                        "Select": "/value2"
                    },
                    "SampleDataType": "string",
                    "SampleDescription": "number",
                    "SampleName": "number",
                    "SamplesSaved": 30
                }
            ]
        }

See complete http/rest examples in the related articles section below.