IoT Platform

1. Overview of the platform

IoT Platform is based on EPOS SmartData definition (see Smart Data documentation for details).

In this way, a SmartData can be defined as a data structure containing the following properties:

SmartData
	version	unit	value	error	confidence	x	y	z	t	dev	signature	workflow

While a Series could be defined as the following data structure:

Series
	version	unit	x	y	z	r	t0	t1	dev	signature	workflow

Data on the platform is organized on SmartData records and Series. Specifically, each SmartData record is seen as a data point of a given time Series, according to its version, unit, time and geographic location.

Each of the previously presented properties semantics is briefly explained next. For further details on the composition of the value of each property, see the Smart Data documentation.

Common properties to smartdata and series:

version: The SmartData version. Possible values:
- "1.1": Version 1, Static. Data from a sensor that is not moving.
- "1.2": Version 1, Mobile. Data from a sensor that is moving.
unit: The SI (or digital) unit of the data. See typical units or the full documentation.
dev: (optional) An identifier for multiple transducers of the same unit in the same coordinate (e.g., 3-axis accelerometer)
workflow: (optional) Executes a server side code, identified by an integer, with the series in insert operations. The integer should be specified in the field workflow of the attach operation. Currently, this code could be user defined but need to be installed by system administrators and the first insert of a series should not use a workflow in its respective attach. The semantics of query operations (get.php) with workflow specified is to return only data inserted using that workflow. If query operations (get.php) do not specify a workflow, they return all data that were inserted with and without workflow specified.
signature: (only for mobile version) An identifier for mobile version of SmartData (version 1.2)

Specific properties:

smartdata: Array of one or more elements, each containing information about a particular measurement (data point).
- value: The data value itself (e.g. the temperature measured by a thermometer).
- error: The detected measurement error.
- confidence: The level of confidence in this measurement.
- x, y, z: The absolute coordinates where the measurement was taken.
- t: The time when the measument was taken, in UNIX epoch microseconds.

series: Information about data to be fetched.
- x, y, z: The absolute coordinates of the center of the sphere of interest.
- d: User-defined tag, normally used as a disambiguator of unit.
- r: The radius of the sphere in which all data points of interest are contained.
- t0: The start time of data to be fetched, in UNIX epoch microseconds.
- t1: The end time of data to be fetched, in UNIX epoch microseconds.

Yet, some other properties can be useful when using the IoT Platform, the Credentials

credentials: Optional information about data domain and user.
- domain: A data domain to be queried. Defaults to "public". (obligatory for non-certificated access or for certificates with multi-domain access)
- username: Used when accessing a private domain.
- password: Used when accessing a private domain.

Typical units representation

2224441636 : Length
2224212260 : Mass
2224183588 : Time
2224180004 : Current
2224180004 : Electric Current
2224179556 : Temperature
2224179500 : Amount of Substance
2224179493 : Luminous Intensity
2224703780 : Area
2224965924 : Volume
2224437540 : Speed
2224437540 : Velocity
2224433444 : Acceleration
2224199972 : Irradiance
2223941924 : Pressure
2224723748 : Electric Potential
2224961828 : Water Flow

Refer to EPOS user guide to check unit value composition.

1.1. Developer's Guide

Developer's guide here.

2. REST API

2.1. Query data

Method: POST
URL: https://iot.lisha.ufsc.br/api/get.php
Body:

{
    "series" : Object
    {
        "version" : string
        "unit" : unsigned int
        "x" : int
        "y" : int
        "z" : int
        "r" : unsigned int
        "t0" : unsigned int
        "t1" : unsigned int
        "dev" : unsigned int
        "signature" : unsigned int
        "workflow": unsigned int
    }
    "credentials" : Object
    {
        "domain" : string
        "username" : string
        "password" : string
    }
}

Optional properties: workflow, credentials, domain, username, password.

2.1.1. Data Aggregation

When querying data, it is possible to execute an aggregation function over it. This can be done by changing the requesting query by adding an aggregation function.

{
          'series' : {
              'version' : string,
              'unit' : unsigned int,
              'x' : int,
              'y' : int,
              'z' : int,
              'r' : 0,
              't0' : unsigned int,
              't1' : unsigned int,
              'dev' : unsigned int,
              'signature' : unsigned int,
              'workflow': unsigned int
          },
          'credentials' : {
              'domain' : string,
              'username' : string,
              'password' : string
          },
          'aggregator' : {
              'name'  : string,
              'range' : unsigned int,
              'parameter' : float, 
              'delay' : unsigned int,
              'spacing' : unsigned int
          }
    }

Time properties range, delay, and spacing expressed in us.

The list of available aggregators and their parameters is presented below.

min: Returns the minimum value of an interval of range "range". Parameters: range.
max: Returns the maximum value of an interval of range "range". Parameters: range.
mean: Returns the mean of the set of values of an interval of range "range". Parameters: range.
lowerThan: Returns the value if lower than a threshold defined by "parameter", else returns null. Parameters: parameter.
higherThan: Returns the value if lower than a threshold defined by "parameter", else returns null. Parameters: parameter.
can: Treats the data from the CAN port with an offset (represented by "range") and a given id ("parameter"). Parameters: range, parameter.
confidence: SmartData confidence becomes its value. No parameters needed.

2.1.2. Fault Injection

Some aggregators have been designed in order to inject faults on SmartData time-series. They are:

drift: Applies a drift of "parameter" to the values of the intervals of range "range" after an offset "delay" and repeating after every "spacing" (if this is higher than 0). The drift varies according to the number of samples it has been applied. Parameters: range, parameter, delay, and spacing.
stuckAt: The values of the SmartDatas on the interval "range" becomes the first interval SmartData value. Counts after an offset "delay" and repeating after every "spacing" (if this is higher than 0). Parameters: range, delay, and spacing.
constantBias: Sums a value of "parameter" to every SmartData value of the intervals of range "range" after an offset "delay" and repeating after every "spacing" (if this is higher than 0). Parameters: range, delay, parameter, spacing.
constantGain: Multiplies the value of "parameter" by every SmartData value of the intervals of range "range" after an offset "delay" and repeating after every "spacing" (if this is higher than 0). Parameters: range, delay, parameter, spacing.
allAnomalies: Apply drift, stuckAt, constantBias and constantGain to different subsequent intervals of range "range", separated by "spacing" and counting after "delay". Parameters: range, delay, spacing, drift, stuck, bias, gain. If any of those (drift, stuck, bias, or gain) is 0, the respective aggregator is not applied.

2.2. Create or Attach series

Create will create a series only if it does not already exist (identical parameters).
Attach will create a series only if there is no series containing it (space and time).

Method: POST
URL for create: https://iot.lisha.ufsc.br/api/create.php
URL for attach: https://iot.lisha.ufsc.br/api/attach.php
Body:

{
    "series" : Object
    {
        "version" : string
        "unit" : unsigned int
        "x" : int
        "y" : int
        "z" : int
        "r" : unsigned int
        "t0" : unsigned int
        "t1" : unsigned int
        "dev" : unsigned int
        "signature" : unsigned int
        "workflow": unsigned int
    }
    "credentials" : Object
    {
        "domain" : string
        "username" : string
        "password" : string
    }
}

Optional properties: workflow, credentials, domain, username, password.

2.3. Insert data

Method: POST
URL: https://iot.lisha.ufsc.br/api/put.php
Body:

{
    "smartdata" : Array
    [
        {
            "version" : string
            "unit" : unsigned int
            "value" : double
            "error" : unsigned int
            "confidence" : unsigned int
            "x" : int
            "y" : int
            "z" : int
            "t" : unsigned int
            "dev" : unsigned int
            "signature" : unsigned int
        }
    ]
    "credentials" : Object
    {
        "domain" : string
        "username" : string
        "password" : string
    }
}

Optional properties: credentials, domain, username, password.

2.4. AI Workflows

SmartDatas on the platform can be submitted to specific workflows to process data before its proper insertion. This workflow execution is related to the series this data belongs (The SmartData relation to the Series is presented previously on the Overview of the platform ). There are two types of workflow a SmartData can go through: the Input Workflows (when it is inserted) and the Output Workflows (when they are part of the response of a get request).

For each SmartData insertion, the platform lists the series in which this SmartData belongs. The SmartData is submitted to the Input Workflow defined by its series. If the series defines workflow as 0, no Input Workflow is executed for the inserted SmartData, in any other case, the Input Workflow with the number defined by the series is executed. In case the workflow specified by the series is not an available Input Workflows on the domain the SmartData is being inserted to, nothing is executed. The very same process is applied on every get request for the Output Workflows.

The workflow receives as parameter a string representing the SmartData JSON and must return a string that represents the processed SmartData JSON.

The workflows are stored on directories according to the domain they bellong (i.e., "smartdata/bin/workflow/<domain>/"). Input Workflows should be named as "in" followed by the workflow number, e.g., "in1". Output Workflows should be named as "out" followed by the workflow number, e.g., "out1"

A simple example of python workflow

#!/usr/bin/env python3
import sys
import json

if __name__ == '__main__':
    #+++++++++++++++++ DO NOT CHANGE THIS LINE +++++++++++++++++
    smartdata = json.loads(sys.argv[1]) # Load json from argv[1]
    #+++++++++++++++++ DO NOT CHANGE THIS LINE +++++++++++++++++


    # ...
    # DO SOMETHING HERE
    smartdata['value'] = 2*smartdata['value'] # example
    # ...


    #+++++++++++++++++ DO NOT CHANGE THIS LINE +++++++++++++++++
    print(json.dumps(smartdata)) # Send smartdata back to PHP
    #+++++++++++++++++ DO NOT CHANGE THIS LINE +++++++++++++++++

2.4.1. Persistency

The workflow is executed for each instance of SmartData, i.e., the SmartDatas are processed by the workflow individually. For the workflows that need some persistence and must execute during all the SmartDatas processing, a daemon should be created. Daemons are meant to be separated processes that receive data from the workflow, do the processing, and either return this to the workflow or insert the processed data on a new timeseries. Whenever a workflow is supposed to execute, the platform checks for the existence of the deamon for this workflow. If the daemon does exist, the platform backend checks if it is a running process and initializes it in case it is not. Each workflow must manage its own data, including daemon's input and output.

The daemons must be placed on the same directory as their respective workflows (i.e., "smartdata/bin/workflow/<domain>/"). Each workflow can have only one daemon. The daemon of a workflow must be named with the name of the workflow, plus the word "daemon", e.g., "in1_daemon". Common names for the files that receives daemon inputs and outputs are composed by the name of the workflow, plus the words "input" or "output", e.g., "in1_input".

The daemon execution is managed by the platform with the help of two files, one holding the process pid, and the other holding the execution log. The pid file is named with the workflow name, plus the word "pid", e.g., "in1_pid". The execution log file is named with the workflow name, plus the word "log", e.g., "in1_log".

Daemons are also meant to have a life cycle and must be finalized after the end of the SmartDatas processing. This can be managed with a watchdog implementation over the input file content.

The following example of workflow writes its input into a file to be processed by the daemon, keeping data persistency

import sys, json
from process_verify import process_is_alive

if __name__ == '__main__':
    '''
     This dummy workflow is used to calculate the average of the last 10 inserted smartdatas
    '''

    # If not running -> setup daemon
    process_is_alive()

    if len(sys.argv) != 2:
        exit(-1)

    #+++++++++++++++++ DO NOT CHANGE THIS LINE +++++++++++++++++
    smartdata = json.loads(sys.argv[1]) # Load json from argv[1]
    #+++++++++++++++++ DO NOT CHANGE THIS LINE +++++++++++++++++


    # ...
    # DO SOMETHING HERE IF IT WILL CHANGE DATA
    # ...


    #+++++++++++++++++ DO NOT CHANGE THIS LINE +++++++++++++++++
    print(json.dumps(smartdata)) # Send smartdata back to PHP
    #+++++++++++++++++ DO NOT CHANGE THIS LINE +++++++++++++++++

    # ...
    # DO SOMETHING HERE IF IT WILL NOT CHANGE DATA (INCREASES PARALELLISM BY UNBLOCKING PHP)
    with open('in1_input', 'a') as fifo:
        fifo.write(json.dumps(smartdata)+"\n")
        fifo.close()
    # ...

2.4.2. Loading previous data

The daemon usually receives its entry from its respective input file. On the first SmartData to be processed, however, this input file would be empty. In this case, an importer would be executed to get historic data from the very same series to fulfill the input file.

This daemon piece of code imports data if input file is not available yet

...
    if not os.path.exists("in1_input"):
        os.system("python3 get_data.py -t 1728000")
...

2.4.3. Inserting new data

In case the workflow does not change the SmartData, it may insert the processed SmartData on other timeseries through a data inserter. This script must create a different timeseries, for the new data. A possibility is to use the very same series configuration but with another dev.

This daemon piece of code exports the calculated SmartData if the put script is available

...
if os.path.exists("put_data.py"):
    os.system("python3 put_data.py -t "+str(smartdata['timestamp'])+" -t0 1728000 -v "+str(np.average(v_buffer)))
...

2.5. Response codes

The HTTP response codes are used to provide a response status to the client.

Possible response codes for an API request:

200:
- get.php: it means that a query has been successfully completed and the response contains the result (which may be empty)
204:
- create.php: it means that the series has been created successfully (there is no content in the response).
- attach.php: it means that the series has been attached successfully (there is no content in the response).
400: it means there is something wrong with your request (bad format or inconsistent field).
401: it means that you are not authorized to manipulate your domain.

2.6. Plotting a dashboard with Grafana

To plot a graph, do the following:

1. Inside Grafana's interface, go to Dashboards => Create your first dashboard => Graph.
2. Now that you are seeing a cartesian plane with no datapoints, click on Painel Title => Edit.
3. This should take you to the Metrics tab. Now you can choose your Data Source and put it's due information.
4. If you are using SmartData UFSC Data Source, fill the Interest and Crendential fields with the information used in section 2.4.
5. You can tweak your plotting settings by using the Axes, Legend or Display tabs. Save your Dashboard by hitting Ctrl+S.

After doing these steps, information should be shown instantly.

3. Binary API

To save energy on the IoT wireless, battery operated network, the platform also accepts native Smart Data structures, encoded as binary EPOS@ARM (32-bit little-endian). Each data point to be inserted into the database is sent as the 75-byte concatenation of the DB_Series with the DB_Record structures:

struct DB_Series {
    unsigned char version;
    unsigned long unit;
    long x;
    long y;
    long z;
    unsigned long r;
    unsigned long long t0;
    unsigned long long t1;
    unsigned long dev;
}
struct DB_Record {
    double value;
    unsigned char error;
    unsigned char confidence;
    long x;
    long y;
    long z;
    unsigned long long t;
    unsigned long dev;
}

All the data in the structures are derived from the TSTP packets and the corresponding Smart Data. For further details on each field, see the Smart Data documentation.

3.1. Create or Attach series

Create will create a series only if it does not already exist (identical parameters).
Attach will create a series only if there is no series containing it (space and time).

Method: POST
URL for create: https://iot.lisha.ufsc.br/api/create.php
URL for attach: https://iot.lisha.ufsc.br/api/attach.php
Body: Series

Byte	40	36	32	28	24	20	12	4	0
	version	unit	x	y	z	r	t0	t1	dev

3.2. Insert data

Method: POST
URL: https://iot.lisha.ufsc.br/api/put.php
Body: Smart Data

Byte	34	30	22	21	20	16	12	8	0
	version	unit	value	error	confidence	x	y	z	t

3.3. Version format

The version field has 8 bits and is composed of a major and a minor version. The major version is related to the API compatibility. On the other hand, the minor version defines some properties of the Smart Data. For instance, the minor version 1 defines a static Smart Data, while the minor version 2 defines a mobile Smart Data.

enum {
    STATIC_VERSION = (1 << 4) | (1 << 0),
    MOBILE_VERSION = (1 << 4) | (2 << 0),
};

4. Client Authentication

The EPOS IoT API infrastructure supports authentication with client certificates. In order to implement it, you should request a client certificate to LISHA in the Mailing List.

If you are using the eposiotgw script to send smartdata from a TSTP network to IoT API infrastructure, you should do the following steps to authenticate with the client certificate.

1. Use eposiotgw available on EPOS GitLab
2. Copy the files .pem and .key provided by LISHA to the same directory of the eposiotgw script
3. Call eposiotgw using the parameter -c with the value equal the name of the certificate file WITHOUT the extension. Both files (.pem and .key) should have the same basename. For example, with the files client-1-A7B64D415BD3E97B.key and client-1-A7B64D415BD3E97B.key, you should call the -c parameter using the value client-1-A7B64D415BD3E97B.

If you are using esp8266 with axTLS library, you should convert the certificates to a suitable format, with two .der files. To do this follow the instructions below:

openssl pkcs12 -export -clcerts -in client-6-A7B64D415BD3E980.pem -inkey client-6-A7B64D415BD3E980.key -out client.p12
openssl pkcs12 -in client.p12 -nokeys -out cert.pem -nodes
openssl pkcs12 -in client.p12 -nocerts -out key.pem -nodes
openssl x509 -outform der -in cert.pem -out cert.der
openssl rsa -outform der -in key.pem -out key.der

5. Scripts

5.1. Python

5.1.1. Get Script Example

The following python code queries luminous intensity data at LISHA from the last 5 minutes.

#!/usr/bin/env python3
import time, requests, json

get_url ='https://iot.ufsc.br/api/get.php'

epoch = int(time.time() * 1000000)
query = {
        'series' : {
            'version' : '1.1',
            'unit'    : 2224179493,
            'x'       : 741868770,
            'y'       : 679816011,
            'z'       : 25285,
            'r'       : 10*100,
            't0'      : epoch - (5*60*1000000),
            't1'      : epoch,
            'dev'   : 0
        },
        'credentials' : {
        	'domain' : 'smartlisha',
                'username' : 'smartusername',
                'password' : 'smartpassword'
        }
    }
session = requests.Session()
session.headers = {'Content-type' : 'application/json'}
response = session.post(get_url, json.dumps(query))

print("Get [", str(response.status_code), "] (", len(query), ") ", query, sep='')
if response.status_code == 200:
	print(json.dumps(response.json(), indent=4, sort_keys=False))

5.1.2. Put Script Example

The following python code inserts a json with a certificate.

#!/usr/bin/env python3

# To get an unencrypted PEM (without passphrase):
# openssl rsa -in certificate.pem -out certificate_unencrypted.pem

import os, argparse, requests, json,ssl

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager


parser = argparse.ArgumentParser(description='EPOS Serial->IoT Gateway')

required = parser.add_argument_group('required named arguments')
required.add_argument('-c','--certificate', help='Your PEM certificate', required=True)
parser.add_argument('-u','--url', help='Post URL', default='https://iot.lisha.ufsc.br/api/put.php')
parser.add_argument('-j','--json', help='Use JSON API', required=True)


args = vars(parser.parse_args())
URL = args['url']
MY_CERTIFICATE = [args['certificate']+'.pem', args['certificate']+'.key']
JSON = args['json']

class MyAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       ssl_version=ssl.PROTOCOL_TLSv1)

session = requests.Session()
session.mount('https://', MyAdapter())
session.headers = {'Content-type' : 'application/json'}
session.cert = MY_CERTIFICATE
try:
    response = session.post(URL, json.dumps(JSON))
    print("SEND", str(response.status_code), str(response.text))
except Exception as e:
    print("Exception caught:", e)

5.2. R

5.2.1. Get Script Example

The following python code queries Temperature data at LISHA from an arbitrarily defined time interval.

library(httr)
library(rjson)
library(xml2)
    
get_url <- "https://iot.lisha.ufsc.br/api/get.php"
    
json_body <-
'{
  "series":{
    "version":"1.1",
    "unit":2224179556,
    "x":741868840,
    "y":679816441,
    "z":25300,
    "r":0,
    "t0":1567021716000000,
    "t1":1567028916000000,
    "dev":0,
    "workflow": 0
  },
  "credentials":{
    "domain":"smartlisha"
  }
}'
    
res <- httr::POST(get_url, body=json_body, verbose())
res_content = content(res, as = "text")
    
print(jsonlite::toJSON(res_content))

The following code gets Temperature data at LISHA from the last 5 minutes.

library(httr)
library(rjson)
library(xml2)
    
get_url <- "https://iot.lisha.ufsc.br/api/get.php"

time <- Sys.time()
time_0 <-as.numeric(as.integer(as.POSIXct(time))*1000000)

json_body <-
'{
  "series":{
    "version":"1.1",
    "unit":2224179556,
    "x":741868840,
    "y":679816441,
    "z":25300,
    "r":0,
    "t0":'
json_body <- capture.output(cat(json_body, time_0 - 5*60*1000000))
json_body <- capture.output(cat(json_body, ',"t1":'))
json_body <- capture.output(cat(json_body, time_0))
end_string <- ', 
    "dev":0,
    "workflow": 0
  },
  "credentials":{
    "domain":"smartlisha"
  }
}'
json_body <- capture.output(cat(json_body, end_string))
 
res <- httr::POST(get_url, body=json_body, verbose())

res_content = content(res, as = "text")
    
print(jsonlite::toJSON(res_content))

5.3. Java

6. Troubleshooting

6.1. TLS support for Post-Handshake Authentication

TLS 1.3 has the Post-Handshake Authentication disabled by default, however, the IoT platform requires PHA to securely connect with clients. This issue can be easily worked around with a custom SSLContext forcing the use of TLS 1.2 which has PHA enabled by default. An example in Python follows:

import ssl

ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
connection = HTTPSConnection("iot.lisha.ufsc.br", 443, context=ctx);