DataGekko Overview

What is DataGekko?

DataGekko is a managed telemetry service running in an enterprise grade setup. It is optimized for fast ingestion, high precision, high reliability and flexible and fast querying.

Easy to use

  • Ingest data using MQTT
  • Manage clients centrally
  • Query data directly from the app

Managed solution

  • No need to worry about uptime
  • Scales with you
  • No infrastructure to maintain and run
  • No development time spent on implementing a custom solution

Flexible

  • Data tagging
  • Long term full resolution data
  • Slicing data across multiple dimensions
  • Easy aggregations

What is time series data

Time series data is a series of values tied to specific points in time. Its purpose is to represent change of the given value over a given timeframe. Some examples are wind speed measurments, seismographs or the closing price of a stock ticker.

Domain applications

  • Computer and software systems
  • Industrial machines
  • Internet of Things
  • Financial systems
  • Event tracking
  • Signal processing

Use cases

  • System operations monitoring
  • Environmental modeling
  • Financial analysis
  • Predictive maintnance
  • Forecasting
  • Machine learning

Data Model

In its raw form the timeseries data model is usually represented as:


Metric: Temperature
Tags: Location=Outside, Position=Rooftop
Timestamp: 2019-01-01 12:01:37
Value: 32.3
										

The value is assigned a timestamp and a metric name along with possible tags for the value to futher describe its source or purpose. Usually this is fine for small scale projects and simple applications. However, more concrete implementations and systems requires several more points of information. This is mostly circumvented by abusing the tag values to hold meta data, which depending on the case might work but is not the right approach. Based on experience so far a key piece of information that is added to timeseries data is some sort of source identification. This can be a device or session identifier in the form of a name or unique identifier. Thus our data model evolves as following:


Metric: Temperature
Source: rooftopSensor1
Tags: Location=Outside, Position=Rooftop
Timestamp: 2019-01-01 12:01:37
Value: 32.3
										

While the addition of the source field is simple, it allows for one more degree of freedom in terms of data representation which allows for queries that separate or join data on another value. On the technical level this is a burden on the querying engine to manage aggregations which is handled for you by the platform transparently.

Getting Started

To get started simply sign up with a new account and you will be presented with your dashboard.
To create a new project select the drop down from the top and click on "Create Project".
Once created, go to "Project Settings" located in the side bar and you will be presented with an API key and secret for your project. You will use these fields in your code to authenticate your devices.

Using your language and MQTT client of choice you can write a script that will send data to our endpoints.
Below is a sample Python script that simply sends one datapoint to be recorded.


import json
import paho.mqtt.client as mqtt
import random
import time

HOST = "edge.datagekko.com"
PORT = 1883

PROJECT_ID = "YOUR-PROJECT-ID"
PROJECT_SECRET = "YOUR-PROJECT-SECRET"
CLIENT_NAME = "testSensor"

current_milli_time = lambda: int(round(time.time() * 1000))

def read_fake_sensor():
    return json.dumps({
        'metric': 'testMetric',
        'timestamp': current_milli_time(),
        'tags': {'direction':'in', 'location': 'roof'},
        'value': random.random()
    })

if __name__ == "__main__":
    client = mqtt.Client(CLIENT_NAME)
    client.username_pw_set(PROJECT_ID, PROJECT_SECRET)
    print("connecting")
    client.connect(HOST, PORT, 60)
    print("connected")
    client.loop_start()
    message = read_fake_sensor()
    client.publish(PROJECT_ID, message, qos=1)
    print("sent: " + message)
    client.loop_stop()

										

A slightly more elaborate sample reads some fake sensor data and sends the recorded value every second to the platform.


import json
import paho.mqtt.client as mqtt
import random
import time

HOST = "edge.datagekko.com"
PORT = 1883

PROJECT_ID = "YOUR-PROJECT-ID"
PROJECT_SECRET = "YOUR-PROJECT-SECRET"
CLIENT_NAME = "testSensor"

class SimpleMqttClient():
    def __init__(self):
        self.client = mqtt.Client(CLIENT_NAME)
        self.client.username_pw_set(PROJECT_ID, PROJECT_SECRET)
        self.client.on_connect = self.on_connect
        self.client.on_message = self.on_message
        print("connecting")
        self.client.connect(HOST, PORT, 60)
        print("connected")
        self.client.loop_start()
        self.connected = -1

    def is_connected(self):
        return self.connected == 1

    def cleanup(self):
        self.client.loop_stop()

    def on_disconnect(self, client, userdata, rc):
        print("client disconnected with code " + str(rc))
        self.connected = 0
        while not rc == 0:
            time.sleep(2)
            print("reconnecting")
            rc = self.client.reconnect()

    def on_connect(self, client, userdata, flags, rc):
        print("connected flags " + str(flags) + " result code " + str(rc) + " client_id  " + str(client))
        if rc > 0:
            print("failed to connect")
        else:
            self.connected = 1

    def on_message(self, client, userdata, message):
        print("message received ", str(message.payload.decode("utf-8")))

    def send_message(self, message):
        if self.connected < 1:
            return
        self.client.publish(PROJECT_ID, message, qos=1)
        print("sent: " + message)

current_milli_time = lambda: int(round(time.time() * 1000))

def read_fake_sensor():
    return json.dumps({
        'metric': 'testMetric',
        'timestamp': current_milli_time(),
        'tags': {'direction':'in', 'location': 'roof'},
        'value': random.random()
    })

if __name__ == "__main__":
    client = SimpleMqttClient()
    try:
        while True:
            client.send_message(read_fake_sensor())
            time.sleep(1)
    except KeyboardInterrupt:
        pass
    finally:
        client.cleanup()

										

The important parts here are:

  • The MQTT host is edge.datagekko.com
  • The MQTT port is 1883
  • The MQTT username is your PROJECT ID
  • The MQTT password is your PROJECT SECRET
  • The MQTT message is a JSON object
  • The MQTT username has the following form:
{
		"metric": "testMetric",
		"timestamp": 1546346096000,
		"tags": {"direction":"in", "location": "roof"},
		"value": 32.3
}
Where:
  • metric is your metric name
  • timestamp is a unix timestamp in milliseconds
  • tags is a JSON object with up to 3 key value pairs
  • value is a 64 bit decimal number