DMO Demo setup

Hi y'll,

we're trying to create a demo using the DMO framework (for starters).

So we've downloaded the OVA (DMO-RELEASE-OVA-38_bb4e37.ova) and followed the installation guide (1.0.1). Following the guide does not always work out of the box and we're wondering if we've missed something.

Our observations:

- the setup seems mixed up looking at the accounts and privileges used, the DMO engine is running using the account root while the the account used for the demo is localadmin. It is not recommended to use root for running aplications.

- running the crudDMUserContext script is no effect in the default OVA setup, even though the tooling seems to be successful. One has to run the script using sudo (or change the ownership of /home/localadmin/DMRoot and its subdirectories (which may lead to other issues).

- a minor detail, the manual mentions _cfg_ while the OVA uses cfg as part of the configuration path.

- On attempting to use the krikkit Java demo it became clear that the Java version installed is too old (1.7) while the installed krikkit version runs on Java 8.

- On attempting to use the krikkit C demo it was found that it generates illegal JSON file for the rule.

- When using the webgui an error is reported but only very generally, no specifics are given. Using the http://jsonlint.com/ site the errors are clearly stated.

- the demo only seems to create a rules, there are no simulated sensors to show the complete functioning.

All in all it seems not as far progressed as we would have liked. Are we missing something?

Is anyone else having the same kind of issues?

For the Java version i've used a remote client running Java 8, using eclipse.

It generates the rule:

{

  "meta" : {

  "ruleid" : "testrule",

  "context" : "BraveNewWorld",

  "timer" : "3000" ,

  "cache" : "1024" ,

  },

  "network" : {

  "protocol" : "tcp" ,

  "Filter-by" : {

  "sport" : "443",

       },

  }, ... etc

Which is not valid, though it is accepted by the DMO service according to the java client:

...

nov 16, 2015 10:52:52 AM org.eclipse.krikkit.client.api.API registerRequestSSL

INFO: Request successfully registered

It seems there are some bugs in public String fillBodyMsg(Request request) of org.eclipse.krikkit\krikkit-java-api\src\org\eclipse\krikkit\client\api\APIHelper.java.

For the C api version it's clear that the function 'int fill_body_msg(int handle)' in org.eclipse.krikkit\krikkit-c-api\src\api_helper.c contains bugs; it puts comma's (,) at the wrong places.

In addition to the earlier post:

- its ok that the dmo service is running under root. However, its a bit annoying that within the localadmin directory  new directories are created by the dmo service using root so that a mixed  tree results:

-rwxr-xr-x. 1 root       root       5230 Oct 26 15:30 crudDMUserContext

drwxr-xr-x. 2 localadmin localadmin 4096 Jan 21  2015 Desktop

drwxr-xr-x. 8 root       root       4096 Oct 26 15:34 DMRoot

drwxr-xr-x. 2 localadmin localadmin 4096 Jan 21  2015 Documents

...

drwxr-xr-x. 7 localadmin localadmin 4096 May  3  2015 org.eclipse.krikkit

drwxr-xr-x. 2 localadmin localadmin 4096 Jan 21  2015 Pictures

...

[localadmin@localhost ~]$ sudo -s

[sudo] password for localadmin:

[root@localhost localadmin]# cd DMRoot/

[root@localhost DMRoot]# ls -l

total 20

drwxr-xr-x. 2 root root 4096 Oct 26 15:34 cert

drwxr-xr-x. 2 root root 4096 Oct 29 13:22 cfg

drwxr--r--. 3 root root 4096 Nov 13 04:33 contexts

drwxr-xr-x. 2 root root 4096 Nov 18 12:08 lic

drwxr-xr-x. 2 root root 4096 Oct 26 15:35 log

drwxrwxrwt. 3 root root   60 Nov 19 01:09 mem

...

To test the framework i used the web gui (Data In Motion Configuration Console)  to create a rule; this results in valid json. I used http://jsonlint.com/ to validate.

So i now have the rule:

{

    "meta": {

        "ruleid": "testtemp",

        "context": "BraveNewWorld",

        "timer": "5000",

        "cache": "256"

    },

    "network": {

        "protocol": "tcp",

        "filter-by": {

            "dport": "8888"

        }

    },

    "application": {

        "protocol": "http"

    },

    "content": {

        "query": "temp>40"

    },

    "action": {

        "name": [

            "getPayload"

        ],

        "type": "event",

        "endpoint": {

            "method": "http",

            "port": "8889",

            "resource": "/hello/goodbye"

        }

    }

}

To test the rule i used a simple python script (Could have used the chrome postman plugin).

The python script generates the following payload: {"temp": "45", "device": "42"}

In addition i wrote a python script to receive and dump messages. I started it twice, one is listening to port 8888 and another to 8889.

The idea is to send the mentioned payload to 8888 so that the dmo would trigger and send a message to 8889.

The payload is received at 8888

[localadmin@localhost ~]$ ./mr.py

Args  1 ['./mr.py']

Here we go, serving HTTP on port 8888...

2015-11-19 01:25:37.951072

PUT /node/device/ HTTP/1.1

Host: 127.0.0.1:8888

Accept-Encoding: identity

Content-Length: 30

{"temp": "45", "device": "42"}

But there are no messages at 8889:

[localadmin@localhost ~]$ ./mr.py 8889

Args  2 ['./mr.py', '8889']

Here we go, serving HTTP on port 8889...

I suspect my rule is defect.

Note that dm.log is empty:

[root@localhost contexts]# ls -l /tmp/dm.log

-rw-r--r--. 1 root root 0 Nov 18 10:02 /tmp/dm.log

[root@localhost contexts]#

and the rule log does not show much:

/home/localadmin/DMRoot/contexts/BraveNewWorld/+testtemp

[root@localhost +testtemp]# more testtemp.xlog

# 012ss

testtemp BraveNewWorld 012ss tcp  8888

[root@localhost +testtemp]#

Hi Peter,

Thank you for the feedback! It seems like the rule has been created successfully if you are able to access the +testtemp and the .xlog and .xR files inside.  Would you mind sharing the exact content of the testtemp.json rule? It seems the rule action->endpoint->addr is missing.  Also it would be great if you can share the contents of the .XR file /home/localadmin/DMRoot/contexts/BraveNewWorld/+testtemp/testtemp.xR

The dmo log you are looking for is located at /home/localadmin/DMRoot/log/_dm_L.0000, that should also give some information on the dm as well.

best,

Lifan

Hi Lifan,

thanks for the help, yes the rule created through the web interface is looking good and is valid JSON.

One would expect that empty addresses would default to localhost.

The contents of the files:

/home/localadmin/DMRoot/contexts/BraveNewWorld/testtemp.json

<see above>

/home/localadmin/DMRoot/contexts/BraveNewWorld/+testtemp/testtemp.xlog

[root@localhost +testtemp]# more testtemp.xlog

# 012ss

testtemp BraveNewWorld 012ss tcp  8888

[root@localhost +testtemp]#

/home/localadmin/DMRoot/contexts/BraveNewWorld/+testtemp/testtemp.xR

[root@localhost +testtemp]# more testtemp.xR

<testtemp>

BraveNewWorld 012ss &_IP_PROTO_  &_TCP_DPORT_  @DCI:_SET_CACHE_(256)  @DCI:_SET_

TIMER_(5000)  @DCI:_DCI_EARLY_  @DCI:_SET_CONTENT_(application\/json)  @DCI:_DCI

_PAYLOAD_  @DCI:_SET_CONNECTION_(http:\/\/127.0.0.1:8889\/\/hello\/goodbye)

        <@DCI:_SET_CONDITION_>

                _mem_ 012q; temp\>40

        </@DCI:_SET_CONDITION_>

</testtemp>

[root@localhost +testtemp]#

/home/localadmin/DMRoot/log/_

[Fri, 20 Nov 2015 01:09:09 PST] [2387896288] DMo v1.0.2 Compiled on Mon Oct 26 2

2:30:05 UTC 2015 by [SER]

[Fri, 20 Nov 2015 01:09:09 PST] [2387896288] _NETXY_XCOPY_ _open_ 127.0.0.1 80

[Fri, 20 Nov 2015 01:09:09 PST] [2387896288] dmo.lic - License check passed

[Fri, 20 Nov 2015 01:09:09 PST] [2387896288] Timer rules enabled

[root@localhost log]#

The logs don't seem to grow or to log much, even after sending the testmessage.

For debugging purposes it would be good to be able to enhance the logging; so that at least its visible if any message is inspected by the DMO service.

/tmp/dm.log:

[root@localhost tmp]# ls -l dm.log

-rw-r--r--. 1 root root 0 Nov 20  2015 dm.log

<empty>

Thanks for your time and effort; its much appreciated!

Modified the rule to include an ip in the endpoint:

{

    "meta": {

        "ruleid": "testtemp",

        "context": "BraveNewWorld",

        "timer": "1500",

        "cache": "1024"

    },

    "network": {

        "protocol": "tcp",

        "filter-by": {

            "dstaddr": "127.0.0.1",

            "dport": "8888"

        },

        "content": "application/json"

    },

    "application": {

        "protocol": "http"

    },

    "content": {

        "query": "temp>40"

    },

    "action": {

        "name": [

            "getPayload"

        ],

        "type": "event",

        "endpoint": {

            "method": "http",

            "addr": "127.0.0.1",

            "port": "8889",

            "resource": "/hello/goodbye"

        }

    }

}

On sending the payload '{"temp": 45, "device": 42}' 10 times with an interval of 1 second no results at the given endpoint were found.

Added a time rule:

{

    "meta": {

        "ruleid": "testtime",

        "context": "BraveNewWorld"

    },

    "action": {

        "name": [

            "FETCHDATA"

        ],

        "type": "timer",

        "period": "1500",

        "endpoint": {

            "method": "http",

            "addr": "127.0.0.1",

            "port": "8889",

            "resource": "/time/up"

        }

    }

}

---

This rule works insofar it generates a GET request every 1.5 seconds, though the timing seems what inaccurate.

2015-11-23 02:49:17.472658

GET /time/up HTTP/1.1

User-Agent: DM-1.0

Host: 127.0.0.1

Connection: close

Content-Length: 0

2015-11-23 02:49:19.481343

GET /time/up HTTP/1.1

User-Agent: DM-1.0

Host: 127.0.0.1

Connection: close

Content-Length: 0

2015-11-23 02:49:21.506493

GET /time/up HTTP/1.1

User-Agent: DM-1.0

Host: 127.0.0.1

Connection: close

Content-Length: 0

2015-11-23 02:49:23.518381

GET /time/up HTTP/1.1

User-Agent: DM-1.0

Host: 127.0.0.1

Connection: close

Content-Length: 0

2015-11-23 02:49:25.537480

GET /time/up HTTP/1.1

User-Agent: DM-1.0

Host: 127.0.0.1

Connection: close

Content-Length: 0

2015-11-23 02:49:27.538504

GET /time/up HTTP/1.1

User-Agent: DM-1.0

Host: 127.0.0.1

Connection: close

Content-Length: 0

2015-11-23 02:49:13.443513

GET /time/up HTTP/1.1

User-Agent: DM-1.0

Host: 127.0.0.1

Connection: close

Content-Length: 0

So i've modified the response on the get to include json payload '{"temp": 55, "warning": 0}'.

Still wondering whether this response is processed somewhere. Event rule still does not seem to fire.

There should be a way to enhance the debugging level of the DMO service.

Hi Peter,

Sorry about the late response, I just got back from vacation.  let me reproduce this setup in the DMO-RELEASE-OVA-38_bb4e37.ova and try to figure out why DMo is not triggering the events.  The rules seem fine from the logs you have copied here.  let me verify in my setup and then we can setup a webex session and I can help you through that.

best,

Lifan

I hope you've had a good time!

In addition, here are the two scripts i used to generate and check for the payload....

mr.py, for receiving, responding and printing messages, the part at which it listens is default 8888 but can be overridden on the command line:

# -*- coding: utf-8 -*-

#!/usr/bin/python

import datetime

import json

import re

import socket

import sys

import string

HOST, PORT = '', 8888

class MessageError(Exception): pass

class MessageReader(object):

    def __init__(self,sock):

        self.sock = sock

        self.buffer = b''

    def get_until(self,what):

        while what not in self.buffer:

            if not self._fill():

                return b''

        offset = self.buffer.find(what) + len(what)

        data,self.buffer = self.buffer[:offset],self.buffer[offset:]

        return data

    def get_bytes(self,size):

        while len(self.buffer) < size:

            if not self._fill():

                return b''

        data,self.buffer = self.buffer[:size],self.buffer[size:]

        return data

    def _fill(self):

        data = self.sock.recv(1024)

        if not data:

            if self.buffer:

                raise MessageError('socket closed with incomplete message')

            return False

        self.buffer += data

        return True

#============

print 'Args ', len(sys.argv), str(sys.argv)

if len(sys.argv) > 1:

   PORT=int(sys.argv[1])

listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

listen_socket.bind((HOST, PORT))

listen_socket.listen(1)

print 'Here we go, serving HTTP on port %s...' % PORT

while True:

    client_connection, client_address = listen_socket.accept()

    mr = MessageReader(client_connection)

    dt = datetime.datetime.now()

    print dt

    header = mr.get_until(b'\r\n\r\n')

    print 'header>>'+(header.decode('ascii'))

    m = re.search(b'Content-Length: (\d+)',header)

    if m:

       length = int(m.group(1))

       data   = str(mr.get_bytes(length))

       print 'data>>'+data

       headers  = 'Content-Type: application/json\r\nAccept: text/plain\r\n'

       response = 'HTTP/1.1 200 OK\r\n\n'

       if header.startswith('GET'):

           print 'Getting'

           sense = '{"temp": 55, "warning": 0}\r\n'

           contentlength = 'Content-Lenght: '+str(len(sense))+'\r\n\r\n'

       if header.startswith('PUT'):

           print 'Putting'

           sense = ''

           contentlength = 'Content-Lenght: '+str(len(sense))+'\r\n\r\n'

       mess = headers + response + contentlength + sense

       print '>>>'

       print mess

       print '<<<'

       client_connection.sendall(mess)

       client_connection.close()

Be aware that this forum might disrupt the formatting somewhat and python is very sensitive to formatting.

The script i use for sensing messages is: sr.py

#!/usr/bin/python

#import requests

import sys

import time

import httplib

import json

NUMBER   = 10 # Number of messages to sent

INTERVAL = 1  # interval in seconds between messages

if len(sys.argv) > 1:

    NUMBER   = int(sys.argv[1])

    INTERVAL = int(sys.argv[2])

payload = '{"temp": 45, "device": 42}'

print payload,len(payload)#,json.dumps(payload)

#r = requests.post("http://127.0.0.1:8888", data=json.dumps(payload))

connection = httplib.HTTPConnection('127.0.0.1:8888')

for i in range(0, NUMBER):

   print "Sending"

   connection.request("PUT", "/node/device/", payload,{"Content-Type":"application/json"})

   result = connection.getresponse()

   time.sleep(INTERVAL)

   print result.status, result.reason, result.read()

   #print r.status_code

   #print r.text

print "Done"

Anyway thanks for your help! I appreciate your time and effort.

Hi Peter,

Thanks for providing me with the scripts.  Can you try running the script from a different vm/machine?  It seems like everything is sent through the lo interface currently.  I'm debugging to see why dm is not sensing the traffic on the lo, in the meantime, I've tried using a different machine/vm to send the sensor data through eth0 or eth1 and it was working fine.  Please try that and let me know.  also you can contact me at lifzhang@cisco.com, this way  i can get back to you quicker.

best,

Lifan