2014-09-13

Installing Cassandra on MINIX NEO X5 min (android multimedia player)

Intro

I started doing some DIY home automation projects. Although I have the mega popular Raspberry Pi available I decided to use the MINIX NEO X5 mini because I felt this device could be used a lot better if it served me as some sort of home automation server. The first part in this story is getting a more server oriented OS on the device. I decided to go with the linux. After a lot of searching and trial and error I decided to deploy an application called Linux deploy and described it in my previous blog post. Trough the rest of the tutorial I'll assume you managed to install a linux instance on your MINIX. I am going to gather a lot of telemetry data with the solution I am building so installing Cassandra seems as a natural choice to me. There will be a lot of writes and Cassandra is good at writing at an incredible scale.

Installing Java

        $ echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | sudo tee /etc/apt/sources.list.d/webupd8team-java.list
        $ echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | sudo tee -a /etc/apt/sources.list.d/webupd8team-java.list
        $ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
        $ sudo apt-get update
        $ sudo apt-get install oracle-java8-installer
        # you'll need to accept license agreement

        # set environment variables
        $ sudo apt-get install oracle-java8-set-default

        # login once again just in case
        $ exit
    

Installing python

Cassandra comes with a very nice tool called cqlsh. The version of linux we currently have installed will not run it without a python available on the system. So we have to install it first.

        $ sudo apt-get install python2.7
    

Let's start the Cassandra

Configuring the Cassandra is a chapter on it's own. We'll make minimal adjustments before starting. We'll configure the Cassandra to respond to queries from other hosts and while we are at it we'll enable the virtual nodes. (Will be easier to scale later).

        $ cd CASSANDRA_INSTALL_DIRECTORY
        $ nano conf/cassandra.yaml

        # uncomment
        num_tokens: 256

        # change to 0.0.0.0
        # this will enable you to contact the cassandra
        # from other computers etc.
        rpc_address: 0.0.0.0

        #save file

        $ cd ..
        $ ./bin/cassandra

        # after seeing something like
        # Startup completed! Now serving reads.
        # press ^C (don't be afraid cassandra still runs)

        $ bin/cqlsh

        Connected to Test Cluster at localhost:9160.
        [cqlsh 3.1.8 | Cassandra 1.2.18 | CQL spec 3.0.5 | 
        Thrift protocol 19.36.2]
        Use HELP for help.
        cqlsh>
    

Shutting cassandra down:

        # find PID of a cassandra process

        $ ps -ef | grep cassandra

        # run kill -9 [the PID number ... i.e. 8212]
    

Running Cassandra on android multimedia player is fun :)

Arduino with MINIX NEO X5 mini

Intro

I received the "MINIX NEO X5 mini" as a gift some time ago. The TV set I have is pretty much filled with multimedia capabilities so having an android based entertainment system was not that appealing to me. I've spent a lot of time thinking how could I put it to better use so I decided to use it as a gathering node for my arduino projects. In order to use the device this way I had to root it. The process of rooting is described here. But be careful since the rooting scripts are not 100% correct. I had to edit some of them in order to make everything work. It's a trial and error. The remote coming with the device is not the best solution when it comes down to interfacing with the android. I strongly recommend using a mouse.

Linux Deploy

The original plan was to install some sort of linux on my "MINIX NEO X5 mini" but it actually turned out that "Linux Deploy" is a decent alternative because there is not a lot of messing around with the system and is really easy to install. Once you run the app you are going to see the IP address of the host system. That info is very useful later on. The icons are placed to the right side.

  • Start
  • Stop
  • Properties
  • Context

Before doing anything we have to setup the properties. The default install platform is Debian. Adapt the mirror URL according to your location. You can find the locations here, mine is for instance http://ftp.hr.debian.org/debian/. Under Settings I enabled autostart so that I don't have to run the application every time MINIX turns off. Once you are ready to install linux go to the properties again and press "Install". This will take around 20-30 minutes. When everything is finished you should see a success message. It's time to start the installation. All you have to do is to press the start button. I'll assume that the rest of the tutorial will be followed along from some other computer in terminal. In the top left corner you can see the IP address of the "MINIX NEO X5 mini". The installed system does not support serial communication out of the box because the modules for serial communication were removed from it. You can add the modules with the following commands:

        $ ssh android@IP_FROM_TOP_LEFT_CORNER
        android@IP_FROM_TOP_LEFT_CORNER's password:

        [type in the default password: changeme]

        Linux localhost 3.0.36+ #172 SMP PREEMPT armv7l
        Debian GNU/Linux 7 (wheezy) [running on Android via Linux Deploy]

        ~$ mkdir temp
        ~$ cd temp

        ~/temp$ wget http://www.tatsch-it.de/wp-content/uploads/2014/06/cdc-acm.ko_.zip
        ~/temp$ sudo apt-get install zip
        ~/temp$ unzip cdc-acm.ko_.zip
        ~/temp$ sudo insmod cdc-acm.ko
        ~/temp$ sudo mkdir /lib/modules
        ~/temp$ sudo mkdir /lib/modules/`uname -r`
        ~/temp$ cd /lib/modules/`uname -r`

        3.0.36+$ sudo mkdir -p drivers/usb/class
        3.0.36+$ sudo mv ~/temp/cdc-acm.ko /lib/modules/3.0.36+/drivers/usb/class/cdc-acm.ko
        3.0.36+$ sudo touch /lib/modules/3.0.36+/modules.order
        3.0.36+$ sudo touch /lib/modules/3.0.36+/modules.builtin
        3.0.36+$ sudo depmod -a
        3.0.36+$ sudo nano /etc/modules

        # add a line "cdc-acm" to the file
        # don't do it in a line starting with "#"
    

After doing the previous steps go back to the the Linux Deploy and press STOP. Wait for the system to shut down and then press START again. Now go back to the shell and reconnect to the linux deploy and check if the module was loaded after the restart it should look something like described in the following section:

        $ ssh android@IP_FROM_TOP_LEFT_CORNER
        android@IP_FROM_TOP_LEFT_CORNER's password:

        [type in the default password: changeme]

        Linux localhost 3.0.36+ #172 SMP PREEMPT armv7l
        Debian GNU/Linux 7 (wheezy) [running on Android via Linux Deploy]

        ~$ sudo dmesg | grep cdc_acm
        [ 1238.018779] usbcore: registered new interface driver cdc_acm
        [ 1238.018827] cdc_acm: USB Abstract Control Model driver for 
        USB modems and ISDN adapters
    

Arduino

We won't do anything too complicated with the arduino. At the moment all we are interested in is the serial communication, so we will write a simple hello world level sketch and upload it to our arduino:

        int counter;

        void setup(){
          Serial.begin(9600);
          counter = 0;
        }

        void loop(){
          counter++;
          
          Serial.print("Hello from android ");
          Serial.print(counter);
          Serial.println();
           
          delay(1000);
        }
    

Upload the sketch and go to tools > Serial Monitor. You should see something like:

        Hello from android 1
        Hello from android 2
        Hello from android 3
        ...
    

Back to the Linux Deploy

        $ ssh android@IP_FROM_TOP_LEFT_CORNER
        android@IP_FROM_TOP_LEFT_CORNER's password:

        [type in the default password: changeme]

        Linux localhost 3.0.36+ #172 SMP PREEMPT armv7l
        Debian GNU/Linux 7 (wheezy) [running on Android via Linux Deploy]

        ~$ ls /dev/tty*0
        # you should see /dev/ttyACM0 
        # in the list after you plugin the arduino to the MINIX

        # let's check what the arduino is sending us

        ~$ cat /dev/ttyACM0
        Hello from android 1
        Hello from android 2
        Hello from android 3
        Hello from android 4
        ^C
    

Your "MINIX NEO X5 mini" can now communicate with the Arduino over the serial port. Just imagine the things the MINIX is now capable of :)

RedEye Sleep Toggle(Free)

If you leave your MINIX on for too long, It might go into some kind of a sleep mode. To prevent this I installed the RedEye Sleep Toggle(Free) and set it up to start when the MINIX is started. Enjoy!

2014-09-12

Pentaho reports with Java API

Intro

I found very little documentation with examples about how to use pentaho reporting API in Java. Most of the time I had trouble in establishing dependencies necessary to get the pentaho reporting to generate the reports out from the designer generated files with a ".prpt" extension. The story got even more complicated when I tried generating the graphs. The exceptions I encountered had very little solutions available on-line. I've decided for the 5.0.3 version because there are couple of books available that describe how to use that specific version. Moreover gradle seems to be very popular now-days so I decided to list the dependencies with it.

Dependency list

You will find most of the dependencies available on the pentaho repository. It might come as a surprise to you but there are also a lot of other dependencies that have to be included. JFreeChart is the first.

  • "org.jfree:jfreechart:1.0.17"
  • "pentaho-reporting-engine:pentaho-reporting-engine-classic-core:5.0.3"
  • "pentaho-reporting-engine:pentaho-reporting-engine-classic-extensions:5.0.3"
  • "pentaho-reporting-engine:pentaho-reporting-engine-classic-extensions-xpath:5.0.3"
  • "pentaho-reporting-engine:pentaho-reporting-engine-legacy-charts:5.0.3"
  • "pentaho-library:libloader:5.0.3"
  • "pentaho-library:libxml:5.0.3"
  • "pentaho-library:libserializer:5.0.3"
  • "pentaho-library:libformula:5.0.3"
  • "pentaho-library:libfonts:5.0.3"
  • "pentaho-library:libformat:5.0.3"
  • "pentaho-library:libdocbundle:5.0.3"
  • "pentaho-library:libswing:5.0.3"
  • "pentaho-library:flute:5.0.3"
  • "pentaho-library:libcss:5.0.3"
  • "pentaho-library:libpixie:5.0.3"
  • "pentaho-library:libsparkline:5.0.3"
  • "pentaho-library:libbase:5.0.3"
  • "pentaho-library:librepository:5.0.3"
  • "com.lowagie:itext:2.1.7"
  • "com.lowagie:itext-rtf:2.1.7"
  • "org.apache.poi:poi:3.9"
  • "org.apache.poi:poi-ooxml:3.9"
  • "bsf:bsf:2.4.0"
  • "rhino:js:1.6R7"
I hope this list will save somebody some time. It took me a while to compile it. The last one was a particularly time intensive to find. I found It thanks to the following post: http://forums.pentaho.com/archive/index.php/t-152813.html


Generating reports from .prpt files

If you ever come across this text you probably won't have trouble loading the MasterReport from the file system or some similar source. In my humble opinion the most complicated output format is actually the HTML. In order to get everything to work I had to extend the HtmlPrinter class. Other methods of including graphs were not an option so I had to include the images as Base64

Extending the HtmlPrinter:

        public class YourSuperPrinter class extends HtmlPrinter {

        // constructor
        super(resourceManager);

        // override print
        super.print(logicalPageKey, logicalPage, contentProducer,
            metaData, incremental);

        // we'll need convert the Image to BufferedImage
        private static BufferedImage toBufferedImage(Image img) {
            if (img instanceof BufferedImage) {
                return (BufferedImage) img;
            }

            BufferedImage bimage = new BufferedImage(
                img.getWidth(null), img.getHeight(null),
                BufferedImage.TYPE_INT_ARGB);

            Graphics2D bGr = bimage.createGraphics();
            bGr.drawImage(img, 0, 0, null);
            bGr.dispose();

            return bimage;
        }

        // override writeImage
        if (image instanceof DefaultImageReference) {
            DefaultImageReference dir = (DefaultImageReference) image;

            if (dir.getImage() != null && encoderType != null) {
                BufferedImage img = toBufferedImage(dir.getImage());

                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ImageIO.write(img, 
                    encoderType.substring(encoderType.indexOf("/") + 1),
                    baos);
                baos.flush();
                byte[] imageInByte = baos.toByteArray();

                String imageData = Base64.encode(imageInByte);

                return "data:" + encoderType + ";base64," + imageData;
            }
            else {
                return dir.getSourceURLString();
            }
        }

        return super.writeImage(image, encoderType, quality, alpha);
    

Now that you have your Super printer you can generate html:

        MasterReport rep;
        AbstractReportProcessor reportProcessor;

        // BEGIN HTML specific
        StreamRepository tp = new StreamRepository(outputStream);
        ContentLocation tr = tp.getRoot();
        HtmlOutputProcessor op = new StreamHtmlOutputProcessor(
            rep.getConfiguration());
        HtmlPrinter printer = new YourSuperPrinter(
            rep.getResourceManager());

        printer.setContentWriter(tr, 
            new DefaultNameGenerator(tr, "index", "html"));
        printer.setDataWriter(null, null);
        printer.setUrlRewriter(new FileSystemURLRewriter());

        op.setPrinter(printer);

        reportProcessor = new StreamReportProcessor(rep, op);
        // END HTML specific

        reportProcessor.processReport();

        outputStream.close();
        outputStream.flush();
    

If everything went fine you should be generating super graphics and charts in your reports.

Let's mention other report formats together with their specifics:

Excel

        FlowExcelOutputProcessor target = new FlowExcelOutputProcessor(
            rep.getConfiguration(), outputStream, 
            rep.getResourceManager());
        reportProcessor = new FlowReportProcessor(rep, target);

        // if you want to download the file
        response.setContentType("application/vnd.ms-excel");
        response.addProperty(
            "Content-Disposition", "attachment; filename=hello.xls");
    

PDF

        PdfOutputProcessor outputProcessor = new PdfOutputProcessor(
            rep.getConfiguration(), outputStream, 
            rep.getResourceManager());
        reportProcessor = new PageableReportProcessor(
            rep, outputProcessor);

        // if you want to download the file
        response.setContentType("application/pdf");
        response.addProperty(
            "Content-Disposition", "attachment; filename=hello.pdf");
    

CSV

        StreamCSVOutputProcessor outputProcessor = 
            new StreamCSVOutputProcessor(outputStream);
        reportProcessor = new StreamReportProcessor(rep, outputProcessor);

        // if you want to download the file
        response.setContentType("text/csv");
        response.addProperty(
            "Content-Disposition", "attachment; filename=hello.csv");
    

RTF

        StreamRTFOutputProcessor outputProcessor = 
            new StreamRTFOutputProcessor(
            rep.getConfiguration(), outputStream, 
            rep.getResourceManager());
        reportProcessor = new StreamReportProcessor(rep,
            outputProcessor);

        // if you want to download the file
        response.setContentType("application/rtf");
        response.addProperty(
            "Content-Disposition", "attachment; filename=hello.rtf");
    

XML

        // no .processReport here, just stream close and flush ...
        XmlTableReportUtil.createStreamXML(report, outputStream);
        outputStream.close();
        outputStream.flush();

        // if you want to download the file
        response.setContentType("application/xml");
        response.addProperty(
            "Content-Disposition", "attachment; filename=hello.xml");
    

TEXT

        // no .processReport here, just stream close and flush ...
        PlainTextReportUtil.createPlainText(report, outputStream);
        outputStream.close();
        outputStream.flush();

        // if you want to download the file
        response.setContentType("application/text");
        response.addProperty(
            "Content-Disposition", "attachment; filename=hello.txt");
    

I hope this post will help somebody, It took me a while to figure all this out.

2014-04-20

Cassandra with Node.js and Arduino

Intro

This post continues where this post stopped. The Cassandra setup used for this post is more or less the same so please read this post if you are interested in cassandra setup before continuing with the rest of the post.

Arduino

Learning big data stuff is most exciting when the data represents something from the real world and not something generated with a help of big loop and then randomized data in it. To create data for this example I've used the following components:

  1. arduino uno
  2. Photoresistor GL5528 LDR
  3. 10K OHM NTC Thermistor 5mm
  4. 2x 10k resistor
  5. Protoboard
  6. Wires
Couple of this inexpensive components combined with arduino give us a nice big data sensor / generator. Now it might not seem that complicated but sampling any data at a one second level will hit on the cassandra limitations after one month of sampling if not done right, so having a simple arduino setup is fun and motivating way to tackle learning cassandra stuff. For now let's concentrate on the arduino part. The wiring is shown here:


The Arduino sketch will be on the gitHub, so we'll concentrate on the important parts. The light level in this example is read at analog 0. Reading analog values in arduino results in values ranging from 0-1023. We'll define light level as a mapping from 0-1023 into 0-100. Arduino already has a built in function for this called map. Also, I had some trouble in my initial experiments with Arduino serial communication and reading pin values. The data written to the serial port simply got corrupted after a while. I've read a couple of forums on this subject and found out that it actually helps when one delays execution after reading a pin value for 1ms. Also to keep the things as stable as possible we'll pause the execution for 1 second after writing to serial port as shown here:

  int light = map(analogRead(0), 0, 1023, 0, 100);
  delay(1);

    ....

  sprintf(sOut, "%d,%s", light, deblank(sTemp));

  Serial.println(sOut);
  delay(1000);
 


Node.js and Cassandra

Parsing the messages that come from the measuring devices is pretty repetitive stuff that causes pretty ugly code. I've learned that the hard way. To make parsing of this messages as easy as possible I've written a small utility package for parsing the messages that come from the measuring devices and it's available on npm.

Using serial ports in node.js doesn't take a lot of steps to setup:

  var serial = require( "serialport" );
  var SerialPort = serial.SerialPort;

  var portName = "/dev/tty.usb-something";

  var sp = new SerialPort(portName, {
      baudrate:9600,
      parser:serial.parsers.readline("\n")
  });

  sp.on("data", function ( data ) {
  var arduinoData = translator.parse(data);
  //...
 

To make the data handling easier and more in accordance with cassandra best practices the readings will be partitioned by date when they were recorded.

  CREATE TABLE room_data (
    day text,
    measurementtime timestamp,
    light int,
    temperature float,
    PRIMARY KEY (day, measurementtime)
  ) WITH CLUSTERING ORDER BY (measurementtime DESC);
 

Also the data will probably be more often fetched for recent time stamps with queries that have limits set on them. To make this fetching easier we've added a clustering statement above. Also to get the current light and temperature level we would just have to run the following query (no where combined with now function):

  SELECT * FROM room_data LIMIT 1;
 

After setting up the cassandra and reading the data from the serial port and parsing the data it's time to write this data into the cassandra. Analyzing the data and doing something useful with it will be in some future posts that I'll make but for now I'll stop with writing the data into cassandra:

  client.execute('INSERT INTO room_data ' + 
   '(day, measurementtime, light, temperature)' + 
   ' VALUES (?, dateof(now()), ?, ?)',
   [
    moment().format('YYYY-MM-DD'),
    arduinoData.light,
    arduinoData.temperature
   ],
   function(err, result) {
    if (err) {
     console.log('insert failed', err);
    }
   }
  );
 

On the fifth line I've used moment.js to format current time into string representation of current date used for partitioning in cassandra. The rest of the code is pretty much the usual sql stuff found in other database environments.

I recorder couple of hours worth of data here. Just in case anybody wants a sneak peak without having to setup everything up. I've exported the data out from cassandra trought cql using this command:

  COPY room_data (day, measurementtime, light, temperature) 
   TO 'room_data.csv';
 

The rest of the example is located on gitHub.

2014-01-11

Hello Cassandra in node.js

Intro

Since I started to work in a team that deals with BigData stuff I came into contact with Apache Cassandra. After years in the relational world it took me some getting used to the many concepts that the Cassandra relies on. Actually in the relational world the concepts would be heavy anti patterns. I went over a couple of tutorials etc. for intro into the Cassandra data model I would recommend this video by Patrick McFadin:

C* Summit 2013: The World's Next Top Data Model


Basic setup

The easiest way to get the Cassandra is to download it from here: http://planetcassandra.org/Download/StartDownload

I somehow dislike when various applications write to /var/something and having to use the root access to install something unless it's absolutely necessary. So I followed this manual to avoid this problem.


cassandra.yaml

The Cassandra is setup out of the box to support queries coming from cql shell ("cqlsh"). The goal of this blog entry is to show how to make a simple connection from node.js to the Cassandra, so there is a bit of tweaking that has to be done in order to get all this working. The necessary configuration is located in this file:

 install_dir/conf/cassandra.yaml
The properties I had to change were (basically this allows logging in with users other than default):
 authenticator: PasswordAuthenticator
 authorizer: CassandraAuthorizer
After that going into bin directory and running cqlsh will require username & password
 ./cqlsh -u cassandra -p cassandra

Cassandra keyspace setup

 CREATE KEYSPACE test WITH REPLICATION = 
  {'class': 'SimpleStrategy', 'replication_factor': 1};

 --check if it's created with this
 DESCRIBE KEYSPACES;

 USE test;

 CREATE TABLE test_table (
  id text,
  test_value text,
  PRIMARY KEY (id)
 );

 INSERT INTO test_table (id, test_value) VALUES ('1', 'a');

 INSERT INTO test_table (id, test_value) VALUES ('2', 'b');

 INSERT INTO test_table (id, test_value) VALUES ('3', 'c');

 SELECT * FROM test_table;
 
If everything is o.k. you should see something like:
  id  | test_value
 ----+------------
   3 |          c
   2 |          b
   1 |          a

 (3 rows)
Add a testuser to make the hello world example work:
  create user testuser with password 'testuser';

  grant all on test.test_table to testuser;

node-cassandra-cql

I tried several Cassandra connection libraries from gitHub for the node.js and the one that I found most easy to work with (and setup) was node-cassandra-cql by jorgebay. The story with the project is pretty much standard. Going into new project empty directory and initializing it with init and then installing module with npm.

 npm init

 npm install node-cassandra-cql

 #copy hellocassandra.js from
 #https://github.com/msval/hellocassandrainnodejs

 node hellocassandra.js
 

Anyway here's my example on gitHub.