frustration with Interface programming

One of the hardest parts for me.....

After writing quite a bit of code, both for the arduinos and the python to lace things together, one of the things that I am dreading is writing the interface to bring it together. I always seem to have these grandiose plans on what I want to do, as opposed to what I can do, and time to do it.

So I sat down and did some research. I know what Id love to do, but I have to be practical in what I need to start off with. I have decided to stick with the web interface, primarily though Im not a huge fan of HTML, I have to admit that diving into HTML5 has changed my opinion of things somewhat. I am now a huge fan of using canvases. I simply love the work that Gerrit Grunwald has done using JavaFX, and even some of the work that he has done using javascript and his SteelSeries of meters, LCD readouts and other things with his own libraries. Id LOVE to incorporate all of that into a java app, but that going to require me to study even more code and procedures. SO.....for now Im going to stick with HTML5, and some simple javascript.

Im wanting to make a page, where I can quickly look at the thing smonitored and get a handle on whats doing what. I want to make a web page for controlling things, and having updated statues, but also a " Engineering page" where I can look at temperatures, and graphs, something that might not look " pretty" but gets the job done. There are several open source free jsp libraries out there that can be used. I found one I liked called RGraph. Its a simple yet pretty robust javascript graphing library set that doesnt have a huge learning curve, and actually if you dont mind tinkering with it, is kinda fun to play with.....

So I started the coding process after learning more about canvas, with HTML5....its pretty easy and a lot of fun to mess with.  Here is a snipplet of the code...........

       $(document).ready(function ()
        {
            var gaugeEXCH = new RGraph.Gauge('cvs', 0, 1000, 300)
                        .set('centerx', 425)
.set('centery', 225)
.set('radius' , 100)
.set('title.top', 'EXCH')
       .set('title.top.size', 10)
       .set('tickmarks.big',5)
       .set('title.bottom', 'Degrees')
                        .set('title.bottom.color', '#fff')
       .set('title.bottom.size', 8)
       .set('background.color', '#00008B')
       .set('background.gradient', 'True')
       .set('border.outer', '#2F4F4F')
       .set('tickmarks.small.color' , 'FFF')  
       .set('tickmarks.big.color' , 'FFF')
       .set('text.color' , 'FFF')
       .set('title.top.color', 'FFF')
       .set('border.inner', '#FF0000')
       .set('centerpin.color', 'gray')
       .set('needle.type' , 'triangle')
       .set('border.gradient' , 'True')
        .draw();
       
            var gaugeHCORE = new RGraph.Gauge('cvs', 0,150,45)
                        .set('centerx', 650)
                        .set('centery', 225)
.set('radius' , 100)
   .set('title.top', 'H-CORE')
       .set('title.top.size', 10)
       .set('tickmarks.big',5)
       .set('title.bottom', 'Degrees')
                        .set('title.bottom.color', '#fff')
       .set('title.bottom.size', 8)
       .set('background.color', '#2F4F4F ')
       .set('background.gradient', 'True')
       .set('border.outer', '#2F4F4F')
       .set('tickmarks.small.color' , 'FFF')  
       .set('tickmarks.big.color' , 'FFF')
       .set('text.color' , 'FFF')
       .set('title.top.color', 'FFF')
       .set('border.inner', '#FF0000')
       .set('centerpin.color', 'gray')
       .set('needle.type' , 'triangle')
       .set('border.gradient' , 'True')
 
 
.draw();


var gaugeSOLAR = new RGraph.Gauge('cvs', 0,200,13)
                        .set('centerx', 875)
                        .set('centery', 225)
.set('radius' , 100)
   .set('title.top', 'SOLAR')
       .set('title.top.size', 10)
       .set('tickmarks.big',5)
       .set('title.bottom', 'Degrees')
                        .set('title.bottom.color', '#fff')
       .set('title.bottom.size', 8)
       .set('background.color', '#9400D3 ')
       .set('background.gradient', 'True')
       .set('border.outer', '#2F4F4F')
       .set('tickmarks.small.color' , 'FFF')  
       .set('tickmarks.big.color' , 'FFF')
       .set('text.color' , 'FFF')
       .set('title.top.color', 'FFF')
       .set('border.inner', '#FF0000')
       .set('centerpin.color', 'gray')
       .set('needle.type' , 'triangle')
       .set('border.gradient' , 'True')
.draw();

var gaugeGEO = new RGraph.Gauge('cvs', 50, 70, 53)


I also found that i have a new best friend when it comes to plain old text editors. I find its easier than trying to use something like a free HTML editor, but I might give one a try, its Notepad++ version 6.4. With the added plug-ins, its really easy to use, but functional.

The code snipplet is just for a couple of meters. Im making a temperature monitoring page with in the Engineering page. Once you figure out the modifications you can make, putting meters on a web page is fun. Im also looking into the way to get as close to "real time" data as I can to be fed to them. So the learning continues. Im almost to the point that I can start mocking up the wiring, and installing the arduinos around the house. In the mean time, the programming continues. The snipplet gives you a page similiar to this one, which is obviously a work in progress.....

In the meantime, between school, kids track practice, and work the coding continues

More to come!

Lots to catch up on...........

I know its been a while since the last post, Im semi-good at these blogs. I have a tendency to get started and then start working on things and then next thing I know its been 2 months...Ill have to do better....or have my ADHD looked at.....

When I last left off, I had closed Version 3 and started on Version 4. Version 4 will be the same base as 3, but with some major modifications.

 1. All of the room nodes, will include temperature monitoring

 2. Major room nodes have been reduced. Instead of a room having several small nodes or MCU(arduinos) the redesign has used one big MCU ( Arduino MEGA) running several lights/sensors per room.

 3. MYSQL full integration. Tables broken down to include just the house, and another for the heating/cooling.

4. New script for check the online status of node, since the Pubsubclient doesnt have a way to use wills/testaments.

5. Dropped individual LCD screens for heating and cooling MCUs and now the nodes report status to the MQTT Broker. A Master MCU, subscribes to the topics, and provides a local display.

6. Looking into incorporating GhostEvents into the setup as a way to schedule events, as opposed to writing a lengthy trigger script I was using before.

7. Looking into ( I have downloaded and starting to configure) a software package called J.A.R.V.I.S, and as its name lends, its based on the Ironman AI Computer Jarvis from the movie. It has an excellent speech to text and voice recognition system, which will greatly reduce the workload,....if nothing else its fun to play with.

The major thing still is the web page design. For the initial testing phase, I simply used various tables set up and had real time data pushed to the tables. I want to totally depart from this. I have seen several good tutorials for using gauges, js, jqueryUI, and others that Im going to try.

I was able to get MYSQL connected to visual basic. I tinkered with going that direction, and making a visual basic windows form app, and just integrate into it what I wanted. When I started looking at adding graphical elements to visual basic, while some are excellent, very cool looking and not that hard to apply, when I saw the price tag for these my jaw dropped. Even for a non-commercial personal use, many started at 299 USD, thats simply not worth the ROI, since its just for me. So its back to using the webserver. This is totally fine with me. Id rather invest in hardware, that drop 300 dollars to make cool gauges on a VBapp.

Im still toying with the use of a full blown dashboard still, but that is going to require quite a bit of work. I have seen several example of these, and they are somewhat easier than I thought, so I might try.

Ok that about covers the last 10 weeks or so. I will start to blog on these parts in the coming weeks as time allows so stay tuned

Happy Coding!

Logic Layer Part2

Ok the Simple Logic layer script is just that pretty basic but as I mentioned earlier is serves as a place to make changes to the system without taking the system down. It basically a buffer between the nodes in the house and the main controller thats doing all the switching.....

So to better understand what I am doing this picture is a better representation

So for the House at least its not all that complicated really. In this blog we will be dealing with the LogicScript....

The Code:

 def on_message(mosq, obj, msg):
    ts =datetime.datetime.now().strftime("%B, %d %Y %I:%M%p")
   
    
    print ("---------------------------------------")
   
    print ("TIME RECEIVED: ")+ ts
    print ("FROM NODE: ") + (msg.topic)
    print ("NODE MSG: ") + (msg.payload)
   
 
    cli.publish('/HOUSE/Control/' , msg.payload) # <----- sends message to Controller (KIT1,1)
    print ("Message forwarded to Controller")
    print ("acknowledged @ ") + ts
    print
    cli.publish('/HOUSE/LogicClient/logicmsg/' , ' forwarded +( msg.payload)+ to Controller')
    print ("MSG update sent to webserver")
    
    if (msg.topic) == ("/ALARMS/Status"): # <-- leave off trailing / when comparing
      print (Fore.WHITE + Back.RED + Style.BRIGHT + ("STATUS: ALARM! "))
      print (Fore.RESET + Back.RESET + Style.RESET_ALL)
  
  # --------------------------------------PopUp Message------------------------------------
    subprocess.Popen(['notify-send', "System Message:",msg.payload])
    return              

    else:
     print ("STATUS: Normal")        
     print ("End of Event")
     print ("---------------------------------------")
     print
   
 
  def on_publish(mosq, obj, mid):
    pass
 

cli = mosquitto.Mosquitto("LogicClient")
cli.on_message = on_message
cli.on_publish = on_publish
cli.connect("192.168.30.4", 1883, 60)
cli.publish('/mainTopic/Announce/LogicClient/','Online')


cli.subscribe("/HOUSE/Kitchen/", 0)
cli.subscribe("/HOUSE/Livingroom/", 0)
cli.subscribe("/HOUSE/Bath/", 0)
cli.subscribe("/HOUSE/Bedrm1/", 0)
cli.subscribe("/HOUSE/Bedrm2/", 0)
cli.subscribe("/HOUSE/FrontPorch/", 0)
cli.subscribe("/HOUSE/Backporch/", 0)
cli.subscribe("/HOUSE/Garage1/", 0)
cli.subscribe("/HOUSE/Hallway/", 0)
cli.subscribe("/HOUSE/Office/", 0)
cli.subscribe("/HOUSE/Test/", 0)
cli.subscribe("/ALARMS/Status/" ,0)



while cli.loop() == 0:
    pass

The script itself is a loop of sorts. It is constantly listening to the network waiting to hear messages that are being published to topis it has subscribed to. If I have tio add or remove anything, then I can edit this script off line and then save and restart. Down time is next to zero.

The script is pretty self explanatory. The output is to a terminal windows which I can comment out once I get everything done. I like to see whats going on visually as I test and finalize parts of the system. The only funky thing really is the subprocess code. This is for my Ubuntu box, so it wont work on Windows. If there is an alarm detected, this causes my linux box to throw up a OSD for 10 secs in the upper right corner. Ubuntu users will know what I mean. Its a way to get my attention should a node post to the Alarms topic

Also since Im currently using a web page for my system to monitor and display statuses, the line cli.publish('/mainTopic/Announce/LogicClient/','Online')  posts to the webpage that this script is running. I still need to add a Last will and testament to this so if the script fails, then the MQTT server will post the testament message and Ill know something is wrong.

There are also some comments on here for me to remember. I found that the trailing "/" had to be ommited on the script or it would ignore messages published to topics....a "gotcha" I found out. 

This is a typical output from the script to the termimal. FP1,1 is the message that the front porch light has been turned on.

and this is an alarm event

So thats it for the Logic Script. It will evolve and refine as I progress but its a start amd it is working well right now in the house.

We will look at the Trigger script and the MySQL update script in future posts. In the meantime

Happy Coding!

Logic Layer....

As I posted in the last blog, I discovered the wonderful world of databases. Actually I had used them in the past, but not programmed anything such as queries or even created them. I had to laugh, cause back in the OLD OLD days, we didnt have databases. In fact if you wanted to store stuff say when running a BASIC program, you had to open a file, write to the file and close it.....sound familiar.....yeah that what we do now with databases.....history repeats itself...

As I mentioned in the last blog as well it became obvious that adding a logic layer just made sense. It gave me one place to modify the system without creating much in the way of downtime, actually none really and I could add or remove just about anything from the system as I wanted.

The Logic Layer is nothing more than a python script. I really do love python, its a powerful language and even just using IDLE, easy to program. I have not come across anything I thought of that I could not do in python, short of creating a GUI, which I would love to do, but thats more learning for later. The Logic layer script you can think of is a "front end" to the controller. Basically the logic script subscribes to ALL the topics on the MQTT server.....all of them. The output of this script is MANY publication to ONE topic. As of right now there are nearly two dozen different topics on the MQTT server, of which about 17 are house related. This will of course grow as the system does. The logic script then listens to all 17 topics, acts on payloads received, and issues payloads to ONE topic in return. This one topic is the ONLY topic the main controller arduino listens to. As I explained, this allows me a simple way to add and remove nodes from the system without ever touching the controller firmware.

The Script..........since Ill assume that you know how to use python, youll need to add some modules to python. I prefer to use pip, with my linux box. Its simply just less headache and easy to use, I like less head aches.....

The top of the script shows the modules youll need....

import mosquitto
import datetime

from colorama import init
from colorama import Fore, Back, Style
import sys
init()

print chr(27) + "[2J"
print
print
print (Fore.CYAN + Back.BLACK+  "    | MySQL DataBase Update Script |")
           
print

mosquitto module at least for ubuntu, can be gotten in the software center, I prefer synaptic, I even installed it back to my 12.10Ubuntu box after I installed ubuntu, its omitted in versions after 11.10 I think. I like synaptic, always have, and I dont have to deal with a bloated GUI of the software center. Since my production box doesnt have 40 cores, and 50GB of RAM, using synaptic is easier and faster. You can add it back to your ubuntu box by:

sudo apt-get install synaptic . Once you install it just pin it to the unity desktop bar

For Windows users, you can install all the python modules as you normally do. I work on both systems, however all my final work is done on a linux box.....

The module colorama, is totally unnecessary. Its  a fun module that allows you to change the fore ground, back ground and make the font bold, or dim in a terminal window. Since I like to, in production, use terminal window outputs to monitor the stability of a script when testing, adding color to represent certain conditions is cool. For instance, if I get an alarm, I can turn the output test to a bold white font, with a red background....catches your eye. Again, totally unnecessary, and you can ignore the code lines referring to it.

Datetime youll need to add...cool module that allows you to format time and date stamps. It totally worth reading or at least glancing over the documentation for datatime. This module came in real handy for the trigger script. Also it will match exactly the DATETIME setting for the time stamps in MySQL.....cool..

The rest of the top of this script is easy. print chr(27) + "[2J" does nothing more than clear the terminal window when the script is started, and then a script header. This just prints to th screen what script I am running, nothing more.

So now you know what modules to grab........next blog we will get into the nuts and bolts of the code...

Happy coding!

Wow time flies.....

Well after getting over the flu, FINALLY, I have been back at the coding for this winter. Here is a little of some of the additions I have been working on:

Version3 is done and put to bed. I decided on the protocol I am going to use from now on for all the MQTT topics, and subscriptions. I kept changing the way I was posting stuff to the MQTT server, through many revisons, it was time to decide on a standard.

  For Room nodes posting to the server I have decided on the standard of /SYSTEM/ROOM/node. Therefore if Kitchen light #1  in the house is activated, the the posted topic is /HOUSE/Kitchen/KIT1/. For the Engineering space. the all nodes post to /ENG/SYSTEM/Node. I know this seems silly maybe but I found out on version 3 that a small change one place creates a ripple effect cause you to change the published topics for nearly everything to reflect the change....SO Lesson learned when planning a system, decide on a protocol early on, stick with it.

Since version 3 is done, the improvements to version 4 included database integration. I HATE database admin stuff, but it soon because a necessary evil. I checked out many be decided of for the open souce database MySQL. I had to educate myself in the use, writing queries, and the proper creation of the database and the associated tables. In all reality, I had given databases a bad wrap, it was quite easy to learn and now I have one large database, with several tables.

The reason for database usage was quite simple. though the nodes can publish to the MQTT server, and other nodes can subscribe to topics and act on the published data, I needed places where nodes could pull current stored data. I am also working on what happens in the event of a power loss at the house. Its my intention that I will eventually be able to restore the house to the pre- loss of power state, once power is restored, by pulling the last data set out of the database. That part hasnt not been implemented yet, but will.

With the new found use of my database, it was necessary to update the table with new data in real time. A little funky for me really since I was relying on MQTT to do all the work. Also with using arduino based MQTT libraries, I discovered that there is no way to pass a "last will and testament" to the MQTT server.Wills arent not currently implemented in the latest MQTT libraries for arduinos.....SO what happens then if a node drops offline? The solution of course is time stamping the data, and then I can check the last time stamps against when I last heard the node....surpass a predetermined time frame, I have to assume the node is not communicating with the MQTT server.......

Finally I have decided on adding a couple of python scripts to the MQTT server. One of these scripts now acts as a logic layer. This script will also channel information between all of the room nodes in the house, to the Controller arduino thats actually switching lights on and off. I decided this was simpler. Adding this "logic layer" allows me to modify virtually anything, without the need to touch the Controller arduino. So for instance of I add something to the system, the logic script can be modified offline, saved and restarted as the new logic script. It provides one place to make changes in the system without the need to pull out the laptop, start up the arduino IDE and program the house control arduino. I will discuss this script in more detail in the next post.

I also added a second python script to the server computer. This script, called "trigger" merely is a simple script, basically running in an endless loop. It monitors the servers clock, and issues "triggers" based on time. So for instance, at 7:30pm I want the front porch light to come on, daily, I simply add the trigger to the script. If I was to modify the script, again I can do it offline, adding what I want, and then save it and restart the script to apply the new trigger. No need for additional RTC shields for arduinos, as all of the triggers fire off MQTT publications to the master controller arduino on the same topic the logic script does. Thus, at 7:30pm, the script will tell the controller to turn on the front porch light.......Ill discuss trigger script in more detail as well in a later post.

So I have been busy, and Ill start sharing more on the new revision in the near future.

In the meantime, happy coding!

I hate the flu

Well I started everything right at the end of 2013..........On January 1 2014, I was introduced to the H1N1 influenza virus, we didnt get along. For the next 2 weeks ( now 16 days) I have slowly worked my way through all the body aches, coughing, and lack of energy associated with the flu, I have determined that I will not skip another year without getting the flu shot. This has been terrible, to say the least.

I hope to continue with the blog in the next few days, as I start to get back to a normal routine