Snippets of Smidgens

MQSH - An SSH alternative with MQTT

Skip to the code: https://github.com/franklovecchio/mqsh

I got fed up with the Java implementations of SSH after this StackOverflow ticket went (mostly) unanswered; since I had been playing around with the idea of another (better) way to invoke commands remotely, I settled on using MQtt as the middle-man.  In an ideal world, I wouldn’t need to execute direct commands to EC2 instances remotely — I could just just an AWS API call…or something.  Oh, well.  

 Why MQtt and not 0MQ?  Aside from v3.1 basic authentication, I like the idea of being able to log, via the broker, any commands sent + what client they were initiated from.* Also, since MQtt has quality of service levels to insure a message was sent and received, it at least ensures the command will be executed on the remote instance.  With AWS and their wonderful security group implementation, it’s very easy to only allow commands from specific instances, public IPs, or other security groups — why bother managing keys when there is a friendly API to manage security :)  And finally, there’s the idea of parallel-ssh; I frequently use the linked command-line tool for managing multiple instances, and using a broker-layer gives applications the same power by having multiple instances subscribed on the same control topic!

You’ll notice that the code examples on Github are half in Java, and half in Javascript; this is because I originally wrote MQsh as a plugin for node-monitor, my Node.js AWS CloudWatch application for monitoring EC2 instances.  Since I’m currently working on a private version of that repo, and most of you won’t have have brokers setup, I figured I’d throw in a middle-man example so the demo is self-contained (mqttjs and mqtt-client are bad-ass).  Once the plugin is done, I’ll be able to trigger CloudWatch alarms if a command failed…pretty cool :)

* It would make sense to create CloudWatch alarms based on these events as well.

Since the code examples + demo path on Github are pretty self-explanatory, no “snippets” here!

Twilio Pusher with Websockets & ZeroMQ

Recently, while building out a web application for RingDNA, we ran into a couple issues regarding websockets:

1) Amazon EC2 load-balancers do not support websocket scaling (yet).  I knew this, but had never implemented a solution.

2) Twilio applications point to webservice URLs, so developing locally can be tedious if you are building UI from call parameters that get pushed via websockets (and don’t want to fake data, interaction, or tunnel).

Both of these problems were solved via ØMQ, though the former won’t be elaborated on in this post.  Since I already had a basic Play! Framework/Twilio/websocket project for development purposes — twilio-push-play (clever, I know…’twas better than “play-with-twilio”) — it made sense to just build on it.  The application makes outbound calls via the Twilio Client and pushes call parameters to browser/socket; simply enter in the your credentials + phone numbers and you’re placing browser calls!  The application knows the difference between a local environment and a cloud “production” environment, and configures ØMQ connection accordingly (cloud app is publisher, local app is subscriber). There is a script for setting up the EC2 application, and for the local instance you just need to specify the EC2 application url (no http://) in the application.conf file.  The local environment is entirely independent of its production counterpart — we want to be able to use local logic with live data!

While the docs are minimal (I have real work to do, ya know?), setting up ØMQ can be kind of painful regarding classpaths; if you run into an issue with the lib included in the repository, you may have to copy the jzmq dependency you built on each machine into twilio-push-play (or better yet, mavenize it).  The EC2 script I talked about earlier and is based on the EBS Elestic Ubuntu AMI ami-a562a9cc:

If you’re using a Mac, you can follow these instructions to setup ØMQ locally.  I had to build jzmq and copy the jar into my twilio-push-play/libs directory after everything was said and done.

Let’s see how I integrated ØMQ!  Since Play! allows for asynchronous threads to be run on application start; this is where we figure out if the app is running on EC2 or not, and either start the ØMQ subscriber or publisher:

If we’re on EC2, we create a publisher.  I kept state the same way I did for websocket connections:

Because the subscriber is blocking, it needs its own thread:

If we’re not on an EC2 box, we subscribe to the publisher at the endpoint we specified earlier:

When the call parameters get pushed to the subscriber, it executes the same model the webservice URL Twilio does!  

MQDB - Distributing Databases with MQtt

Lately (as in the past 6 months), I’ve had a lot of “oddball” ideas regarding pubsub protocols and what can be done with them.  Aside from this blog, I’m working on a Cassandra pubsub implementation using MQtt and websockets, as well as a basic Twilio Client demo that uses pubsub via VOIP/SMS (in case you were wondering, the platform we just introduced last week - m2m.io - already has this packaged up and authenticated!).  I’ve found the possibilites for any of the MQ protocols to be (nearly) limitless!  After all, isn’t real life much like MQ in terms of exchanges? (that was rhetorical…it is, bitches) 

Happenstance

The idea for distributing databases with MQtt - from noSQL to SQL - came to me while drinking Jack Daniels and enjoying a late-summer sunset in San Luis Valley (wearing cowboy boots, no less).  I believe Netflix said it best, (and I’m paraphrasing here) “different databases for different data”.  At my former employer, we used Cassandra, SimpleDB, and SQLite for various architectures and needs.  While we did have a connection between them, it was API based — I wanted more.  Then, I started playing around with Redis in my spare time.

Trials and Tribulations of the DB Kind

When I needed a one-off storage mechanism for a project on small EC2 instance, say a project like sabisushi.com, Redis started to become my go-to datastore (in a non-distributed environment) because of its weight (Cassandra is a bit heavy).  Somewhere in between experimenting Amazon introduced Elastic Cache, and it spawned my first failed idea: a client-side distributed app in Node.js which emulated Elastic Cache using the Redis PubSub implementation.  The main flaw?  One application (in a team of distributed applications, presumably) would always have to be online.  So, I took another stab at it…and ended up with this: mqdb (written in Java). Let’s go into detail…

Cassandra as a Template

I thought to myself, “What if I follow a similar Cassandra distribution pattern and use MQtt to connect them?”.  MQtt, in terms of speed, is freaky fast — like Android MQtt client -> accelerometer -> 3G -> DB in realtime freaky fast, so I wasn’t concerned there (though I probably wouldn’t hit anywhere near the throughput of Redis’ capabilities?).  What I came up with is obviously much more basic in terms of how data is replicated, but the effect is somewhat the same: a distributed, replicated database from a simpler architecture (Redis was meant to be master/slave), and with no single point of failure (the m2m.io broker(s) are scaling + distributed themselves).  The final product consists of two separate applications; the first app stands in front of the Redis nodes. For the Redis connection, I used the Jedis Java client and abstracted the MQtt connection out (though not fully…yet).  The second app is a demo which would has code that would be implemented into whatever your needs are (say a distributed application of sorts).

Tokening

I emulated a basic form of Cassandra tokening (I’m ignoring numeric key/set names), and it’s probably best explained like so: let’s say you have 6 Redis databases on 6 different EC2 boxes and those are going to be your “cluster” (each EC2 box is considered a node).  When you initially stand up the MQtt application in front of each node, you have a replication factor (RF) of 6 — each piece of data you write from your distributed app hits all 6 nodes.  Now, 6 might be over-doing it; maybe you want a specific portion of the written data to be stored on nodes 1, 2, and 5 (key or set names from a-m), and another to hit nodes 2, 4, and 6 (key or set names from n-z); this would give you a RF of 3…so to speak.  You could also split up the data in a more random fashion based on hash of key/set name and token the nodes based on hashes, but we’re keeping it simple (and the latter example is ideal).  When you want to tell a node to change the way it stores data, you simply publish on a specific m2m.io space (which are namespaces in MQtt terms) with a payload that relates to the new ranges (note: this does not make the Redis node fetch the data from other nodes that match its new range — wayyy to deep for this example!).  

Settin’ Up the Application

I differentiated the m2m.io spaces for ease of understanding and use.  By default, each node subscribes on all stuff that relates to “redis”, and writes all data it receives.  The integrated application only subscribes on the space which the nodes return responses on (if they’ve been asked to fetch a key/set).  

DAO to the Max

While my example is using a Redis DAO, you could theoretically stand up clients in front of different databases and have a cross-database platform…think legacy + modern!  Very cool.

Models

Each MQtt payload is abstracted into a model for ease of use.  I have 3 models; 1 for data being written, 1 for management payloads (e.g. re-tokening node), and 1 for publishing data being read off the node.

MQDB, Redis Appplication

We’ve arrived at the application which stands in front of the Redis DB; it handles all the MQtt connections, payload management, DAOs, etc (the MQtt class needs to be abstracted for use, still — it currently just shows the methods I used).

MQDB, Integrated Application (+ demo)

This is an example of what you create do with a distributed application layer that needed a memcache or DB; I’ve added in a demo method which runs on initialization to show you a basic flow (it re-tokens a single node, writes to it, then reads back from it).  Oh, you should probably spin up an EC2 box with Redis on it first…here is the gist!

Inspiring The Internet Of Things: A Comic Book

Sabi 2.0

While most of the digital backstory regarding Sabi was exposed in a wonderful article written by the guys, I thought it was about time to elaborate on the technical aspects of what powers the ordering system, call queue, API, and datastore (but probably not in that order).

Amazon EC2 Hosting

When it comes to domain names, Sabi did what everybody else does — use GoDaddy.  I wanted to put up a cool backend, though, and I always feel like using a hosting service to host application frameworks is hindering.  So, I spun up a small ephemeral Ubuntu AMI and scripted it to clone the Sabi repository from GitHub. 

Redis Datastore / Amazon S3

I can barely remember MySQL; while it does have its place, I’m stuck in a NoSQL mind-set.  Besides Redis having really great Node.js bindings, it was an in-memory DB that fit the bill for Sabi’s needs perfectly (though I am an avid Cassandra user, this was not the place for something so heavy).  Sometime last week, I added a simple cron-job that pushes the DB to Amazon’s S3 nightly, just in case the cloud goes down :)  I’ve also been playing with a client-side distributed cache using Redis’ pub/sub…redis-distributed-cache

Express Framework

I have been in love with Node.js since I started reading Hacker News. I’ve written one really intense application with it — a scaling box monitor built on Websockets, REST, and NoSQL (node-monitor) — and in doing so, my love affair only blossomed. It’s really easy to get something prototyped quickly, and with my recent discovery of coding Javascript in Eclipse (HOLY!), the time-span was reduced five-fold!  It seemed the decision to use Node.js as the backend was a no-brainer, I just needed a framework. 

The Sabi API

Knowing that I needed a way to manage the order data, an intermediary API was born.  Since I’ve used (API) frameworks like Ruby’s Sinatra, Java’s Play!, and PHP’s CodeIgniter, I spent most of the first night drinking whiskey and reading up on Express.  And data modeling.  NoSQL is really all about putting data in the same way you want to get it out — I hashed out what they might need along with the current uses I dreamed up.  Whenever possible, I decided on storing a JSON value relating to a key.  It was relatively easy to implement a couple simple routes and basic auth, though Express’ documentation didn’t really help with the latter piece (StackOverflow to the rescue!).  When an order is placed, the Express code looks something like (please ignore my incorrect spelling of queue!):

The interaction between the website/admin setup and the API takes place with a simple jQuery .ajax call that references a PHP file that houses the username/password auth and static variables like e-mails, etc.  That same PHP file uses the Sabi PHP SDK, which looks something like:

The Website (or supporting Internet Explorer)

Web design is a mother-f*cker. The guys gave me an initial image, and I built from there.  Wherever possible, I used @font-face tags in place of images, and tried my best to avoid style tags (sigh, there are some still lingering in the forms section).  In the process of trying to support at least IE8+, the most useful tool I found was augment.js, a javascript snippet that makes IE aware of certain prototype objects!  The page obviously uses lots of jQuery, and I even pull some of the information from the DB (like menu items, to start).  When you add or remove items, you should really be thinking:

Admin Console

The admin console is built on one principle - realtime.  While I’m sure they exist, I’ve never seen a point-of-sale system built with websockets…why not try?  Since I could force the guys to use Google’s Chrome browser (which just recently caused headaches in another project), I could guarantee websocket support.  Since Express integrates rather easily with node-websocket-server, everytime anything useful happens on the API side, I push to an administration console that is displayed on computers in the kitchen.  On page open (and websocket open), I do calls via a websocket like “get queued orders”, “get customers”, and other associated data.  I was able to bypass jQuery .ajax calls by creating a simple websocket API; e.g. JSON recognized switch statements from browser to the websocket server running in conjunction with Express.  I didn’t have time to play with the Express templating system, but that’s something I can get to in the future.  

Call System

Even though there is a super-simple online ordering system, some people still like to call in their orders, and we don’t support IE 6 (OH THE HUMANITY).  They received 80+ calls one day in the first week, so I implemented a call queueing system using Twilio.  At its most basic level, when a call comes in, the caller is redirected to the kitchen phone.  If nobody answers in 10 seconds or so, the caller is put on hold for 5 minutes.  They are told they can press 1 at any time to leave a voicemail order, or hold. 

The cool part — websocket integration + analytics.  When a call initially comes in, it shows up in a queue displayed on the admin console by pushing Twilio data through the websocket, and also increments calls by creating a URL for Google Analytics:

Whoever is manning (or womaning) the computer can direct the call to another Sabi associate by selecting them from a dropdown, or they could obviously answer it.  If the caller leaves a voicemail order, we push some data through the websocket and render an HTML5 audio tag next to the ended call.  

The people manning the computers have the ability to update orders via statues:  order finished in kitchen, order being delivered, success, failure, or never made.  When the order is “being delivered”, we send a text message notifying the customer.

The beauty of websockets is that we’re mostly setup for distributed POS now!

Building an OpenVBX Twilio AMI on EC2

A while back, I built an OpenVBX AMI for Isidorey: http://blog.isidorey.com/2011/06/twilio-openvbx-ami-update.html.  I’m just now getting to posting the script that sets up up OpenVBX and its dependencies.  I’d also like to add in some options to install plugins off-the-bat, as well as a Redis backend instead of MySQL :)  

I’m assuming a fresh installation here, and that further down the road you will have content in your /var/www directory with an .htaccess file of your choosing.  I place OpenVBX at /calls because that is the simplest form of integration I could think of (clients don’t care what they’re using, as long as it works!)

There’s really not much to it — this would be run as a user data file on init of an EC2 instance; I forgot how to install Pear silently (somewhere along the way the script changed…sigh).  I’ll dig into that when I have time!

APIGEE API Documentation Generator

UPDATE:  I broke this because I was including css/js from another box that died (so was the Isidorey link) :)  Luckily, I have everything in S3 — I just need to take the time and put the demo back together, but at least the code is available.

Let’s talk APIGEE for a Colorado-second; it’s an awesome API tool I’ve come to love for numerous reasons — the most obvious being the ease-of-use it brings to deploying new features [and it seems like we do that quite a bit at Isidorey].  One of the things that seemed necessary, at least to cut down on the time I spent documenting [who am I kidding, I pushed as much of that off to the interns as I could :)], was being able to generate all of the API documentation from the XML [wadl] file that APIGEE uses. 

There are still some nuances to hash out, but I’ve updated the generator to use Twitter’s new Bootstrap template [see live demo below].  It’s written in Java and uses the Play Framework to render the ‘DOCUMENTATION’ call.  All of the code is on github!

How it works:

I use a custom XML file that has <description> tags for describing segment and header parameters, and I generate the webpage using the custom file while also generating an APIGEE-supported file by SED-removing the tags into another XML file.  While it looks like APIGEE supports a <doc> tag for header parameter descriptions, I currently see no way to document URL, or segment based parameters, and lots of new frameworks (Codeigniter, Play, etc) are all about this!

Since the Isidorey API is really a grouping of APIs, and APIGEE does not support tags for grouping them, I’ve added in some optional code to make separate drop-down menus out of each one.  Bootstrap still has some CSS issues and does not wrap the menus, so this is probably not a good solution for larger APIs with many calls (I think I can hack it to scroll, though).  

The live demo:

http://ec2-67-202-48-67.compute-1.amazonaws.com:9000/DOCUMENTATION

The Isidorey demo, which pulls the XML file from S3! (you must allow the deps from non-SSL box):

https://api.isidorey.net/v0.9/DOCUMENTATION

The future:

While I’d like to see these features built-in to APIGEE, I’m going to add tabs for responses using the custom tags, e.g. failure and success examples.

The code:  …has disappeared.