Andrew Mussey

Testing Hubot Scripts

Tue, 11 Aug 2015 16:31:06 +0000

Note: This guide currently works for Hubot versions up to 2.13.2. There are incompatibilities between the testing library and the async module introduced in Hubot version 2.14.0.

The Cloud Control Panel team uses a fork of Github’s Hubot for a large variety of tasks. Everything from our deployment pipeline to cross team communication is integrated with with our customized version of the IRC bot.

As the team’s script codebase for Hubot verged on 3000 lines of code, we began looking for a testing strategy. The official Github repo didn’t contain any sort of tests by default, so the team turned to third-party libraries.

Our initial interaction with testing came through the library hubot-mock-adapter. It appeared to be very in depth, covering a wide range of different scenarios and script types. However, I was deterred by the complexity of the tests. Most of our team spends very little time writing CoffeeScript, so I wanted the tests scripts to be as straightforward as possible.

I next turned to mtsmfm’s hubot-test-helper library. The library does an impressive job mocking out the basic functionality of the bot while still maintaining simplicity in the test cases.

The rest of this blog post covers some various testing cases as well as a couple pitfalls I ran across while building the tests. If you would prefer to go straight to the code repo, it is available on Github with instructions on running the tests: amussey/hubot-testing-boilerplate

Basic message and response

This is about as simple as it gets: a user says one thing, Hubot replies with another. For the following test, we’ll be using part of the ping.coffee script.

module.exports = (robot) ->
  robot.respond /PING$/i, (msg) ->
    msg.send "PONG"

To test this, we’ll do the following:

Helper = require('hubot-test-helper')
expect = require('chai').expect

# helper loads a specific script if it's a file
helper = new Helper('./../scripts/ping.coffee')

describe 'ping', ->
  room = null

  beforeEach ->
    # Set up the room before running the test
    room = helper.createRoom()

  afterEach ->
    # Tear it down after the test to free up the listener.
    room.destroy()

  context 'user says ping to hubot', ->
    beforeEach ->
      room.user.say 'alice', 'hubot PING'
      room.user.say 'bob',   'hubot PING'

    it 'should reply pong to user', ->
      expect(room.messages).to.eql [
        ['alice', 'hubot PING']
        ['hubot', 'PONG']
        ['bob',   'hubot PING']
        ['hubot', 'PONG']
      ]

As it can be seen above, messages can be sent into the room using the room.user.say function. The entire contents of the room can be read back using room.messages. Each message is returned as a list, with the first element being the name of the user, and the second element being the actual message that they sent.

When run, the output of this test should appear as follows:

  ping
    user says ping to hubot
      ✓ should reply pong to user

The full ping script can be found here, and the full ping test script can be found here.

Private Messages

Inside of secret.coffee, we can see a script with Hubot replying over private message.

module.exports = (robot) ->
  robot.respond /tell me a secret$/i, (msg) ->
    msg.sendPrivate 'whisper whisper whisper'

We can test that this private message was transmitted in the following way:

Helper = require('hubot-test-helper')
expect = require('chai').expect
helper = new Helper('./../scripts/secret.coffee')

describe 'secret', ->
  room = null

  beforeEach ->
    room = helper.createRoom()

  afterEach ->
    room.destroy()

  context 'user asks hubot for a secret', ->
    beforeEach ->
      room.user.say 'alice', '@hubot tell me a secret'

    it 'should not post to the public channel', ->
      expect(room.messages).to.eql [
        ['alice', '@hubot tell me a secret']
      ]

    it 'should private message user', ->
      expect(room.privateMessages).to.eql {
        'alice': [
          ['hubot', 'whisper whisper whisper']
        ]
      }

The naming on the two tests makes them relatively self-explanatory: a check to make sure that Hubot sent back the expected message (through room.privateMessages), and a check to make sure that Hubot did not post anything to the public channel. If it’s not apparent in the test above, Hubot stores all of the private messages it sends in room.privateMessages. The array keeps track of each message keyed by the username that the message was sent to. In this case, we only have the one key (alice), since she was the only one to receive a message.

The secret.coffee script and the secret.coffee test script are available on Github here and here, respectively.

  private-message
    user asks hubot for a secret
      ✓ should not post to the public channel
      ✓ should private message user

Updating the Brain

To show interaction with the brain, we’ll refer to the remember.coffee script. This script adds two commands to Hubot: hubot remember <text> which stores a provided string in the brain, and hubot memory, which will recall that string.

module.exports = (robot) ->
  robot.respond /remember (.*)$/i, (msg) ->
    robot.brain.data.memory = msg.match[1]
    msg.reply 'Okay, I\'ll remember that.'

  robot.respond /memory$/i, (msg) ->
    if not robot.brain.data.memory?
      robot.brain.data.memory = null

    if robot.brain.data.memory == null
      msg.reply 'I\'m not remembering anything.'
    else
      msg.reply robot.brain.data.memory

To test the contents of the brain, we can reach it by referencing room.robot.brain.data.*. So, two simple tests could be written as follows:

Helper = require('hubot-test-helper')
expect = require('chai').expect
helper = new Helper('./../scripts/remember.coffee')

describe 'remember', ->
  room = null

  beforeEach ->
    room = helper.createRoom()

  afterEach ->
    room.destroy()

  context 'user asks Hubot for memory contents', ->
    beforeEach ->
      room.robot.brain.data.memory = 'brain contents'
      room.user.say 'mary', 'hubot memory'

    it 'should reply with the contents of the memory', ->
      expect(room.messages).to.eql [
        ['mary', 'hubot memory']
        ['hubot', '@mary brain contents']
      ]

  context 'user asks Hubot to remember something', ->
    beforeEach ->
      room.user.say 'jim', 'hubot remember this'

    it 'should have the memory set to "this"', ->
      expect(room.robot.brain.data.memory).to.eql 'this'

These tests will check that the correct brain key is being read, and that the value is being set in the correct brain key.

  remember
    user asks Hubot for memory contents
      ✓ should reply with the contents of the memory
    user sets memory and asks for memory contents
      ✓ should have the memory set to "this"

A more complete test suite for this script can be found here.

Stubbing an object

When using third party libraries in a script, we will want to test the functionality of the script without relying on the third party libraries.

A quick and dirty example is set up below using the Moment.js library.

moment = require('moment')

module.exports = (robot) ->
  robot.respond /convert (.*)$/i, (msg) ->
    msg.send moment.unix(msg.match[1]).toString()

To test this, we’ll want to mock out both the moment.unix() call (which creates a moment object at the input timestamp) and the moment.unix().toString() call (which returns the moment object in the form of a text string). That way, regardless of changes to the library (or the timezone of the user), the function will output with consistency.

Helper = require('hubot-test-helper')
expect = require('chai').expect
assert = require('chai').assert
sinon = require('sinon')

# helper loads a specific script if it's a file
helper = new Helper('./../scripts/timestamp.coffee')

describe 'timestamp', ->
  room = null
  moment = null
  momentUnixStub = null
  momentUnixToStringStub = null

  beforeEach ->
    moment = require('moment')
    momentUnixToStringStub = sinon.stub()
    momentUnixToStringStub.returns("Sun Oct 16 2011 16:17:56 GMT+0000")
    momentUnixStub = sinon.stub moment, "unix", () ->
      return {toString: momentUnixToStringStub}

    room = helper.createRoom()

  afterEach ->
    moment.unix.restore()
    room.destroy()

  context 'user asks hubot to convert', ->
    beforeEach ->
      room.user.say 'jim', 'hubot convert 1318781876'

    it 'should echo message back', ->
      expect(room.messages).to.eql [
        ['jim', 'hubot convert 1318781876']
        ['hubot', 'Sun Oct 16 2011 16:17:56 GMT+0000']
      ]

    it 'should have called toString', ->
      expect(momentUnixToStringStub.callCount).to.eql 1

    it 'should have called unix() with the correct parameters', ->
      expect(momentUnixStub.args[0]).to.eql [ '1318781876' ]

Inside the beforeEach block, we create two stubs: one for moment.unix, the other for moment.unix.toString. The generated stubs are stored so they can be tested against, be it for call count or the parameters with which they were called.

  timestamp
    user asks hubot to convert
      ✓ should echo message back
      ✓ should have called toString
      ✓ should have called unix() with the correct parameters

The timestamp.coffee script can be found on Github here, and the timestamp.coffee test script can be found on Github here.

Mocking the Request object

Occasionally, there will be methods off of Hubot’s request object you’ll want to mock out. One of the biggest functions I found myself wanting to mock was the msg.random function. You can see a simplified version of the built in script, shipit.coffee, using the msg.random function below:

squirrels = [
  "https://img.skitch.com/20111026-r2wsngtu4jftwxmsytdke6arwd.png",
  "http://i.imgur.com/DPVM1.png",
  "https://dl.dropboxusercontent.com/u/602885/github/squirrelmobster.jpeg",
]

module.exports = (robot) ->
  robot.hear /ship\s*it/i, (msg) ->
    msg.send msg.random squirrels

In order to predictably test that a squirrel is being output, we need to set the msg.random to output something consistant. We can do this by mocking out the request object on the test Hubot.

Helper = require('hubot-test-helper')
expect = require('chai').expect
helper = new Helper('./../scripts/shipit.coffee')

class MockResponse extends Helper.Response
  random: (items) ->
    "http://i.imgur.com/DPVM1.png"


describe 'shipit', ->
  room = null

  beforeEach ->
    room = helper.createRoom({'response': MockResponse})

  afterEach ->
    room.destroy()

  context 'user says "ship it"', ->
    beforeEach ->
      room.user.say 'alice', 'ship it'

    it 'should respond with an image', ->
      expect(room.messages[1]).to.eql ['hubot', 'http://i.imgur.com/DPVM1.png']

By extending Helper.Response into MockResponse and redefining the random function, we can ensure consistent output of random while still maintaining the functionality of the rest of Responses functions. This custom Response object can then be pushed into our test Hubot when creating the room (room = helper.createRoom({'response': MockResponse})).

The rest of the test script for shipit.coffee can be found here.

  shipit
    user says "ship it"
      ✓ should respond with an image

Mock HTTP servers

In the event a script wants to communicate with the outside world, we’ll have to put an HTTP listener in place in order to mock out that communication. A simple example is the pug me script, pugme.coffee:

module.exports = (robot) ->

  robot.respond /pug me/i, (msg) ->
    msg.http("http://pugme.herokuapp.com/random")
      .get() (err, res, body) ->
        msg.send JSON.parse(body).pug

We can mock out the HTTP server on the other end of that request using the following test:

Helper = require('hubot-test-helper')
expect = require('chai').expect
nock = require('nock')

helper = new Helper('./../scripts/pugme.coffee')

describe 'pugme', ->
  room = null

  beforeEach ->
    room = helper.createRoom()
    do nock.disableNetConnect
    nock('http://pugme.herokuapp.com')
      .get('/random')
      .reply 200, { pug: 'http://imgur.com/pug.png' }

  afterEach ->
    room.destroy()
    nock.cleanAll()

  context 'user asks hubot for a pug', ->
    beforeEach (done) ->
      room.user.say 'alice', 'hubot pug me'
      setTimeout done, 100

    it 'should respond with a pug url', ->
      expect(room.messages).to.eql [
        [ 'alice', 'hubot pug me' ]
        [ 'hubot', 'http://imgur.com/pug.png' ]
      ]

In this case, the beforeEach function has a callback function done that will be called after the timeout within the before each is done. The setTimeout done, 100 will cause the beforeEach to pause for 100 ms before continuing with the tests. This will give the mock HTTP responder (and subsequently Hubot) adequate time to respond before the test assertion is run¹.

The nock HTTP listeners can also be chained to define multiple endpoints. For example:

  beforeEach ->
    room = helper.createRoom()
    do nock.disableNetConnect
    nock('http://pugme.herokuapp.com')
      .get('/random')
      .reply 200, { pug: 'http://imgur.com/pug.png' }
      .get('/bomb?count=5')
      .reply 200, { pugs: ['http://imgur.com/pug1.png', 'http://imgur.com/pug2.png'] }
      .get('/count')
      .reply 200, { pug_count: 365 }

Another important piece to note here are the nock listeners being torn down in the afterEach block (nock.cleanAll()). Not doing so can result in some odd, unpredictable results.

The complete pugme.coffee script can be referenced here, and the complete test suite can be found here.

  pugme
    user asks hubot for a pug
      ✓ should respond with a pug url

Common Pitfalls

Tearing Down Mocks

As noted in the section about Mock HTTP servers, make sure that mocks are always torn town. If you are working on one test and another randomly start failing, make sure that you are tearing things down correctly in the prior tests.

Environment booleans

If you pass a boolean through an environment variable (for example, in the case of the shipit.coffee script), keep in mind that the boolean value will be passed through as a string. While attempting to write tests for shipit.coffee, I ran into trouble setting process.env.HUBOT_SHIP_EXTRA_SQUIRRELS to false. The code on lines 40 and 41 originally read:

  if process.env.HUBOT_SHIP_EXTRA_SQUIRRELS
    regex = /ship(ping|z|s|ped)?\s*it/i

However, because environment variables are stored as strings, the variable the contained the value 'false'. The conditional would then see that, rather than the value being a boolean false, it existed as a string, therefore making if 'false' to be true. To get around this, I was forced to change to code to read:

  if process.env.HUBOT_SHIP_EXTRA_SQUIRRELS is 'true'
    regex = /ship(ping|z|s|ped)?\s*it/i

Conclusion

I hope that this blog post, along with the contents of the hubot-testing-boilerplate repo, help you to build out your test suite. The above code is by no means perfect, so pull requests are more than welcome. If you have any additional questions, please leave them in the comment section below!

Updating the Firmware on the LSI 9201-16i

Thu, 19 Feb 2015 12:35:12 +0000

After recently upgrading my FreeNAS system to from 8.3 to 9.3, I began receiving alerts about the LSI driver for my SAS controller, the LSI 9201-16i, being out of date:

WARNING: Firmware version 5 does not match driver version 16 for /dev/mps0.

While attempting to update the card with the UEFI shell, I found the process to be somewhat convoluted. The following guide details step-by-step how to upgrade the firmware and BIOs on the LSI 9201-16i SAS card using a UEFI shell.

Downloading the new Firmware

You can grab your desired firmware version from the LSI download page. Only the two most recent versions are available from the product’s page, so I found myself having to go through the full product download page. The previous versions can be found under the “Archive” link.

You’ll want to grab two files from the archive: Installer_PXX_for_UEFI, and 9201_16i_package_PXX_IT_Firmware_BIOS_for_MSDOS_Windows. I was specifically trying to upgrade to P16 for FreeNAS 9.3, for which the UEFI and DOS packages can be found here and here, respectively.

While these are downloading, grab a thumb drive. The drive doesn’t require any special formatting so long as it has some variation of a FAT filesystem. Create a folder on the thumbdrive, and copy the following files from each ZIP:

From the EFI ZIP:

sas2flash_efi_ebc_rel/sas2flash.efi

From the DOS ZIP:

Firmware/HBA_9201_16i_IT/9201-16i_it.bin
sasbios_rel/mptsas2.rom

Updating the Firmware & BIOs

With the thumbdrive prepared, plug it in to your system and reboot into the UEFI shell. Once in the shell, mount your thumbdrive (typically, the drive is either fs0 or fs1, making the command mount fs0). To confirm you have the correct drive, you can use most of your typical *NIX navigation commands: ls to list the contents of the directory, and cd to navigate into your subfolder (if you created one).

With your thumb drive selected, list out the RAID cards in the system with the sas2flash.efi -listall command.

If you see your card, it’s time to run the update command: sas2flash.efi -o -f 9201-16i_it.bin -b MPTSAS2.ROM.

One the command has completed, list the cards again to make sure it has run successfully.

If you see the expected firmware version, you’re all set. Issue a reset command to restart the system.

I hope this helps shed some light on this process!

UpShot Screenshots: Keeping Short Links Short

Tue, 20 Jan 2015 09:31:06 +0000

While looking for a way to automate my screenshot sharing, I recently stumbled across UpShot.it. UpShot is a small utility for OSX that attaches itself to your Dropbox Public folder and uploads screenshots as you take them. The URL for the uploaded screenshot is copied to your system’s clipboard for easy sharing. It’s a great tool, especially if your throwing screenshots back and forth frequently on a team.

One particularly cool feature of UpShot is its ability to mask the long Dropbox URLs behind custom domains. The documentation for UpShot explains how to set up a custom .htaccess file on your server to redirect traffic from the domain to the Dropbox URL. However, this method suffers one downfall: when a user navigates to your custom URL, your server sends a 302, the user is redirected to the Dropbox URL, and the shortened URL is lost.

There are a couple ways around this problem, some even officially mentioned. I investigated two of my own solutions: enabling mod_proxy, or reading the image through a script. Since my custom URL is pointing to hosting on a shared server, I don’t have the ability to turn Apache packages on or off. This leaves writing a script.

For compatibility’s sake, I’ve opted to write this script in PHP. Before diving into that, we’ll need to set up a new .htaccess file. The server needs to redirect all requests for screenshots to our custom PHP script, screenshot.php:

.htaccess

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^/?$ http://amussey.com [L]
RewriteRule ^([A-Za-z0-9.\-]+)$ screenshot.php?file=$1 [L,QSA]

Obviously, you’ll want to replace http://amussey.com with your own domain name. This line isn’t necessary, but serves as an easy redirect if someone decides to go snooping around.

screenshot.php

<?php
// Be sure to fill in your public Dropbox ID!
// You can get this from the UpShot preferences pane.
define(DROPBOX_URL, "https://dl.dropboxusercontent.com/u/[ Your Public Dropbox ID ]/Screenshots/");

// Treat all warnings as exceptions.
function exception_error_handler($errno, $errstr, $errfile, $errline ) {
    throw new ErrorException($errstr, $errno, 0, $errfile, $errline);
}
set_error_handler("exception_error_handler");

// Check to make sure the request is actually for an image file.
$file_extension = strtolower(substr($_GET["file"], -4));
if (($file_extension != ".png" && $file_extension != ".jpg") || strlen($_GET["file"]) != 8) {
    http_response_code(400);
    die("Not a valid image.");
}

// Let's get that screenshot!
$url = DROPBOX_URL.$_GET["file"];

try {
    return_image($url);
} catch (Exception $e) {
    // Failed to find the image.  Try again.
    sleep(3);
    try {
        return_image($url);
    } catch (Exception $e) {
        http_response_code(404);
        die("Image not found.");
    }
}

function return_image($url) {
    $imginfo = getimagesize($url);
    header("Content-type: ".$imginfo['mime']);
    readfile($url);
}
?>

Of course, there are some drawbacks to this method. Like the Google App Engine (dropbprox) method, this solution uses 2x the bandwidth (fetching the image from Dropbox, and serving the image again from the PHP script) and increases latency. However, some of the increased latency is built in to this script and designed to improve performance; I found that the split second it took to upload the image to Dropbox was often just long enough to open the link and see a 404. The above script has been designed so that, if the first fetch of the image fails, it will retry in 3 seconds before 404ing.

Now that we have this script, why not take it one step further and add in analytics? (Note: Complete version of both of these files can be found on github.)

.htaccess

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^/?$ http://amussey.com [L]
RewriteRule ^([A-Za-z0-9.\-]+)$ index.php?file=$1 [L,QSA]

screenshot.php

<?php
// Be sure to fill in your public Dropbox ID!
// You can get this from the UpShot preferences pane.
define(DROPBOX_URL, "https://dl.dropboxusercontent.com/u/[ Your Public Dropbox ID ]/Screenshots/");
define("ANALYTICS", true);
define("ANALYTICS_JSON", "dashboard/analytics.json");

// ... Inserted after $url = DROPBOX_URL.$_GET["file"]; ...

if (ANALYTICS) {
    $analytics_json = json_decode(file_get_contents(ANALYTICS_JSON), true);
    if (!isset($analytics_json[$_GET["file"]])) {
        $analytics_json[$_GET["file"]] = 0;
    }
    $analytics_json[$_GET["file"]]++;
    file_put_contents(ANALYTICS_JSON, json_encode($analytics_json));
}

// ... The rest of the PHP script ...
?>

These lines will write hits to an analytics.json file inside of a dashboard subdirectory. To review those hits, grab this dashboard.php file and copy it into the dashboard subdirectory as index.php.

Now if we want to see what files have been hit, we can just go to http://[My Custom Domain]/dashboard. It may be good to take an extra couple steps and set up some basic authentication on this folder so it’s not publicly accessible.

That’s all there is to it! Enjoy your shortened links!

Rackspace Cloud Monitoring Dashboard

Fri, 02 Jan 2015 14:31:25 +0000

As a group that heavily dogfoods our own products, the MyCloud team are regular users of the APIs fronted by the Rackspace Cloud Control Panel. Of these technologies, the Cloud Control Panel itself is run on top of a series of Cloud Servers that are watched by Cloud Monitoring. We tie these alarms into a variety of services to notify our DevOps team, from IRC bots to PagerDuty alerts.

While both the Cloud Control Panel and Cloud Intelligence offer good overviews of the servers and monitors, our team was in need of a system that could achieve the following:

Provide a quick overview of the status of all of the alerts.
Span multiple accounts.
Serve as a dashboard.

To fulfill these needs, we spent a portion of a team hackweek in early November designing a Cloud Monitoring dashboard. This dashboard, pictured below, provides an at-a-glance system status for all of our servers.

Each circle represents a different server on the account. The numbers in the center represent the number of OK alarms and the total number of alarms on the server, respectively. The segmented green ring provides a visual representation of the number of alarms in a WARNING or a CRITICAL state.

On the backend, the dashboard is a Python Flask app with a dependency of Redis for caching and some basic persistent storage. The Flask app serves both the frontend HTML application and a JSON API (API details can be found on the project’s GitHub page). Once a user has been added to the dashboard through the settings page, the API authenticates with the user’s token and performs an Overview call to the Cloud Monitoring API. The results of this call are stripped down to the required components and stored in Redis.

On the front end, the dashboard uses Jinja2 for templating, Twitter Bootstrap for layout, and jQuery for dynamic content. The dashboard will refresh its contents from the API every 20 seconds.

The dashboard was specifically designed to run on Heroku’s free tier with Redis Cloud as the database provider. To try out the Cloud Monitoring dashboard with your own servers, click here:

To view the source, visit the project’s GitHub page: https://github.com/amussey/cloud-monitoring-dashboard