yacn.me

ENGW3301 Project 3 Blog Post

Mon, 16 Jun 2014 00:00:00 -0400

ENGW 3301 Project 3 Blog Post

NOTE: This is supposed to be a guest post on The Intercept

Introduction

In June 2013, through the combined efforts of Edward Snowden, Laura Poitras, Glenn Greenwald, and many others, the world learned operational details of the ongoing global surveillance being conducted by the NSA and its global partners. These revelations, along with those that have since been revealed, emphasize the necessity that journalists working in this domain use strong privacy software to protect themselves from this global adversary.

Where We Are Now

There are numerous software systems in place to assist in this mission, such as TAILS (The Amnesic Incognito Live System), Tor, GPG, and OTR (Off-the-Record) Messaging. When used in concert correctly, these systems provide journalists with the privacy and protection they need. However, while they are vastly easier to use than ever before, they still require dedication and a desire to use them despite user-experience issues. As Glenn Greenwald puts it, "It's reall annoying and complicated, the encryption software" [1]

Additionally, using the OTR protocol for instant messaging with sources requires that you either host and maintain your own chat server, or that you trust the 3rd party hosting the chat server you're using is not going to log the ongoing communications. Additionally, this 3rd party provides a single point of failure in your security protocol; all an adversary would need to do is to subpoena the chat server for records of any and all communications. So, while we have come quite far, there are still a ways to go.

Where We're Headed

To solve the issues listed above, I propose a software system that would provide ephemeral, encrypted instant message communications between individuals with ease of use at the forefront of the design. Such a system would allow journalists to create a temporary server that they use to communicate with a source. Once their communications are over, they'll be able to completely destroy the server, erasing all traces of communication. In addition to removing the dependency on a 3rd party, this system allows for "hiding in plain site". In other words, by using commercial Infrastructure as a Service providers such as Amazon Web Services or RackSpace, the traffic being generated by these ephemeral servers would be masked by the ocean of network traffic these providers already generate.

Conclusion

If this sounds at all exciting to you, please get involved with the project in any way you can! The source code is open source, so if you have development experience you can contribute that way, or you could simply write an articel about this project to help generate more interest. Additionally, please share this with anyone you feel may benefit from this sort of software system.

[1] How Laura Poitras Helped Snowden Spill His Secrets

Evolution of Scrum at BitSight

Fri, 11 Apr 2014 00:00:00 -0400

The Evolution of Scrum at BitSight

At Northeastern University, I am in a 5-year program which allows for me to take 4 years of classes while also obtaining 18 months of work experience. I completed my first co-op at BitSight Technologies (and am currently still working there part-time and will be returning for my second), which practices agile project management using Scrum. Working at BitSight was my first experience with Scrum and it has been interesting to watch how its user has evolved over the last 11 months. When I started in May of 2013, BitSight was much smaller. The Engineering team consisted of about 10 full-time staff with about 5 graduate-student co-ops. At this point, standups still consisted of everybody meeting together around the kanban board. Eventually, as more full time staff and co-ops were hired, this became very unsustainable as standups were taking ~30 minutes to complete as 20 people is far too many (5 - 7 being the ideal amount). This was noticed fairly quickly and standups now consist only the necessary representatives from each team / project. The standups are still slightly larger than ideal, but I predict that as BitSight continues to grow, the standups will eventually branch off into smaller, more-focused groups.

Additionally, the Scrum Master initially was an employee who had been, for lack of a better phrase, stuck with it. They knew the basics, but not enough to efficiently lead the meetings and optimize team velocity. When a new engineer with Scrum experience was hired, they eventually took the reigns as Scrum Master and turned it around. This passing of the torch has lead to a rotating Scrum Master. That is, every so often, a new employee is trained how to be Scrum Master and then proceeds to lead the standups until eventually the next employee is selected and the cycle repeats. I think this is a brilliant plan, as it gets everyone more involved in the Scrum process which means that everyone will have a more detailed understanding of how it works (and why), leading to increased focus and increased agility. It will definitely be interesting to see how BitSight continues to change as it grows bigger and bigger.

"The Corruption of Agile" Response

Fri, 4 Apr 2014 00:00:00 -0400

"The Corruption of Agile" Response

The Corruption of Agile is (yet another) article detailing how the agile movement has deviated from a set of values to a set of practices, with a pseudo-religious tone surrounding said practices. The article argues that this shift from values to practices was heavily influenced by the vast number of "Agile consultants", who took the set of values stated in the Agile manifesto and translated them into rote programming practices, which the article argues loses the original values. In the end, the article argues that whether or not a company can be called agile depends on the culture of said company and not the practices the company uses. Now, I do agree with this point, as I have heard of many companies simply wanting to "buy agile", i.e. pay for a consultant to come in, run a few seminars, take their check, and leave. Tada, now we're agile! Now lets keep doing everything the same way as before, but now we're "agile". Right? Wrong. The article's conclusion is simply a prettier way of saying "if you want to call yourself agile, you need to put up or shut up".

One point that struct a chord with me while reading was the author's constant comparison of TDD to some sort of religious movement. While they do have a point when they're refering to the fanatics, i.e. those who see any other way of doing things as incorrect, I don't believe that this should discount the entire methodology. In a follow up, the author admits that the test coverage provided by TDD is valuable, though they argue that the same benefit could be reaped if tests are written after the fact. I do not disagree with this statement, the most important part of test-driven development is the resulting tests that you use to validate your code works correctly. However, I do believe that by following TDD practices writing tests is easier, simply because you're writing your tests while also writing your code, meaning you have the idea of what you want this function to do already in your head. When writing tests first, I've found it much easier to understand the full scope of the problem being worked on, which often leads to me implementing a better solution overall. In the end, as long as the test coverages are equal, it really doesn't matter how you get there, as long as you get there, I just find TDD makes it easier to get there.

"Dia" Review

Thu, 20 Mar 2014 00:00:00 -0400

"Dia" Review

Overview

Dia (pronounced "dee-a"), is FOSS tool used for general-purpose diagramming which runs on all major platforms (*nix, Windows, Mac OS X).

Finding Dia

I found Dia when searching for a diagramming tool that I could run locally on my own machine instead of needing internet access to use one of the various SaaS diagramming tools, such as Creatly or Lucidchart. Additionally, I wanted something that was free. This was both out of cheapness on my part (I don't often diagram things so it seems like a waste of money to pay for something I won't use) and due to frustration with the freemium SaaS offerings mentioned. The frustration stems from various features of the services only being available to members who pay, such as the ability to export your diagrams in a variety of formats or a limiting the set of shapes and objects you can use in your diagrams and still be able to export it. Dia satisfies all of these requirements and I have been using it for all of my diagramming needs since the end of February.

User Experience

Dia's UX, compared to the UX offered by tools like Crealy, Lucidchart, Gliffy, or OmniGraffle, is very rough. The application itself uses GTK+ to manage it's UI, which means the developers only need to write their UI for GTK+, which then handles displaying the UI on the variety of platforms it runs on. The downside to this approach is that the interface doesn't look native on any platform and doesn't integrate well with any (save for *nix platforms). The slight upside is that users only need to learn one interface, as it will be the same on any platform your application is used on.

Anyways, as I use OS X, the UX was affected even more. While *nix and Windows use the Control key to anchor most of their keyboard shortcuts, OS X uses the Command key. Dia/GTK does not take this into consideration, meaning I need to use the Control key for all keyboard shortcuts, which undermines all the muscle memory developed for standard keyboard shortcuts. Fortunately, it is fairly simple for me to get into the groove of using the Control key as I've used Windows for most of my life, but the initial context-switch can be fairly jarring.

Worth It?

The question you must be asking yourself now is, "Does Dia overcome it's rough UX or should I stick with my current tool of choice?". The answer to this question is "It depends". If you are going to be doing a lot of charting, I would definitely recommend going with a more commercial solution as it will be better polished and offer more features. However, if you just need a tool to create the occasional UML diagram for a class/homework assignment/one-off project/etc, I highly recommend Dia. The ability to work on my diagrams while not tethered to an internet connection while also being able to export my diagrams in any format I desire in addition to being completely libre outweighs the jarring UX for me as I feel the above features make it worth it.

"The Visible Ops Handbook" Review

Tue, 11 Mar 2014 00:00:00 -0400

"The Visible Ops Handbook" Review

My boss was kind enough to lend me his copy of The Visible Ops Handbook to read over Spring Break. I'd seen the book recommended many times, so when I saw the distinctive cover on his desk I had to ask about it. My initial reaction was that it was much thinner than I ever thought it would be, I was used to big compsci books and this is certainly not like those. However, short as it may be, there's definitely a lot of really great information packed in there. The book is only about 80 pages, so I was able to finish it on the flight to Indiana for Spring Break. The handbook contains a series of four steps detailing how you can transform your organization into a higher performing one. The four steps are:

Stabalize the Patient (Modify First Response)
Catch and Release (Find Fragile Artifacts)
Establish Repeatable Build Library
Enable Continuous Improvement

Stabalize the Patient

This step mainly concerns with stabalizing the organization's infrastructure ("the patient"). The biggest point they hit upon is that 80% of outages are caused by the organization themselves. To rememdy this, they focus on change management and control. Tripwire (software which detects changes to specified files) was heavily mentioned here, though that is because one of the authors (Gene Kim) founded Tripwire. They of course mention that alternatives exist, but none are discussed as in-depth (relatively speaking, as in, named) as Tripwire.

Catch and Release

This step is a response to a fragile infrastructure, that is, infrastructure that has been created and configured by hand, meaning it is not able to repeatably created and destroyed identically. An inventory must be done of all assets, configurations, and services to find the those which have the lowest successful change rate, the highest MTTR, and the highest downtime costs. Once these assets have been enumerated, the authors suggest that they be "frozen", meaning no changes are applied to them without good reason. Now that the infrastructure has been inventoried, the organization can focus on creating a provsioning process such that they have identical, repeatable infrastructure that can be easily replicated, which nicely leads into the next step.

Establish Repeatable Build Library

This step focuses on implementing an effective release management process by creating repeatable builds for the most important assets, with the end goal being to make it cheaper to rebuild than repair a given server (or, as my boss likes to put it, treating infrastructure like cattle, when one gets sick, just take it out back and shoot it and bring the next one in).

Enable Continuous Improvement

The previous steps have focused on building a tight loop of "Release=>Control=>Resolution". This step focuses on implementing metrics surrounding those process areas, because, as H. James Harrington said:

Measurement is the first step that leads to control and eventually to improvement. If you can't measure something, you can't understand it. If you can't understand it, you can't control it. If you can't control it, you can't improve it.

Source

Conclusion

I found the handbook to be a fast, concise read. I believe that any Operations Engineer/ System Administrator/etc should read it as it contains many valuable insights into increasing your organizations performance. Highly recommended.

Plex Issue Tracker and Custom Icon Font

Sat, 8 Mar 2014 00:00:00 -0500

Plex Issue Tracker

I've been hosting a Plex Media Server and sharing it with friends for almost a year now (started once I moved off campus and my internet allowed it) and in that time there's been a few bumps in the road, but overall it has been running quite smoothly. There's been the occasional glitch, e.g the Plex Media Scanner stops running (meaning no new shows are detected) necessitating a restart of the Plex service or the Plex service itself stops working and needs a restart. These issues are minor and take minimal effort to fix as I simply need to RDP in and restart the service. However, there's the occasional issue which requires a bit more manual intervention, such as shows having improper metadata. Since I'm not the only user of the server, there's actually a number of shows present that I do not keep up with at all. What ends up happening is I'll be hanging out with one of my friends and they'll mention some issue they're having with Plex. I then make a mental note to remember to fix it when I return home. Being the human I am, I don't have perfect memory, which means things sometimes slip through the cracks.

With that in mind, I decided to create a place to file support requests for GUNTER, my Plex server. I created a GitHub repository gunter-issues to hold any support requests / issues related to Plex or GUNTER. Using the free service waffle.io, I was able to turn that repo into an open and online agile/kanban board. Now, people can file issues with the knowledge that if I forget about it, it's right there on the board. It is my hope that by introducing this, GUNTER/Plex can only improve.

Plex Info Page and Icon Fonts

While integrating the issue tracker into this site, I wanted to add a link to the a page containing information and resources about Plex. Since it was going to be in the upper-right corner, I wanted it to fit in with the Font Awesome icons I was already using. I came up with the idea of putting together a version of the Plex Media Center logo in the style of the Font Awesome icons.

I started by finding the PMS logo and then modified its coloring to fit in with the Font Awesome icons. I ended up with a PNG with transparency, but then realized that this wouldn't scale well if I wanted to use it at a bigger size. I decided at that point to convert it to an SVG. Unfortunately while the software I used to modify the image (Pixelmator) supports creating vector-like graphics, it does not support exporting to SVGs. However, I was able to use a piece of software called Vector Magic to take the PSD exported by Pixelmator and feed it in and with about 5 clicks, I had an SVG!

I then spent quite a bit of time futzing around with the CSS sizing and positioning, trying to get the SVG to appear, but nothing seemed to work. Frustrated and out of ideas, I googled "create own font awesome icons svg" and stumbled upon this awesome RubyGem called Font Custom, which takes a directory of SVG files and converts them to a CSS file that you can include on your sites. With this in hand, I was able to convert my SVG into a font and then it was a simple matter of adding an item to the navigation list like all the other links! In the end, I think it looks and feels like a native Font Awesome icon and was well worth the effort to get it going.

Migrating from dvm to boot2docker

Fri, 7 Mar 2014 00:00:00 -0500

Migrating from `dvm` to `boot2docker`

Preconditions

About two weeks ago, I wrote about how I got dvm working with Docker using a specified network range for the docker0 bridge. A quick overview of dvm, for the uninitiated: It's a Bash script wrapped around a Vagrantfile which uses a Vagrant box based off the boot2docker image. The original boot2docker script itself is simply a Bash script wrapped around the VBoxManage command. There's an on-going effort to rewrite the boot2docker script in Go, but it is currently experimental. Anyways, dvm uses Vagrant to interact with VirtualBox, where Vagrant itself is a wrapper around interacting with VirtualBox. Since it uses Vagrant, it needs a Vagrant Box to create the VM, meaning that each time a new boot2docker image is released, a new Vagrant Box must also be created before dvm can use the latest version.

You've probably already guessed why I'm migrating away, there's just simply too many moving pieces that can each become a bottleneck. dvm can bottleneck with bugfixes while I can also be bottlenecked by waiting for new Vagrant Boxes to be released. To eliminate this, I have decided to move away from using dvm to using the original boot2docker (I'm actually using boot2docker-cli, the Go port of the tool, but more on that later).

Moving previous work

The downside to this is that I needed to find a way to perform the same customizations I was using with dvm to change the network range of the docker0 interface. With dvm, adding local customizations is as simple as editing the provisioning script which is run wheneve the VM is created. Using boot2docker itself wasn't as straight forward. As boot2docker is run entirely from memory, a persistence disk must be attached to the VM to provide a place to save things so they won't be deleted. While dvm handled this in the Vagrantfile, the boot2docker script actually creates it for you as well and mounts it to /mnt/sda1. To add local customizations, you can place what you want to do into a script titled bootlocal.sh and place that inside /var/lib/boot2docker (which is symlinked to /mnt/sda1/var/lib/boot2docker).

The tasks I needed to take care of in this script were to:

Install bridge utils if not present
Modify Docker's init script to add the --bip option
Create a folder on the persistence disk for use as a mount point for SSHFS

To accomplish 1 and 2, I was able to re-use most of the work done in dvm's Vagrantfile. For 3, it was simply a matter of creating a folder. Once I had written the script, however, I realized I needed to get it on the VM disk before it'd be useful! I could have simply re-typed it while SSH'd into the VM, but that's not very DRY and I'll likely need to add more files to the VM at a later point. Luckily, the boot2docker maintainers provide a workaround using FUSE for OS X and SSHFS, so I was able to mount a folder from my host laptop on the VM with minimal fuss.

The one issue I ran into was with the command used to mount the folder. In the steps given, they have you create a file b2d-passwd containing the password for the docker user on the VM. Then, when you go to mount the folder on the VM, they have you run:

sshfs docker@localhost:/mnt/sda1/myapp ~/myapp -oping_diskarb,volname=b2d-myapp -p 2022 -o reconnect -o UserKnownHostsFile=/dev/null -o password_stdin < ~/.boot2docker/b2d-passwd

where /mnt/sda1/myapp is the mount-point on the VM and ~/myapp is the folder on the host being shared with the VM. The part that I had issue with was -o password_stdin < ~/.boot2docker/b2d-passwd. No matter what I tried, I always got a timeout error while waiting for the prompt. When I removed that part of the command, the reason for this became immediately obvious: I was being asked yes/no about the VM's SSH key, meaning the prompt would never show! As of now, I haven't found a workaround, it is easy enough to type yes and tcuser when I want to mount a folder.

Bumps in the road

Once that was working, it was trivial to move the bootlocal.sh script from the shared folder to its place in /var/lib/boot2docker. Then I simply logged out, stopped the VM, and copied the VMDK to a safe place. Now when I want to create a new boot2docker VM, I just need to copy the VMDK into place (~/.boot2docker) before running boot2docker init. Once that completes, I have a fresh VM with all my local customizations. The next logical step for this would be to create a customized ISO that includes the bootlocal.sh to remove the need to remember to copy the VMDK back into place when creating a new VM.

However, this is where the issues have started springing up. When I tried to run Test Kitchen through the new boot2docker VM, I ran into the issue where the VM hangs waiting for the container to be ready. Reading through that thread it seemed like any issues with the software itself (Test Kitchen, kitchen-docker, boot2docker, docker itself, etc) should have been fixed in the versions I am using (Test Kitchen 1.2.1, kitchen-docker installed from GitHub, boot2docker 0.6.0, docker 0.8.1). This issue seemed familiar though, and upon further digging I narrowed it down to routes getting crossed and confused. Aha! This was why I moved to dvm in the first place. dvm is slightly different from the default boot2docker because it creates a host-only network to run Docker on. This solves the issue of criss-crossed routes, so I set out to find if this feature existed in boot2docker.

In my research, I found this hostonly interface-related pull request. At the time of this writing, the pull request is still open, so this feature hasn't made its way into the main boot2docker. Fortunately though, at the bottom of the pull request the discussion gets moved to this pull request on the new boot2docker-cli repo. That request was merged, so the feature in some form exists. This is why I am as of now using the boot2docker-cli tool instead of the normal one.

What's left to do

Once that was all figured out, my next move was to try to get test kitchen and kitchen-docker working together with this new boot2docker VM. I re-created my VMDK w/ the bootlocal.sh script and tried to run kitchen test. Unfortunately, this fails right after pulling in the ubuntu images. Looking through the logs, it seems like the containers are unable to access the Internet. Also, the boot2docker-cli creates a new hostonly interface for each VM it creates, but it also never deletes them after the VMs are deleted. This also leads to issues with criss-crossed routes as packets are being sent to the wrong hostonly interface. Lastly, I seem to be unable to run boot2docker-cli ssh or boot2docker ssh to log into the VMs while I'm connected to my work's VPN. This is strange since I was able to use both of those commands just fine while at work, but as soon as I turned my VPN off, they both started working. I suspect it is also an issue of criss-crossed routes.

In summary, the following work is left to do before the migration from dvm to boot2docker is complete:

Figure out why containers cannot reach the internet
Modify boot2docker-cli init/delete to keep track of the interfaces it creates so they can be deleted with the VM.
Figure out why I'm unable to SSH into the VM while logged into my work's VPN.

I've started reading about Go so that I can contribute patches, so hopefully I should be able to make some progress on number 2 before too long. It seems to be a simple matter of creating a map of the interfaces created to the VMs their attached to, but there's all of Go to grapple with first. It's going to be a busy upcoming week too, as the 2014 Northeast CCDC is this Friday, Saturday, and Sunday at UNH. Fun fun fun, so much to do, so little time.

Using a Kanban Board as a To-Do List

Tue, 4 Mar 2014 00:00:00 -0500

Using a Kanban Board as a To-Do List

Even after missing my flight home for winter break and spending the entire day (8AM to midnight when I was finally picked up by my parents) in the airport, I was still excited. Mostly to finally escape airports and go back to sleep, but also because I knew I had two weeks ahead of me which I could use to do whatever I wanted as it was my free time. I had just finished my first co-op at BitSight where they use Scrum for project management, so I was very used to and in the groove of using a kanban board to visualize all the tasks currently being worked on. I wanted to continue using such a process as I felt it would help me switch into working mode more easily. I found waffle.io, which creates an online kanban board for you based on the issues on a given GitHub repository and the tags associated with said issue. I created an empty repo to use for my todo list and set off to work.

I found through using a kanban board that I was able to work on multiple things at once much easier since it helped to lower the amount of time spent context switching because all of the necessary information was right there in front of me: what I did last time, what there is to do, what's the status of the things I'm working on.

Over the past few months I have sort of left it hanging and unused, not because I didn't find it helpful, but because I've been very busy with my coursework and haven't had much (if any) time for personal projects. Why didn't I use it for my school work you ask? Simply because I already had a workflow and software setup for doing just that. I use iStudiez Pro to manage and track my coursework and grades. It's really a joy to use and it makes staying on top of the large workload that much more manageable. There are apps for OS X, the iPhone, as well as the iPad, and you're able to sync your data across all devices with ease, making it easier than ever to stay on top of my coursework.

Once the summer starts, I'll have more free time to devote to personal projects so I'm certain I'll be using said online kanban board again to manage and track my progress, the workflow just makes sense to me.

Sensu EC2 Node Handler

Mon, 24 Feb 2014 00:00:00 -0500

Sensu EC2 Node Handler

The Problem

At work, we run Sensu as the base for our metrics, monitoring, and alerting pipeline. A while back, we were running into an issue with phantom keepalive alerts due to how Sensu was determining whether or not a node still existed. The way it was determining this was to query our Chef server to check whether or not the client and node existed there. However, due to how we provision and terminate nodes (using an internal knife plugin), the clients and nodes on the Chef server are not removed until the next time they are to be created. (Really, we should update the deletion component to remove the clients and nodes, but it is a low priority at this point).

Due to this workflow, when we would terminate a cluster, we would encounter a "keepalive storm", where we would be flooded with keepalive alerts for the cluster we just terminated. This would cause our Sensu server to become overloaded with queued up events to handle. When our next Chef run would execute, the Sensu server would be signaled to restart. However, due to the massive event queue that had built up, the time needed to restart the process would exceed the threshold set for the service's script, set in /etc/default/sensu. I believe the default was 10 seconds, we increased it to 30 while investigating to try and soothe the inevitable Chef runs.

Nevertheless, the process would nearly always fail to restart, which caused the Chef run to fail and would leave teh Sensu server in an incorrect state. The remedy this state, we owuld need to manually stop and start each of the Sensu components (API, server, client) manually. This was very annoying and inconveniencing, so I set out to find a fix for the situation.

The Solution

To solve our phantom keepalives and subsequent "keepalive storms", I created a Sensu handler for the keepalive event. When it handles a keepalive event, instead of checking the Chef server to determine whether the node still exists, it checks EC2. If the node is not found in EC2, it is removed from Sensu as a client.

When we implemented this handler at work, we were able to eliminate the "keepalive storms" and phantom keepalive alerts, thus increasing the stability of our Sensu server. You can check out the plugin in the official Sensu Community Plugins repository on GitHub. Hopefully it will be of some use to others.

Fixing dvm with Test Kitchen

Mon, 24 Feb 2014 00:00:00 -0500

Fixing dvm with Test Kitchen

Finding dvm

For the past few weeks at work, I've been conducting research on the side with the goal of setting up a continuous integration environment for our Chef cookbooks. I've decided to use Test Kitchen with the kitchen-docker driver as the base for our integration testing framework (of which the specifics are out of the scope of this post).

Now that I'd chosen our stack, it was time to get kitchen-docker, Test Kitchen, and Docker working together. I needed a solution which would allow me to work with Docker locally on OS X. Luckily, there are two related projects allowing me to do just that, dvm and boot2docker (dvm is actually just a wrapper around Vagrant using a boot2docker box).

Bumpy start

The first issue I ran into was Docker choosing a network range which conflicted with the network range our work runs in. I'd actually run into this issue before, several months ago when I was first testing out Docker. At the time, there was no immediate workaround, so put it in the back of my mind for later. Docker has since been updated to attempt to be smart about choosing the network range by looking for any conflicting networks. However, this issue still affects us because Docker is being run through a VM, meaning it will not detect the conflicting network.

A more recent addition to Docker is the --bip="" option, which you can use to specify a CIDR range for the docker0 bridge that Docker creates. However, to use this new feature with dvm, I was going to need to make some modifications to support it.

Modifying dvm

The way dvm works is by allowing you to set configuration options in a file ~/.dvm/dvm.conf. Those settings are environment variables which are exported when creating the boot2docker VM. To add support for --bip, I added the configuration directive DOCKER0_CIDR to the config file. I then modified the Vagrantfile to use the new argument if specified. I saved my changes, crossed my fingers, and typed dvm up. To my dismay, passing in that option didn't seem to affect anything!

(Re)building the bridge

After some more reading, I learned that if the docker0 bridge has already been created, the IP range assigned to it will not be affected by the --bip option. To use the option, you need to first make sure the docker0 bridge does not exist when starting Docker, meaning that if it does exist, it will need to be deleted.

Deleting a bridge seems simple enough, you just need to run brctl delbr nameofbridge. Unfortunately, the brctl utility is not included in the release of Tiny Core Linux used by boot2docker. I now set off to find a bridge-utils package I could install on the distro. Fortunately I didn't need to dig for too long, there was a link to such a package in one of the docs for boot2docker.

Now all I needed to do was update the Vagrantfile to check whether or not the DOCKER0_CIDR option was set, and if so, setup the arguments, stop the currently running Docker daemon, install the bridge utils, and remove the docker0 bridge. I ran dvm up and low and behold, it worked!

By this time I'm feeling pretty good, I've gotten much farther than ever before. I run kitchen create to create a container and, you guessed it, it worked! Running kitchen converge also worked as expected, which is great as now my container can access hosts within our local network as well as the internet at large! Satisfied with myself, I ran kitchen destroy to delete the containers. Oh no, it doesn't work!

One last jam

Unfortunately I ran into a jam (damn you Chris Christie!), running kitchen destroy throws an exception! Looking at the message returned, it seems to be an issue related with trying to stop and remove the container, something appears to be holding a lock over it:

|ruby-1.9.3-p448| isaacs-air in ~/git/yacn/dvm/tmp
± |fix/unable-to-stop-delete-containers ✗| → kitchen destroy
-----> Starting Kitchen (v1.2.1)
-----> Destroying <default-ubuntu>...
368ccd5979763ce38c072d9751417ba457827146548d7a04676b3a5795c806ca
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #destroy action: [Expected process to exit with [0], but received '1'
---- Begin output of docker -H tcp://192.168.42.43:4243 rm 368ccd5979763ce38c072d9751417ba457827146548d7a04676b3a5795c806ca ----
STDOUT:
STDERR: Error: container_delete: Cannot destroy container 368ccd5979763ce38c072d9751417ba457827146548d7a04676b3a5795c806ca: Driver aufs failed to remove root filesystem 368ccd5979763ce38c072d9751417ba457827146548d7a04676b3a5795c806ca: rename /var/lib/docker/aufs/mnt/368ccd5979763ce38c072d9751417ba457827146548d7a04676b3a5795c806ca /var/lib/docker/aufs/mnt/368ccd5979763ce38c072d9751417ba457827146548d7a04676b3a5795c806ca-removing: device or resource busy
2014/02/24 14:07:29 Error: failed to remove one or more containers
---- End output of docker -H tcp://192.168.42.43:4243 rm 368ccd5979763ce38c072d9751417ba457827146548d7a04676b3a5795c806ca ----
Ran docker -H tcp://192.168.42.43:4243 rm 368ccd5979763ce38c072d9751417ba457827146548d7a04676b3a5795c806ca returned 1]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

Yikes, that doesn't seem good. Well, lets see if the container is still there. I logged into the dvm machine via dvm ssh and ran docker ps

|ruby-1.9.3-p448| isaacs-air in ~/git/yacn/dvm/tmp
± |fix/unable-to-stop-delete-containers ✗| → dvm ssh
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
boot2docker: 0.5.4
docker@boot2docker:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
docker@boot2docker:~$ exit

Hmm, so it looks like the container was actually stopped. Researching this issue led me to this thread. The issue seems to be related to using a symlink where Docker expects its runtime root to be. By default, this is /var/lib/docker, but in boot2docker, this has been symlinked to /mnt/sda/var/lib/docker. This causes Docker to be unable to stop or remove any of the containers it creates, as it doesn't seem to follow the symlink.

The remedy is to use the -g option to set the path to use as the root of Docker's runtime to the directory the symlink points to. Looking at the file in boot2docker where the Docker daemon is started (/usr/local/etc/init.d/docker), it seems like the -g option should already be provided. All the more curious and needs further investigation.

A `sed` to far

The culprit, it turns out, is the Vagrantfile used by dvm! In the provisioning script used to restart the Docker daemon with the args provided in DOCKER_ARGS, there's a sed command which looks like:

sed -i -e 's|docker -d .* $EXPOSE_ALL|docker -d #{args}|' $INITD

where:

$INITD is /usr/local/etc/init.d/docker
$EXPOSE_ALL is a variable set in that file
#{args} is the string interpolation of the args defined in DOCKER_ARGS.

This command is only run if you modify either args or DOCKER_ARGS, which is how I believe it slipped through the cracks.

What happens is the part of the line in /usr/local/etc/init.d/docker that is specifying the -g option is wiped out when the new args are added to the command. There were a few options to solving this:

Update the README to state that if you modify DOCKER_ARGS, you must make sure to include the -g option
Update the Vagrantfile to check if DOCKER_ARGS is modified, and if so, to check that it includes the -g option, adding it if it does not.
Update the sed command to save the -g option and use a backreference to it.

I chose to implement option 3 as it seemed the simplest and most transparent way to accomplish the goal.

One last run

With the changes in place, it was time to test them out again. I created a new dvm VM with the updated Vagrantfile and a dvm.conf specifying DOCKER0_CIDR. Once that was created, I checked to make sure the docker0 bridge had the new IP range. So far, so good, so I ran kitchen create and watched as Test Kitchen pulled in the Ubuntu repo and created a container for me. Still looking good, but all of this did work before. I enter kitchen destroy for the final test and...

|ruby-1.9.3-p448| isaacs-air in ~/git/yacn/dvm/tmp
± |fix/unable-to-stop-delete-containers ✗| → kitchen destroy
-----> Starting Kitchen (v1.2.1)
-----> Destroying <default-ubuntu>...
2118a87d5cf2f5d6bc502a3f086b4ab828e6c3023624e2da9e8590eae043ec1f
2118a87d5cf2f5d6bc502a3f086b4ab828e6c3023624e2da9e8590eae043ec1f
Finished destroying <default-ubuntu> (0m0.14s).
-----> Kitchen is finished. (0m0.20s)

|ruby-1.9.3-p448| isaacs-air in ~/git/yacn/dvm/tmp
± |fix/unable-to-stop-delete-containers ✗| → kitchen list
Instance Driver Provisioner Last Action
default-ubuntu Docker ChefZero <Not Created>

WooHoo! It works!

Conclusion

To summarize, I needed to update dvm to allow me to set the IP range of the docker0 bridge. This required me to install bridge-utils for brctl to remove the previously created docker0 bridge before my new IP range could take effect. Once that was implemented, I discovered a bug in the provisioning script for dvm causing which prevented stopping or deleting containers. Both fixes are currently open as pull requests:

yacn.me

ENGW3301 Project 3 Blog Post

ENGW 3301 Project 3 Blog Post

Introduction

Where We Are Now

Where We're Headed

Conclusion

Evolution of Scrum at BitSight

The Evolution of Scrum at BitSight

"The Corruption of Agile" Response

"The Corruption of Agile" Response

"Dia" Review

"Dia" Review

Overview

Finding Dia

User Experience

Worth It?

"The Visible Ops Handbook" Review

"The Visible Ops Handbook" Review

Stabalize the Patient

Catch and Release

Establish Repeatable Build Library

Enable Continuous Improvement

Conclusion

Plex Issue Tracker and Custom Icon Font

Plex Issue Tracker

Plex Info Page and Icon Fonts

Migrating from dvm to boot2docker

Migrating from dvm to boot2docker

Preconditions

Moving previous work

Bumps in the road

What's left to do

Using a Kanban Board as a To-Do List

Using a Kanban Board as a To-Do List

Sensu EC2 Node Handler

Sensu EC2 Node Handler

The Problem

The Solution

Fixing dvm with Test Kitchen

Fixing dvm with Test Kitchen

Finding dvm

Bumpy start

Modifying dvm

(Re)building the bridge

One last jam

A sed to far

One last run

Conclusion

Migrating from `dvm` to `boot2docker`

A `sed` to far