Yujin Kim's blog

At work, we are building a system that follows the general trend of automated system delivery to get more efficient at what we do.

Here’s a little diagram that illustrates the jackpot we are trying to hit – our nirvana.

The premise of the whole thing is not very different from what many people are already talking about -

• Puppet maintains all the facts and configurations of the system
• Utilizing the APIs available from the hosting providers, we can dynamically provision new hosts

We have added a few addition to the automated tasks.  We have done this in the past in a lose sense in that we have 4-5 sub-systems that do its part, but not quite integrated all together.  The goal is to close all the lose ends and make everything tick together.

• Standard directory structure, and forced layout of application directories (We allow no exception to this)
• Our custom CMDB engine provides information about the host and the application s/application runtime that will be installed for our mostly Java based stack.
• Our application build system (as well as CI) integrates with the CMDB to handle the graceful deployment of files – i.e it knows how to build production, staging etc based on the information it retrieves from our CMDB.
• We plan to export our Java application statistics using JMX both to mCollective.
• This came from my discussion with Teyo at PuppetLabs, and there might be an opportunity to add maven repository structure as one of the package provider in puppet, which will make a lot of things easier to support Java shops like ours.

I had a first demo of the work that have been in progress to our internal development team and received positive feedbacks from folks (and we have a lot of smart folks on our team), so it was encouraging.   It’s not quite where I want it to be, but as the things progress more, I hope to share my learnings from my experience in trying to putting everything together.

I’ll be honest. I haven’t really played with mCollective yet and even thought it was ironic that to solve the problem of building out complicated infrastructure, I need to build a complicated system (with mcollective being mostly Java based set up with Java based messing queue systems).

But now with Puppet acquiring the intellectual property of it, and seeing the type of Vision that Luke and co illustrated on their announcement page, it is making me rethink (or I should really say learn) more about mCollective and its future as illustrated in Puppet Lab’s diagrams.

The separation of management network with the hosting datacenters just strikes me a brilliant idea. Just recently having gone through two major hosting migration, and seeing many hosting migration going on everywhere, from EC2 to private cloud, or private cloud to dedicated cloud, etc, having this integration achieves a few awesome goals from business perspective.

1. Ability to manage multiple datacenters from a single source.
2. Ability to move hosting with much smaller effort in the future
3. Huge cost savings when moving things across different hosting options

Consider the following scenario:

You may have multiple data centers, hosting options for different reasons. In this example, you have one hosting vendor handling the .net application cluster, another for ruby on rails, and another for Java. Some of them could be on EC2, some of them on a large enterprise scale hosting center, and some on a small hosting companies. Due to the various type of hosting tier model you have in your organization, this type of scenario is very likely in many organizations.

Consider how many human hours are spent to manage all these, probably for most part manually or automated but per each data center. For various reasons, these options might need to be changed rapidly. The decision could come down to you to reduce cost, to consolidate the data centers, to move out the data center, or even to add new data center to the mix. There are millions of reasons why you may switch hosting providers and there are millions of places out there that does this at least once or twice in 5 years.

Now with Puppet and mCollective integration in the picture, this whole system could really REMOVE (not reduce) costs associated with these type of activities for the next 5 years. More over, now organizations can be come almost “hosting provider agnostic” and can deploy applications to various hosting providers/data centers relying on the same configuration management system. The only one that gets locked up would be the management network where puppet master, mcollective host, dashboard and some sort of CMDB resides on.

I’m looking forward to see how the integration is going to develop but I’m ready to test this out now and how I can plan on this idea now.

While it’s an old news that google wave is being phased out, it appears that google has miscalculated the launch of this potentially game changing product and is also killing it prematurely. What’s new is that Google just opened up the service for Google App for business, and this google wave could have been a very effective way to communicate in the workspace. My boss Rajiv has been a strong proponent of using real time chat for operation management (see Benefits of Using Real-Time Group Chat (IRC) in Technology Operations Management)

With google adding the service to the google application for business offerings, it would have been easier to adopt the corporate users (that is if the corporate is allowing to set up one of the domains with google app) and allow corporate users to “wave” the conversation for various technology operations such as releases, troubleshooting, or even corporate events.

Focusing on the troubleshooting, it’s pretty common that if you are on a troubleshooting bridge, whenever a new person joins the call, oftentimes, the person managing the call has to provide the status updates, and that often could be somewhat distracting to the people who are investigating the issue as it disrupts the flow and conversation that had been taken place. Wave could have been a great solution for 1) providing the history of the threads 2) allowing private conversation without interrupting the group calls and 3) providing a very complete tracking of the issues at hand. This product could have been hence integrated with many Runbook Automation systems, as well as google search engine and would have allowed a room for a birth of fairly interesting correlation engine using google’s sophisticated search algorithm. After all, Wave could have replaced mailing list archive, forums, and any support type Q&As or FAQs – which is pretty much all the stack exchange type sites are doing these days anyway.

I wish google have waited a bit longer before announcing the end of the product, as I believe many would have found a good use out of it. Rather, Google might have been caught up too much on the social aspect of it. Most likely Google will find a way to utilize the code base in some other way and perhaps they will think of a way to target corporate/SMB market first next time.