Here’s a little diagram that illustrates the jackpot we are trying to hit – our nirvana.

The premise of the whole thing is not very different from what many people are already talking about -

• Puppet maintains all the facts and configurations of the system
• Utilizing the APIs available from the hosting providers, we can dynamically provision new hosts

We have added a few addition to the automated tasks.  We have done this in the past in a lose sense in that we have 4-5 sub-systems that do its part, but not quite integrated all together.  The goal is to close all the lose ends and make everything tick together.

• Standard directory structure, and forced layout of application directories (We allow no exception to this)
• Our custom CMDB engine provides information about the host and the application s/application runtime that will be installed for our mostly Java based stack.
• Our application build system (as well as CI) integrates with the CMDB to handle the graceful deployment of files – i.e it knows how to build production, staging etc based on the information it retrieves from our CMDB.
• We plan to export our Java application statistics using JMX both to mCollective.
• This came from my discussion with Teyo at PuppetLabs, and there might be an opportunity to add maven repository structure as one of the package provider in puppet, which will make a lot of things easier to support Java shops like ours.

I had a first demo of the work that have been in progress to our internal development team and received positive feedbacks from folks (and we have a lot of smart folks on our team), so it was encouraging.   It’s not quite where I want it to be, but as the things progress more, I hope to share my learnings from my experience in trying to putting everything together.

Related posts:

1. Puppet and mCollective

But now with Puppet acquiring the intellectual property of it, and seeing the type of Vision that Luke and co illustrated on their announcement page, it is making me rethink (or I should really say learn) more about mCollective and its future as illustrated in Puppet Lab’s diagrams.

The separation of management network with the hosting datacenters just strikes me a brilliant idea. Just recently having gone through two major hosting migration, and seeing many hosting migration going on everywhere, from EC2 to private cloud, or private cloud to dedicated cloud, etc, having this integration achieves a few awesome goals from business perspective.

1. Ability to manage multiple datacenters from a single source.
2. Ability to move hosting with much smaller effort in the future
3. Huge cost savings when moving things across different hosting options

Consider the following scenario:

You may have multiple data centers, hosting options for different reasons. In this example, you have one hosting vendor handling the .net application cluster, another for ruby on rails, and another for Java. Some of them could be on EC2, some of them on a large enterprise scale hosting center, and some on a small hosting companies. Due to the various type of hosting tier model you have in your organization, this type of scenario is very likely in many organizations.

Consider how many human hours are spent to manage all these, probably for most part manually or automated but per each data center. For various reasons, these options might need to be changed rapidly. The decision could come down to you to reduce cost, to consolidate the data centers, to move out the data center, or even to add new data center to the mix. There are millions of reasons why you may switch hosting providers and there are millions of places out there that does this at least once or twice in 5 years.

Now with Puppet and mCollective integration in the picture, this whole system could really REMOVE (not reduce) costs associated with these type of activities for the next 5 years. More over, now organizations can be come almost “hosting provider agnostic” and can deploy applications to various hosting providers/data centers relying on the same configuration management system. The only one that gets locked up would be the management network where puppet master, mcollective host, dashboard and some sort of CMDB resides on.

I’m looking forward to see how the integration is going to develop but I’m ready to test this out now and how I can plan on this idea now.

No related posts.

With google adding the service to the google application for business offerings, it would have been easier to adopt the corporate users (that is if the corporate is allowing to set up one of the domains with google app) and allow corporate users to “wave” the conversation for various technology operations such as releases, troubleshooting, or even corporate events.

Focusing on the troubleshooting, it’s pretty common that if you are on a troubleshooting bridge, whenever a new person joins the call, oftentimes, the person managing the call has to provide the status updates, and that often could be somewhat distracting to the people who are investigating the issue as it disrupts the flow and conversation that had been taken place. Wave could have been a great solution for 1) providing the history of the threads 2) allowing private conversation without interrupting the group calls and 3) providing a very complete tracking of the issues at hand. This product could have been hence integrated with many Runbook Automation systems, as well as google search engine and would have allowed a room for a birth of fairly interesting correlation engine using google’s sophisticated search algorithm. After all, Wave could have replaced mailing list archive, forums, and any support type Q&As or FAQs – which is pretty much all the stack exchange type sites are doing these days anyway.

I wish google have waited a bit longer before announcing the end of the product, as I believe many would have found a good use out of it. Rather, Google might have been caught up too much on the social aspect of it. Most likely Google will find a way to utilize the code base in some other way and perhaps they will think of a way to target corporate/SMB market first next time.

Related posts:

1. My take on DevOps

I believe much of the focus is on comparing the traditional system engineers and the “devops” type engineers. While I have no disagreement on this concept, I think the role of devops also plays an important and positive role to the development team. Many senior software engineers are often very good architect with good understanding of infrastructure. At work, I have had a great honor working with very talented people who just haven’t had a chance to go through some of the processes involved in troubleshooting applications – such as reading thread dumps, analyzing heap dumps, using profilers, or even reading AWR reports, or even some of the commonly used system level debugging tools such as vmstat, iostat, strace/dtrace, etc.

As much as DevOps are helping bridging the gap between software engineers and operations, it can also help software engineers gain more experience or knowledge around those tools and in a way influence software engineers to learn how to think like operation people do. This can only make the software engineers become better architects and make them equipped with troubleshooting knowledges.

I have been involved in various kind of interesting and challenging troubleshooting of application/infrastructure issues with other engineers(both software and system). It has given me understand the applications much better and allow me to suggest code improvements, and software engineers giving my team feedbacks on what can be better on operation wise. It also has given some of the software engineers exposures to the above mentioned tools that allowed them to be able to diagnose the applications before the code gets deployed to production systems.

While the world is not a perfect place, I do really believe that devops movement is providing positive influence to the development team, and vice versa. The true value of devops, IMHO, is reinforcing the fact that every one is providing service to the same client in the end. In other words, from clients eyes, it’s really the technology department that’s providing what they need, and not software engineering team or the operation team.

Bottom line: devops is really about creating a culture, and a mindset that we are all in this together and let’s work it out together.

No related posts.

I was hoping to install Fedora 12 but Fedora unfortunately wasn’t able to recognize HFS+ file system very well (which was apparent when I looked at fdisk).  Installing Fedora using the GUI came pretty close but it didn’t know how to install bootloader. I also was going to try OpenSuse but overall impression I got was ubuntu has the better support (since it’s community driven.) BTW, I installed the server edition knowing I wanted this box to just act as a server for puppet and other things.

Anyhow, this post is just to keep track of the changes I’ve made

sudo apt-get install puppet
sudo apt-get install zip
sudo apt-get install gcc
sudo apt-get install chkconfig
sudo apt-get install sysv-rc-conf
sudo apt-get install whois
sudo apt-get install libio-socket-ssl-perl
sudo apt-get install libnss-mdns

Also installed ddclient for dyndns.

wget http://cdn.dyndns.com/ddclient.tar.gz
mkdir /etc/ddclient
cp sample-etc_ddclient.conf /etc/ddclient/ddclient.conf
vi /etc/ddclient/ddclient.conf
cp sample-etc_rc.d_init.d_ddclient /etc/init.d/ddclient
vi /etc/init.d/ddclient
mkdir /var/cache/ddclient
sysv-rc-conf 
/etc/init.d/ddclient start

Updated the network interface to have static ip

cd /etc/network
cp -a interfaces interfaces.bak
iface eth0 inet static
address x.x.x.x
netmask x.x.x.x
network x.x.x.x
broadcast x.x.x.x
gateway x.x.x.x

I’m now working on getting S3 storage set up so I can back up stuff there.

http://xentek.net/articles/448/installing-fuse-s3fs-and-sshfs-on-ubuntu/

sudo aptitude install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev comerr-dev libfuse2 libidn11-dev libkadm55 libkrb5-dev libldap2-dev libselinux1-dev libsepol1-dev pkg-config fuse-utils sshfs

No related posts.

So I’ve started building out what I ended up calling bundles – basically stock distribution of such php applications as wordpress, drupal, phpBB etc using the standard maven 2 suggested layout with a pom.xml in it.

The pom.xml would look like this

Using this structure, you can put your own customization to the stock drupal code in your own maven2 project, while keeping the original source code in tact, hence giving you a easier way to manage base version upgrades etc.

Another added benefit to this is you can utilize zillions of maven2 plugins out there, including eclipse plugin to make your php app accessible in your eclipse. The most useful plugin is you can use Caucho’s resin plugin, allowing you to run your php application using quercus without needing any apache/php set up for your development environment.

No related posts.