I've been reading about slow growth for Rackspace cloud and how apps are pulling support. I shouldn't be so surprised given that my own usage of the Rackspace cloud has also dwindled, despite the ORD datacenter being one of the most rock solid facilities I have ever used. I know that Rackspace has spent the last few years working hard and innovating, but somehow they seem to still be missing the boat. Here is a list of key things that made me go back to AWS and that Rackspace can implement to reverse this trend.

1. Bring Your Own Kernel/Image - One of the big issues I have with Rackspace is flexibility. This is also an area where AWS shines. When it comes to vm images, AWS has built a marketplace around them. There are thousands. Rackspace has a 27 starter images in their "First Generation" platform and 49 images in their "2nd Generation".

2. Get rid of storage templates - Rackspace's new Open Stack storage lets you add block storage from 100GB-1024GB in size. On AWS I can allocate pretty much any size I want.

3. Notify Customers When Logging Into an Instance - One of the eerie things about using Rackspace cloud is that I randomly get logins from Rackspace staff. They have a few daemons they set up and like to run on the instance. Since I use Arch Linux and I keep it up to date, their daemon sometimes gets turned off due to dependencies it has getting upgraded (they don't use a package!). So every once in a while I log into an instance and see someone's been there, tinkering with the daemon and getting it to run. I wouldn't mind this if there was communication. Or maybe I would. I would rather get a ticket "Please run this software on your instances or ask us if you'd like us to install it."

4. Add an API for Health Checks/Failover - Provide basic scripting with the monitoring so that failover can be initiated. (Short cut: Buy a DNS provider like Ultra or Dyn)

5. Mirror Databases in Both Datacenters - Let me configure MySQL replication between DFW and ORD.

Being able to copy cloudfiles and machine images between the two datacenters (which AWS introduced a few months back) is very handy for building resilient distributed infrastructure as well, but I think that the list of five items above would go a long way to stem the bleeding. Since Rackspace does have more memory options, it could prove to be the more flexible platform. While I'm at it though, three more wish list items:

6. Unify the US and UK Control Panels - Having to log in and manage Rackspace UK separately is a pain.

7. Multi-Factor Authentication for the Console - there are many ways to get MFA into your infrastructure but the management console is still MFA-less. I have read that Rackspace is working on this, and they may very well be working on other items on this list.

8. Encrypted Server Images - Since there is console support you could completely secure an instance. This isn't practical for every application. I encrypt a lot of my storage at Amazon, but since AWS doesn't have interactive console support, the boot chain remains unencrypted. This is another way that Rackspace could pull ahead, although much like the templating on OS and block storage it seems like Rackspace is really tied to the philosophy of managed services instead of manage your own. Encrypting the images could become a hurdle to upgrading to managed services.

The pace of innovation coming out of Amazon is really amazing for a company that size. I know many hard working people at Rackspace as well and they aren't standing still. Maybe the difference is a little marketing, but a few of the items on my list point to a shift in perspective on the Rackspace cloud product offering itself. An "open" cloud behind a set of pre-defined templates doesn't seem to be all that flexible or fun.

Harvard Business Review has published an essay about the "sandwich approach" to giving negative feedback by Roger Schwarz. The essay takes issue with the practice of providing positive and negative feedback together. While I wouldn't say that I stick to the positive-negative-positive format of the sandwich, I do try to balance feedback when I need to address an issue with a co-worker or employee. I try to find something positive to say that is in proportion to the negative points that I have to get across. Something light for a small issue, or larger praise to accompany the discussion and resolution of a larger issue.

The article states that this waters down and undermines the message, and I can understand that. Sometimes people only hear the good things and ignore the "room for improvement".

The method that the article encourages is a "mutual learning" approach that encourages the employee to particiate in creating the structure of the discussion and the resolution:

I want to start by describing what I saw that raised my concerns and see if you saw the same things. After we agree on what happened, I want to say more about my concerns and see if you share them.

The "sandwich" approach is painted as "unilateraly controlling". I agree with that terminology in situations where the negative feedback and resolution is laid out by the manager. Typically when I address an issue, I bring collaboration into the resolution, my philosophy being that as a manager I am there to help them understand why it needs to be fixed and how to fix it.

Managers do make other mistakes with the sandwich approach. They may not keep the positive in league with the negative in terms of tone and relevant size, and they may be uncomfortable with the negative part of the conversation to the point where they minimize and gloss over it, instead of ensuring that the person heard and understood the feedback. They also may not work out a resolution and set expectations of how to measure the improvement.

In defense of my balance method I would offer an additional point. Many roles in the teams that I work with in the technology field are creative ones. Getting the most out of creative people means leading them through inspiration and not just task driving. For those people, providing negative feedback in a direct manner is quite a buzz kill and I have seen them take a few days to get their stride back.

Another issue that I have with the proposed "mutual learning" approach is that when explaining an issue or observation, asking if the person agrees puts them on the defensive. Hearing excuses then escalates my assertiveness. While I am sure that some HR policies create a rigid framework for reviews, and that there may be bad managers out there that are dictators, situations like this should never be one-way discussions. At every step the person receiving the feedback should be able to make their points, and I would frame this as a conversation and open dialog.

Overall the message of the article is something I am very in tune with - being transparent as a manager. In these kind of situations, making the process transparent and not masking the feedback in either dimished or amplified persepective is the key to improvement. It also helps the person being managed learn and grow themselves.

Update April 2, 2013: New Version

The AMI version and package listing is now available from the sidebar on the blog.

For detailed information, please see this page

I have an updated Arch Linux image for Amazon EC2 that is systemd. I created a boot script that sets the hostname and root keys. It will even update DNS in Route53 and send you an email letting you know the instance IP.

Released under the MIT license on github.

I am working on cleaning up the base image that I use on Amazon EC2 and publishing the AMI as well.

I've been running Arch Linux on my workstations and on servers for a long time. Every once in a while I see a debate in an Arch Linux forum about it's suitability for use on production servers. Being a rolling release distribution, it is different than other distributions that concentrate on enterprise and long-term support like RedHat Enterprise and CentOS. Without getting too much into the pros and cons - one of the key reasons that I use Arch on servers is earlier access to newer technologies like the 3.0 Linux kernel series (with built-in xen support). Overall, though, it is due to my familiarity with and love for it. The OS that I load on my servers is there to support my applications. I find Arch is simple and light yet thorough and stable in getting the job done. If you are running Arch on servers or are interested in doing so, here are some practices that I recommend.

This is not meant to be an exhaustive list and there are different approaches to systems administration. I welcome feedback and discussion of these concepts. I have seen projects centered around creating an off-shoot of Arch to use on servers. Ultimately I think they miss the point. The idea is not to make Arch act like CentOS. With some simple tweaks to your deployment and management process, Arch is a fine distro to use on servers.

Dealing with "Rolling Release"

Many of these things apply not just to Arch, but any rolling release distribution. Recently, Arch Linux has gone through some fundamental changes in the base layers of the operating system like the network configuration and system initialization. Updating needs to be a regular and intelligent process. You can automate much of it but you really should do the base updates manually.

1. Have a server in each datacenter or cloud that acts as a "base" server for testing updates. Always have a server that you can test updates on. I use Linode, Rackspace, and Amazon EC2 clouds and I have dev servers in each of those that are there to test updates on so that I can work out any issues before updating mission-critical instances. Once you update the base server, you can image it out appropriately for your environment.

2. Keep Snapshots so that you can "roll back". This is one of those things you should be doing no matter what you are using.

3. Update often. I run updates weekly on my workstations and weekly or bi-weekly on my servers. With a rolling release distribution, the more out of date you are, the more work you have to do with each update. If you don't have a proper environment for testing updates, make an image of your server and run updates against that in something like Virtualbox.

4. Watch the News/Forums/Mailing Lists for Update Issues. I update my workstations first and run that for a few days before updating my servers - unless it is a critical security issue. Package updates fixing security issues should always be done as soon as is practically possible.

5. Don't Run Updates via Daemon or Cron. I do not recommend running system updates via cron. You just never know when an update will require more than just the basics. If you are pushing your own applications via custom repositories, those can be automated if appropriate. (See Custom Repositories below)

6. Script tricky updates. A quick bash script can make updates against servers very simple and painless. I typically run my updates from one spot. Configuration management tools can help here too. (See Configuration Management below)

7. Remove pacman from SyncFirst and HoldPkg in /etc/pacman.conf. The default pacman.conf will stop and prompt to update pacman if there is a new version of it before updating the system. For workstations this is fine, but for servers or when you are running scripted updates, this will get in the way. If you are updating your workstations first and the server last, you will know if you need to update pacman first.

8. Create scripts to bring machines current from your vendor's image. Ideally you are running your own images made from your own base instances, but if you are using the vendor's images - such as the "Arch Linux" installs from Rackspace or Linode, you should have a script that takes that image and brings it current. This script needs to be tested and updated regularly as part of your update cycle.

Understanding the Philosophy

The key to running any operating system on your servers is understanding it and the philosophy behind it. Arch Linux is a lot like python. The driving philosophy is simplicity. Many times people over-think or assume that a given task is harder than it is.

9. Run Arch as your daily driver. Nothing will bring your knowledge up like interacting with the system on a daily basis. Linux is a great deskop OS, give Arch a try.

Also see The Arch Way entry in the Arch Linux wiki.

Custom Repositories

10. Keep private repositories in their own conf file. Instead of the main pacman.conf file, you can create your own configuration file and call pacman with --config filename. This allows you to update the packages from your repo independently of system updates.

11. SSL and ACL protect private repos. I put my custom repos behind a classic ACL username and password. Yes this is present in the .conf file URL in plaintext, but I can always change the ACL if I find it gets compromised. Using SSL will protect it in transit. Of course this isn't foolproof so if you are concerned about your proprietary packages leaking, watch the logs or load the sensitive packages in a different way.

12. Sign your packages. Arch's package management system now supports package signing and verification.

13. Keep workstation and server repos separated. I build custom packages that I use on workstations and servers. I like to keep them in different repos so that the server repo stays as clean as possible.

Configuration Management

14. Configuration Management is your friend. If you are managing multiple servers, configuration management tools like Puppet, Chef, or CFEngine are your friend. Employing one of these tools properly will keep your servers consistent and greatly ease management and deployment.

Again, these practices are not meant to be all-encompassing. There are probably many other things that could go into this, but I hope sharing my approach can help others. The Arch Linux Wiki, Forums, and IRC Channels are always helpful resources.