DevOps Days Open Space: DevOps for Legacy Code and Real Servers

I proposed the topic for this OpenSpace: DevOps for Legacy Code and Real Servers.  Here are some of the insights I gleaned from this session.

Legacy servers

• Cloud platforms are evolving to manage real, legacy servers in addition to virtual machines.
• Chef, for example, can manage both physical and virtual servers.  It can also manage clean OS installations as well as update existing servers.  There's a tool called Blueprint that will attempt to reverse engineer Chef automation for an existing server.
• It's difficult to re-create systems that weren't automated in the first place.  However, it greatly reduces your risk if you invest the time and effort to do that.  What if the server was destroyed in a fire or something?
• Sometimes people have even lost the source code for applications that are running in production.  That is a very risky state to be in.
• Another option is to clone the system into a VM first, snapshot it, and then do your exploratory work on the VM.
• You can also copy some of your production web traffic to your staging servers.
• Or, you can start deploying new applications to VMs, and gradually shift your enterprise code to VMs.

Mainframe systems

• Mainframes are the backbone of many legacy systems, and they are not going away.  
• People who are used to working with mainframes have a different culture and language than people who are used to developing new web applications.  There's a communication gap to bridge before they can benefit from DevOps principles and practices.
• One option is to just get an enterprise's web applications to adopt DevOps and punt on the mainframe applications.  But why can't we do the same thing for mainframe applications?
• Mainframes have limited logging and monitoring systems.  Why?
• Mainframes have limited tooling.  Why?
• It's very difficult to see what's going in within an application.
• It's very difficult to debug applications.
• Deployments have to be completed with zero downtime.
• LPARs, CICS regions, etc. could actually be considered a type of virtualization.  Is there a way we could make them behave more like VMs?
• Could mainframe developers take some of the best practices from .Net and Java?
• A more open place, like a university, might be more willing to experiment with DevOps first.

Universal Principles
These principles from DevOps can apply to legacy servers and mainframes just as easily:

• Source Control Everything (including infrastructure code)
• Version Control Everything (including infrastructure code)
• Automate Everything (including infrastructure code)
• Test Driven Development: Test First, Test Everything
• Test for Operational Quality (performance, transaction load, security, etc.)
• Agility
• Focus on the Business Outcome, not the features or requirements
• Improve teamwork between Dev and Ops
• Collect metrics so you can find problems earlier

DevOps Days Open Space: How Can Ops Teams Give Feedback to Dev Teams?

This was another interesting Open Space that I participated in: How Can Ops Teams Give Feedback to Dev Teams?

Chaos Monkey can teach developers where things might break.  You need to couple that with some sort of monitoring tools so you can find bugs of the performance/throughput/overload type as well.

People in ops would like developers to program more defensively.  Developers are not generally taught how to do this.  It's also not usually part of their culture. 

One great way to tech developers is by writing tests that fail.  Developers are great at fixing tests that fail.

Another best practice is to embed developers in operations and vice versa.  Some companies have done this with teams of people for months or years at at time.  Others rotate people between the teams for one day every couple of weeks.  Set it up like an apprenticeship, where people can start out with a mentor and gradually become responsible for their own things.

Operations people can review code!

Developers can have pagers!

Product managers need to care about operational constraints and include those in the requirements that they put on the development teams.

You need to get everyone in the company to think about business value and happy customers.  Constantly.

You need to get everyone in the company to watch dashboards.  Give each person a few graphs to watch on a dashboard.

DevOps Days Open Space: Private Cloud Myths, Facts, and Tips

I'm at the DevOps Days conference this week, and some of my favorite sessions have been the OpenSpace collaborations.  I was the scribe for the session on "Private Cloud: Myths, Facts, and Tips", so here is what we discussed:

Myth: The cloud will not fail.
Fact: It will fail, so write software that can handle failures.  In fact, entire data centers will fail.  Have at least two geographically remote clouds that can fail over to each other.
Tip: The Yahoo guys have found that the more technology you have to prevent a data center from failing, the more points of failure you introduce into the system.  They've found that it's much more effective to make it easy to fail over to a new data center.  In fact, they fail over during peak traffic times on purpose: for practice, to build confidence, and as a way to take servers out of production for a little while to upgrade them.
Tip: Infrastructure automation makes it easy to create a new backup system.

Myth: The cloud will move my enterprise forward into the future.
Fact:  There are issues you have to solve first.
Tip: Teach your developers how to write stateless applications.
Tip: The chaos monkey can help your developers learn how to write stateless applications, and how to handle failures of different services.
Tip: Getting people to write stateless code is a cultural and a management issue.

Myth: Any workload can run in the cloud.
Fact: Some workloads will not behave in a cloud-like manner.  You have to design applications for that.
Tip: A best practice is to tell developers that their application could be moved to the public cloud at any time, without notice. 
Tip: A full penetration test on a regular basis is extremely valuable.  Especially if the output of that is a set of tests that failed, that your developers can fix!

Myth: My current ops team can run a private cloud easily.
Fact: You need to train people on how to run a cloud.  It's not traditional ops.  You have to learn to let go.  You have to give people control over their own machines.  You have to let them provision their own VMs.
Tip: Sysadmins should learn how to write infrastructure code.  Either that or they may be reduced to changing out broken servers on racks.
Tip: One way to prevent people from overloading your cloud capacity is to put fixed leases on the machines.  If you have infrastructure as code, you can provision a new system in a few minutes.  So, for developer unit test VMs, it's fine to delete those after 24 hours.  The developer may manually delete VMs even earlier.  Have a way to flag the small percentage of machines that should live for a long time.

The Women at DevOps Days

I recently posted a tweet about DevOps Days Mountain View 2012, and I feel the need to explain it a bit: https://twitter.com/DukeAMO/status/218541579219116032/photo/1

First of all, why did all of the women there that evening decide to get together, in a cage of all places?  Well, DevOps Days was held in a very odd sort of place, the Silicon Valley Cloud Center.  It's a former data center, apparently, and the evening dinner and party was basically in the garage.  For some reason, the garage is partitioned into several large cages, surrounded by chain link fences.  We thought it would be fun to get together, and the easiest place to do that was in one of the cages, because there were tables in there.  The fact that it was slightly subversive to do that was part of the fun.

Did we do that to make some sort of political statement?  No, not really.  The fact of the matter is, when you're a woman at an operations conference, it's very obvious that you're in a minority.  I haven't done any scientific studies, but I'm going to guess that the ratio of men to women attendees at both DevOps Days and the larger Velocity conference this week was around 95% to 5%.  It's a shame that more women aren't attracted to this field, because it's quite rewarding, and there's no reason for women to stay away.  But the women who do buck the trend are rare birds, and we don't mind being a little bit different.  We also get used to it.  On a day to day basis, I don't even notice that the vast majority of the people I work with are men.  It's normal.

We had our little toast together, and had a great time talking for the rest of the evening.  By the time I left, there were just as many men at our table as there were women, and we were happy to have them.

Here's to the rare birds!

Are you treating your servers like snowflakes or rubber stamps?

This is more of an agile blog, but I thought I'd cross-post this from our DevOps team blog, just in case anyone's interested:

Are you treating your servers like snowflakes or rubber stamps?
https://www.ibm.com/developerworks/mydeveloperworks/blogs/devops/entry/snowflakes_or_rubber_stamps?lang=en

I also submitted this proposal for DevOps Days Mountain View 2012.

How IBM is using DevOps to build DevOps tools
http://devopsdays.org/events/2012-mountainview/proposals/How%20IBM%20is%20using%20DevOps%20to%20build%20DevOps%20tools/

Comments on both are welcome!

How can you make remote development work?

I have a few years of experience with working from home, and several years of experience with working with cross-site teams. There are a few things that are necessary for a remote worker. A remote worker needs to be:

• Motivated
• Independent
• Self-managing
• Responsible
• Good at time management
• Excellent at communications (oral and written)
• Honest and trustworthy
• Good at ignoring distractions
• Willing to ask for help if needed

While most of these are disciplines that you can get better at with practice, some are harder to learn.

Working from home can be great, but it's a special skill, because it's easy to get distracted with things that need to be done around the house. Also, there's no one there to see if you're goofing off, so you have to motivate yourself to focus on the task at hand. It's easier to focus and be productive if you have specific tasks that are due on a regular (daily) basis. Scrum helps with this by providing a quick daily checkpoint meeting.

Ideally, remote workers should be in the same time zone as each other, or at least close to the same time zone. I've found that working with people 5 hours away (the UK, for example) is workable. You just learn to block off your mornings for communication and meetings, and do independent work in the afternoons and evenings. Working with people 7 or more hours away is painful. You end up losing days of work because you can't find someone who can help you get past a blocking issue. Perhaps you send an email describing a problem and asking for help... then the person on the other end misunderstands something... and now you're into a second lost day. We work with people in India, but they're actually on a 12-8 PM working schedule, so we at least have a couple of hours of overlapping time in the Eastern US. While you *can* call people early in the morning or late in the evening to keep things moving along, nobody likes to do that on a daily basis.

One other tip, especially for software engineers, is that it really helps to bring people into the office for a week or so, once or twice a year. First of all, face to face communication is high-bandwidth, so you can get a lot of work done in a short amount of time (planning, design work, strategy sessions, training, etc.). People tend to trust and respect each other a little more once they've met in person. They're also more comfortable asking questions, or asking each other for favors, which means people are more productive when they get home. Finally, it builds a stronger team and improves morale.

A year or two ago, multinational development teams (even within one scrum team) were the norm here. But we're trying to consolidate projects locally as much as we can now. My current project is probably 75% local and 25% remote.

Doing Planning Poker Remotely?

Has anyone out there worked out a clean way to do Planning Poker remotely? We have development teams with people in multiple locations, so we conduct our planning sessions via conference call and screen sharing.

If you're not that familiar with Planning Poker, it's a quick way to size stories without getting down to the level of person-days. Story point sizings are based on your past experience - an 8-point story should take rougly as much design/dev/test/doc effort as an 8-point story you've finished in the past. You talk for a few minutes about a story, and then everyone in the planning session does a "secret ballot" where they choose a story point sizing. Then you discuss the largest and smallest sizings, and come to a consensus on what the sizing should be. You can use the story point sizings to plan how much work you can do in an entire release, across several sprints.

There's a Planning Poker plug-in for Sametime. The plug-in was OK - you could set up a planning session with a group of people, and enter the stories to be sized. Then when you selected a story, each person could submit their estimate, and it wouldn't show the estimates until everyone had submitted. But it's often a problem for people to get the plug-in installed and working. For example, if people had the wrong version of Sametime, or if they were using a Mac, it wouldn't work for them.

Another thing we have done is a standard group chat where everyone just types in their estimates, and everyone can see them right away. There are a couple of problems with that - for one thing, some people (mostly newbies) will refuse to submit their estimates until other people (usually the team lead / scrum master) submits theirs. Also, once a few people submit their estimates, all of the other estimates seem to magically become the same number. Which means that people are changing their answers based on what they already see on the screen. Sigh.

A third thing I've seen scrum masters do is set up a second display where they have an individual chat session with each person, and no one can see the sizing estimates except for the scrum master. That works resonably well, but it's still kludgy.

Can anyone recommend a better way to do this? What do you do?