Welcome

What works and what doesn't work in software development? For the first 10 years or so of my career, we followed a strict waterfall development model. Then in 2008 we started switching to an agile development model. In 2011 we added DevOps principles and practices. Our product team has also become increasingly global. This blog is about our successes and failures, and what we've learned along the way.



The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. The comments may not represent my opinions, and certainly do not represent IBM in any way.

Friday, June 29, 2012

DevOps Days Open Space: How Can Ops Teams Give Feedback to Dev Teams?

This was another interesting Open Space that I participated in: How Can Ops Teams Give Feedback to Dev Teams?

Chaos Monkey can teach developers where things might break.  You need to couple that with some sort of monitoring tools so you can find bugs of the performance/throughput/overload type as well.

People in ops would like developers to program more defensively.  Developers are not generally taught how to do this.  It's also not usually part of their culture. 

One great way to tech developers is by writing tests that fail.  Developers are great at fixing tests that fail.

Another best practice is to embed developers in operations and vice versa.  Some companies have done this with teams of people for months or years at at time.  Others rotate people between the teams for one day every couple of weeks.  Set it up like an apprenticeship, where people can start out with a mentor and gradually become responsible for their own things.

Operations people can review code!

Developers can have pagers!

Product managers need to care about operational constraints and include those in the requirements that they put on the development teams.

You need to get everyone in the company to think about business value and happy customers.  Constantly.

You need to get everyone in the company to watch dashboards.  Give each person a few graphs to watch on a dashboard.

DevOps Days Open Space: Private Cloud Myths, Facts, and Tips

I'm at the DevOps Days conference this week, and some of my favorite sessions have been the OpenSpace collaborations.  I was the scribe for the session on "Private Cloud: Myths, Facts, and Tips", so here is what we discussed:

Myth: The cloud will not fail.
Fact: It will fail, so write software that can handle failures.  In fact, entire data centers will fail.  Have at least two geographically remote clouds that can fail over to each other.
Tip: The Yahoo guys have found that the more technology you have to prevent a data center from failing, the more points of failure you introduce into the system.  They've found that it's much more effective to make it easy to fail over to a new data center.  In fact, they fail over during peak traffic times on purpose: for practice, to build confidence, and as a way to take servers out of production for a little while to upgrade them.
Tip: Infrastructure automation makes it easy to create a new backup system.

Myth: The cloud will move my enterprise forward into the future.
Fact:  There are issues you have to solve first.
Tip: Teach your developers how to write stateless applications.
Tip: The chaos monkey can help your developers learn how to write stateless applications, and how to handle failures of different services.
Tip: Getting people to write stateless code is a cultural and a management issue.

Myth: Any workload can run in the cloud.
Fact: Some workloads will not behave in a cloud-like manner.  You have to design applications for that.
Tip: A best practice is to tell developers that their application could be moved to the public cloud at any time, without notice. 
Tip: A full penetration test on a regular basis is extremely valuable.  Especially if the output of that is a set of tests that failed, that your developers can fix!

Myth: My current ops team can run a private cloud easily.
Fact: You need to train people on how to run a cloud.  It's not traditional ops.  You have to learn to let go.  You have to give people control over their own machines.  You have to let them provision their own VMs.
Tip: Sysadmins should learn how to write infrastructure code.  Either that or they may be reduced to changing out broken servers on racks.
Tip: One way to prevent people from overloading your cloud capacity is to put fixed leases on the machines.  If you have infrastructure as code, you can provision a new system in a few minutes.  So, for developer unit test VMs, it's fine to delete those after 24 hours.  The developer may manually delete VMs even earlier.  Have a way to flag the small percentage of machines that should live for a long time.

Thursday, June 28, 2012

The Women at DevOps Days

I recently posted a tweet about DevOps Days Mountain View 2012, and I feel the need to explain it a bit: https://twitter.com/DukeAMO/status/218541579219116032/photo/1

First of all, why did all of the women there that evening decide to get together, in a cage of all places?  Well, DevOps Days was held in a very odd sort of place, the Silicon Valley Cloud Center.  It's a former data center, apparently, and the evening dinner and party was basically in the garage.  For some reason, the garage is partitioned into several large cages, surrounded by chain link fences.  We thought it would be fun to get together, and the easiest place to do that was in one of the cages, because there were tables in there.  The fact that it was slightly subversive to do that was part of the fun.

Did we do that to make some sort of political statement?  No, not really.  The fact of the matter is, when you're a woman at an operations conference, it's very obvious that you're in a minority.  I haven't done any scientific studies, but I'm going to guess that the ratio of men to women attendees at both DevOps Days and the larger Velocity conference this week was around 95% to 5%.  It's a shame that more women aren't attracted to this field, because it's quite rewarding, and there's no reason for women to stay away.  But the women who do buck the trend are rare birds, and we don't mind being a little bit different.  We also get used to it.  On a day to day basis, I don't even notice that the vast majority of the people I work with are men.  It's normal.

We had our little toast together, and had a great time talking for the rest of the evening.  By the time I left, there were just as many men at our table as there were women, and we were happy to have them.

Here's to the rare birds!

Thursday, May 24, 2012

Are you treating your servers like snowflakes or rubber stamps?

This is more of an agile blog, but I thought I'd cross-post this from our DevOps team blog, just in case anyone's interested:

Are you treating your servers like snowflakes or rubber stamps?
https://www.ibm.com/developerworks/mydeveloperworks/blogs/devops/entry/snowflakes_or_rubber_stamps?lang=en

I also submitted this proposal for DevOps Days Mountain View 2012.

How IBM is using DevOps to build DevOps tools
http://devopsdays.org/events/2012-mountainview/proposals/How%20IBM%20is%20using%20DevOps%20to%20build%20DevOps%20tools/

Comments on both are welcome!

Friday, January 27, 2012

How can you make remote development work?

I have a few years of experience with working from home, and several years of experience with working with cross-site teams. There are a few things that are necessary for a remote worker. A remote worker needs to be:
  • Motivated
  • Independent
  • Self-managing
  • Responsible
  • Good at time management
  • Excellent at communications (oral and written)
  • Honest and trustworthy
  • Good at ignoring distractions
  • Willing to ask for help if needed

While most of these are disciplines that you can get better at with practice, some are harder to learn.

Working from home can be great, but it's a special skill, because it's easy to get distracted with things that need to be done around the house. Also, there's no one there to see if you're goofing off, so you have to motivate yourself to focus on the task at hand. It's easier to focus and be productive if you have specific tasks that are due on a regular (daily) basis. Scrum helps with this by providing a quick daily checkpoint meeting.

Ideally, remote workers should be in the same time zone as each other, or at least close to the same time zone. I've found that working with people 5 hours away (the UK, for example) is workable. You just learn to block off your mornings for communication and meetings, and do independent work in the afternoons and evenings. Working with people 7 or more hours away is painful. You end up losing days of work because you can't find someone who can help you get past a blocking issue. Perhaps you send an email describing a problem and asking for help... then the person on the other end misunderstands something... and now you're into a second lost day. We work with people in India, but they're actually on a 12-8 PM working schedule, so we at least have a couple of hours of overlapping time in the Eastern US. While you *can* call people early in the morning or late in the evening to keep things moving along, nobody likes to do that on a daily basis.

One other tip, especially for software engineers, is that it really helps to bring people into the office for a week or so, once or twice a year. First of all, face to face communication is high-bandwidth, so you can get a lot of work done in a short amount of time (planning, design work, strategy sessions, training, etc.). People tend to trust and respect each other a little more once they've met in person. They're also more comfortable asking questions, or asking each other for favors, which means people are more productive when they get home. Finally, it builds a stronger team and improves morale.

A year or two ago, multinational development teams (even within one scrum team) were the norm here. But we're trying to consolidate projects locally as much as we can now. My current project is probably 75% local and 25% remote.

Monday, August 16, 2010

Doing Planning Poker Remotely?

Has anyone out there worked out a clean way to do Planning Poker remotely? We have development teams with people in multiple locations, so we conduct our planning sessions via conference call and screen sharing.

If you're not that familiar with Planning Poker, it's a quick way to size stories without getting down to the level of person-days. Story point sizings are based on your past experience - an 8-point story should take rougly as much design/dev/test/doc effort as an 8-point story you've finished in the past. You talk for a few minutes about a story, and then everyone in the planning session does a "secret ballot" where they choose a story point sizing. Then you discuss the largest and smallest sizings, and come to a consensus on what the sizing should be. You can use the story point sizings to plan how much work you can do in an entire release, across several sprints.

There's a Planning Poker plug-in for Sametime. The plug-in was OK - you could set up a planning session with a group of people, and enter the stories to be sized. Then when you selected a story, each person could submit their estimate, and it wouldn't show the estimates until everyone had submitted. But it's often a problem for people to get the plug-in installed and working. For example, if people had the wrong version of Sametime, or if they were using a Mac, it wouldn't work for them.

Another thing we have done is a standard group chat where everyone just types in their estimates, and everyone can see them right away. There are a couple of problems with that - for one thing, some people (mostly newbies) will refuse to submit their estimates until other people (usually the team lead / scrum master) submits theirs. Also, once a few people submit their estimates, all of the other estimates seem to magically become the same number. Which means that people are changing their answers based on what they already see on the screen. Sigh.

A third thing I've seen scrum masters do is set up a second display where they have an individual chat session with each person, and no one can see the sizing estimates except for the scrum master. That works resonably well, but it's still kludgy.

Can anyone recommend a better way to do this? What do you do?

Friday, June 4, 2010

Working with a remote team

I'm having a little adventure this month - we just hired five contractors in Bucharest, Romania to work on my current project. They're new to our product, and we have to get them to a point where they're doing productive work very quickly. It's a challenge for sure.

I've worked with remote teams before, and this experience is re-enforcing a few rules of thumb that I have:
  • Time zones matter. Romania is 7 hours ahead of us, which means that my day starts at 9 AM and theirs ends at 10 AM. So, we've blocked off 9-10 each day to talk to each other. This is significantly easier than working with teams in India or China, where the difference is close to 12 hours. (Meetings at 8 PM? 7 AM?) However, it's significantly more difficult than working with teams in Brazil, which are only 1 hour off, or even teams in Ireland, which are 5 hours off. The Romanian team is working while I'm asleep, so I have to make sure they all have enough to do before I stop working the night before. Also, if they get stuck on something, it's easy to lose a day. So to keep them productive, ideally each of them should have a few things they could be working on. That way if they're blocked on one task they can work on another until we sync up again.
  • You should be in the same building as the people you work with most closely. Yes, you can talk to someone 4-5 times a day over the phone, but it's significantly more efficient if you can talk in person. Especially if your work sometimes involves drawing things, like design diagrams or GUI prototypes. It's also very helpful if your work involves making difficult decisions, because you can read people's body language in person. Reading body language can help you see if someone is sure or unsure of what they're saying.
  • Teach the remote team to be as independent as possible. If they get stuck on something, they need to be able to work past the problem on their own, so you don't lose a day. Also, it helps if they're working on their own set of files/features, so they're not interfering with people who they can't communicate with immediately.
  • Language skills are critical. One of the worst feelings is explaining something to a technical team on the phone, and hearing crickets in response. Did they understand what you said or not? Are they talking to each other on mute? Even a thick accent can make communication difficult, because you may have to ask someone to repeat themselves over and over. After a few tries, people may give up and try to guess what the other person is saying. That can lead to misunderstandings. You can mitigate this somewhat by speaking more slowly and carefully, and by rephrasing what you heard the other person say. But this just slows down communication, so it's best to hire people you can communicate with easily in the first place.
  • Meeting people in person, even once, helps a lot. There are people that I spent a week with in training sessions three years ago. I still go to them first with a problem, and they are more likely to help me than someone I've never met. And vice versa. In some ways, I feel like I know them better than people I've worked with for years over the phone. Meeting people face to face naturally creates a certain comfort level and familiarity that's very difficult to develop otherwise.
  • Training takes a lot of time and money. This sounds obvious, but it takes a long time to get a new set of people up to speed. On our product this ranges from a month to a year or more, depending on the specific assignment. The hidden cost is that the people doing the training are not able to spend as much time on their other assignments. This is an argument for retaining existing employees whenever possible! Also, when searching for contractors or new hires, you should put a high premium on people who already have a skill set close to what you need, even if they cost more. One other tip in this area - the time to train five or six people at once is similar to the time required to train one or two, so you should try to add people to a project in batches, not one at a time.
  • Record your training and demo sessions. We record most of our training sessions and demos (end of sprint demos and customer demos). We use Lotus Live/Sametime Unyte/Webdialogs, or Camtasia Studio, to record screen sharing along with the sound from the conference call. We post them to a shared server and link to them from our development wiki. We use those recorded sessions over and over again. They come in handy for training new developers and testers, creating beta scenarios, training sales and support teams, creating new training classes, etc. We get pretty good reuse out of the slide decks too.

Anyone else have some favorite tips? Maybe you could help me!