This was another interesting Open Space that I participated in: How Can Ops Teams Give Feedback to Dev Teams?
Chaos Monkey can teach developers where things might break. You need to couple that with some sort of monitoring tools so you can find bugs of the performance/throughput/overload type as well.
People in ops would like developers to program more defensively. Developers are not generally taught how to do this. It's also not usually part of their culture.
One great way to tech developers is by writing tests that fail. Developers are great at fixing tests that fail.
Another best practice is to embed developers in operations and vice versa. Some companies have done this with teams of people for months or years at at time. Others rotate people between the teams for one day every couple of weeks. Set it up like an apprenticeship, where people can start out with a mentor and gradually become responsible for their own things.
Operations people can review code!
Developers can have pagers!
Product managers need to care about operational constraints and include those in the requirements that they put on the development teams.
You need to get everyone in the company to think about business value and happy customers. Constantly.
You need to get everyone in the company to watch dashboards. Give each person a few graphs to watch on a dashboard.
Friday, June 29, 2012
DevOps Days Open Space: Private Cloud Myths, Facts, and Tips
I'm at the DevOps Days conference this week, and some of my favorite sessions have been the OpenSpace collaborations. I was the scribe for the session on "Private Cloud: Myths, Facts, and Tips", so here is what we discussed:
Myth: The cloud will not fail.
Fact: It will fail, so write software that can handle failures. In fact, entire data centers will fail. Have at least two geographically remote clouds that can fail over to each other.
Tip: The Yahoo guys have found that the more technology you have to prevent a data center from failing, the more points of failure you introduce into the system. They've found that it's much more effective to make it easy to fail over to a new data center. In fact, they fail over during peak traffic times on purpose: for practice, to build confidence, and as a way to take servers out of production for a little while to upgrade them.
Tip: Infrastructure automation makes it easy to create a new backup system.
Myth: The cloud will move my enterprise forward into the future.
Fact: There are issues you have to solve first.
Tip: Teach your developers how to write stateless applications.
Tip: The chaos monkey can help your developers learn how to write stateless applications, and how to handle failures of different services.
Tip: Getting people to write stateless code is a cultural and a management issue.
Myth: Any workload can run in the cloud.
Fact: Some workloads will not behave in a cloud-like manner. You have to design applications for that.
Tip: A best practice is to tell developers that their application could be moved to the public cloud at any time, without notice.
Tip: A full penetration test on a regular basis is extremely valuable. Especially if the output of that is a set of tests that failed, that your developers can fix!
Myth: My current ops team can run a private cloud easily.
Fact: You need to train people on how to run a cloud. It's not traditional ops. You have to learn to let go. You have to give people control over their own machines. You have to let them provision their own VMs.
Tip: Sysadmins should learn how to write infrastructure code. Either that or they may be reduced to changing out broken servers on racks.
Tip: One way to prevent people from overloading your cloud capacity is to put fixed leases on the machines. If you have infrastructure as code, you can provision a new system in a few minutes. So, for developer unit test VMs, it's fine to delete those after 24 hours. The developer may manually delete VMs even earlier. Have a way to flag the small percentage of machines that should live for a long time.
Myth: The cloud will not fail.
Fact: It will fail, so write software that can handle failures. In fact, entire data centers will fail. Have at least two geographically remote clouds that can fail over to each other.
Tip: The Yahoo guys have found that the more technology you have to prevent a data center from failing, the more points of failure you introduce into the system. They've found that it's much more effective to make it easy to fail over to a new data center. In fact, they fail over during peak traffic times on purpose: for practice, to build confidence, and as a way to take servers out of production for a little while to upgrade them.
Tip: Infrastructure automation makes it easy to create a new backup system.
Myth: The cloud will move my enterprise forward into the future.
Fact: There are issues you have to solve first.
Tip: Teach your developers how to write stateless applications.
Tip: The chaos monkey can help your developers learn how to write stateless applications, and how to handle failures of different services.
Tip: Getting people to write stateless code is a cultural and a management issue.
Myth: Any workload can run in the cloud.
Fact: Some workloads will not behave in a cloud-like manner. You have to design applications for that.
Tip: A best practice is to tell developers that their application could be moved to the public cloud at any time, without notice.
Tip: A full penetration test on a regular basis is extremely valuable. Especially if the output of that is a set of tests that failed, that your developers can fix!
Myth: My current ops team can run a private cloud easily.
Fact: You need to train people on how to run a cloud. It's not traditional ops. You have to learn to let go. You have to give people control over their own machines. You have to let them provision their own VMs.
Tip: Sysadmins should learn how to write infrastructure code. Either that or they may be reduced to changing out broken servers on racks.
Tip: One way to prevent people from overloading your cloud capacity is to put fixed leases on the machines. If you have infrastructure as code, you can provision a new system in a few minutes. So, for developer unit test VMs, it's fine to delete those after 24 hours. The developer may manually delete VMs even earlier. Have a way to flag the small percentage of machines that should live for a long time.
Thursday, June 28, 2012
The Women at DevOps Days
I recently posted a tweet about DevOps Days Mountain View 2012, and I feel the need to explain it a bit: https://twitter.com/DukeAMO/status/218541579219116032/photo/1
First of all, why did all of the women there that evening decide to get together, in a cage of all places? Well, DevOps Days was held in a very odd sort of place, the Silicon Valley Cloud Center. It's a former data center, apparently, and the evening dinner and party was basically in the garage. For some reason, the garage is partitioned into several large cages, surrounded by chain link fences. We thought it would be fun to get together, and the easiest place to do that was in one of the cages, because there were tables in there. The fact that it was slightly subversive to do that was part of the fun.
Did we do that to make some sort of political statement? No, not really. The fact of the matter is, when you're a woman at an operations conference, it's very obvious that you're in a minority. I haven't done any scientific studies, but I'm going to guess that the ratio of men to women attendees at both DevOps Days and the larger Velocity conference this week was around 95% to 5%. It's a shame that more women aren't attracted to this field, because it's quite rewarding, and there's no reason for women to stay away. But the women who do buck the trend are rare birds, and we don't mind being a little bit different. We also get used to it. On a day to day basis, I don't even notice that the vast majority of the people I work with are men. It's normal.
We had our little toast together, and had a great time talking for the rest of the evening. By the time I left, there were just as many men at our table as there were women, and we were happy to have them.
Here's to the rare birds!
First of all, why did all of the women there that evening decide to get together, in a cage of all places? Well, DevOps Days was held in a very odd sort of place, the Silicon Valley Cloud Center. It's a former data center, apparently, and the evening dinner and party was basically in the garage. For some reason, the garage is partitioned into several large cages, surrounded by chain link fences. We thought it would be fun to get together, and the easiest place to do that was in one of the cages, because there were tables in there. The fact that it was slightly subversive to do that was part of the fun.
Did we do that to make some sort of political statement? No, not really. The fact of the matter is, when you're a woman at an operations conference, it's very obvious that you're in a minority. I haven't done any scientific studies, but I'm going to guess that the ratio of men to women attendees at both DevOps Days and the larger Velocity conference this week was around 95% to 5%. It's a shame that more women aren't attracted to this field, because it's quite rewarding, and there's no reason for women to stay away. But the women who do buck the trend are rare birds, and we don't mind being a little bit different. We also get used to it. On a day to day basis, I don't even notice that the vast majority of the people I work with are men. It's normal.
We had our little toast together, and had a great time talking for the rest of the evening. By the time I left, there were just as many men at our table as there were women, and we were happy to have them.
Here's to the rare birds!
Subscribe to:
Posts (Atom)