tag:blogger.com,1999:blog-44540055983091940072024-03-17T20:03:15.809-07:00AgileFallAnonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-4454005598309194007.post-76953854897069212322014-11-10T13:24:00.000-08:002014-11-10T13:40:17.400-08:00How to get NFS shares working on Softlayer "minimal install" RHEL servers<p>If you choose the Red Hat v6 "minimal install" for your Softlayer RHEL servers, the libraries needed to use the NFS space (samba) are not installed. To install them, log in and type:</p>
<p><code>sudo yum install samba-client samba-common cifs-utils</code></p>
<p>Accept the installation, then mount the drive. For example:</p>
<p><code>mkdir /mnt/shared</p>
<p>mount -t cifs -o username=WWWW,password=XXXX //nas301.service.softlayer.com/YYYY /mnt/shared</code></p>
<p>That's it!</p>
<p>Note: The username has been replaced by WWW, the password has been replaced by XXXX, and the account has been replaced by YYYY. The nas301 server might even be replaced by a different one for your account. Get the details for your own NFS shares, or order additional storage, by logging into SoftLayer and going here: <a href="https://control.softlayer.com/storage/file">https://control.softlayer.com/storage/file</a> or, in the old portal, here: <a href="https://manage.softlayer.com/NetworkStorage/summaryForNasType/NAS">https://manage.softlayer.com/NetworkStorage/summaryForNasType/NAS</a>.</p>Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com6tag:blogger.com,1999:blog-4454005598309194007.post-91127938758684298382014-08-19T11:08:00.000-07:002014-08-19T11:08:35.469-07:00Food Network's Disturbing Breach of PrivacyI found a recipe on Food Network that I wanted to try later, so I clicked on the "Save Recipe" button. It asked me to log in or create an account, so I decided to try the "Log In With Facebook" option. Here are the permissions it asked for:
"Food Network will receive the following info: your public profile, friend list, email address, custom friends lists, birthday, current city, photos and personal description and your friends' religious and political views."
No, no, no, no, and no! Food Network, you should be ashamed of yourself.Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com2tag:blogger.com,1999:blog-4454005598309194007.post-89881783071426136202014-08-08T13:25:00.001-07:002017-10-23T23:55:46.574-07:00SoftLayer Gotchas, Tips, and TricksNOTE: THIS IS OUTDATED!
<br />
Here are many things I learned about SoftLayer the hard way, through trial and error. Of course, SoftLayer's offerings are evolving, so some of these "facts" will change over time, but reading through this list should still help.<br />
<br />
Some of this is RHEL-centric, because I do the majority of my SoftLayer work on RHEL.<br />
<br />
<h1 dir="ltr">
Useful links</h1>
<div>
<ul dir="ltr">
<li><strong>"Old Portal"</strong> <a href="https://manage.softlayer.com/">https://manage.softlayer.com/</a> and <strong>"New Portal" a.k.a "Customer Portal"</strong> <a href="https://control.softlayer.com/">https://control.softlayer.com/</a> They're still in the process of migrating functionality from the "Old Portal" to the "New Portal", so sometimes you have to jump back to the old one.</li>
<li>Getting Started: <a href="http://knowledgelayer.softlayer.com/gettingstarted">http://knowledgelayer.softlayer.com/gettingstarted</a></li>
<li>SoftLayer Development Network: <a href="http://sldn.softlayer.com/">http://sldn.softlayer.com/</a></li>
</ul>
</div>
<h1 dir="ltr">
Support</h1>
<ul dir="ltr">
<li>SoftLayer tech support via the online ticketing system is very good. I open tickets whenever I get stuck, and they get back to me quickly. They also know what they're talking about; no high school students running the help desk here.</li>
</ul>
<h1 dir="ltr">
Networking</h1>
<ul dir="ltr">
<li>You can't run the SoftLayer SSL VPN and another VPN client at the same time. It will seem like it's working, but you won't be
able to connect to the SoftLayer systems.</li>
<li>Use the SoftLayer private network to administer your systems. It's more secure and saves money.</li>
<li>If you're using the private network, you must have the private network on eth0, and also have a static route:<strong> </strong>10.0.0.0/8 via your private subnet gateway server. If you don't have that static route, you won't be able to connect to other SoftLayer servers on the private network and vice-versa. SoftLayer compute instances and bare-metal servers have that route by default unless you re-install the OS yourself. Usually this only happens if you're running a hypervisor within SoftLayer and creating your own VMs there.</li>
<li>If you're deploying CCIs or VMs or servers with a public Internet
connection, you must never enable root SSH access with an easily guessed
id/password, even for a minute! The machine will be hacked very
quickly, by Internet worms that try to log in to all available IP
addresses with common IDs and passwords. Monitor bandwidth usage
regularly to ensure that none of the systems are consuming huge amounts
of bandwidth; this is a symptom of an Internet worm that has managed to
install itself.</li>
<li>If you need to ensure that some of your servers are on the same
VLAN, use SAN storage, not a local disk, for the primary disk. If you
use a local disk, your server will probably be assigned to a different
VLAN, and you won't be able to move it later. </li>
<li>If you want multiple bare-metal servers to be on the same VLAN, order all of them at the same time and be consistent with your storage configuration (local vs. SAN disks).</li>
<li>VLAN spanning: By default, servers on different private VLANs can't
communicate with each other. There's a setting where you can enable
VLAN spanning, and then all of your private VLANs will be able to
communicate with each other (regardless of data center). </li>
<li>When creating new servers, you don't get a choice of subnet (only
the VLAN). If you want to assign servers to specific subnets, request
Portable IP Subnets (public or private) and use those instead. </li>
<li>Use the
subnet details page (https://control.softlayer.com/network/subnets/*) to "sign out" portable IP addresses by entering the hostnames
in the Notes field. </li>
<li>For public IP addresses, you can set up Reverse DNS from the same subnet details page.</li>
<li>You can add a dedicated hardware firewall in front of any public
subnets, and configure firewall rules.</li>
<li>If you want very flexible control over your firewall and VPN access, you can add a Vyatta gateway (or an HA pair of Vyatta gateways) instead. Each Vyatta gateway can manage 1 "pod", where a pod is all of your own VLANs behind the same router pair (public + private) in the same datacenter.</li>
<li>ESX servers will only be connected to the SoftLayer private
network. Their public NICs are disabled. This was never a problem for us.</li>
<li>The private SoftLayer network is not intended for large file
transfers. It is a shared resource, and file transfer speeds can be
slow at times. The public network will give you faster file
transfers.</li>
</ul>
<h1 dir="ltr">
Compute Instances (a.k.a. CCIs)</h1>
<ul dir="ltr">
<li>These are essentially VMs running on a Xen hypervisor.</li>
<li>If you try to install an unsupported OS (such as RHEL 6.4) on a
CCI, you may make the CCI unusable. You will also not be able to backup
and restore the CCI. It's a bad idea.</li>
<li>On Xen, dmidecode doesn't work. </li>
<li>You can't install VMWare drivers on Xen. You can, however, order a VCenter CCI from SoftLayer and use that to manage your SoftLayer ESX servers.</li>
<li>The maximum size for the primary disk is 100 GB. You can add additional disks if needed.</li>
<li>LVM is not supported. If you try to use LVM to make multiple disks
look like a single partition, you may make the CCI unusable. You will
also not be able to back and restore the CCI. This is also a bad idea.</li>
<li>Provisioning time may vary between datacenters. When ordering a CCI, you can select an option to deploy it to the first available datacenter if provisioning speed is more important than where it ends up.</li>
</ul>
<h1 dir="ltr">
Bare Metal Servers</h1>
<ul dir="ltr">
<li>If you install the available AFP firewall, it will probably conflict with iptables unless you know what you're doing.</li>
<li>Bare-metal servers are not available with RHEL. CentOS is available.</li>
<li>If you don't see the server configuration you want in the web order form, you may be able to order exactly what you want by starting up a sales chat.</li>
<li>If you try to install an unsupported OS (such as RHEL 6.4) on a
bare metal server, you will not be able to backup and restore the
server. This is a bad idea.</li>
</ul>
<h1 dir="ltr">
Red Hat Tips</h1>
<ul dir="ltr">
<li>Bare-metal servers are not available with RHEL. CentOS is available. Some of our software requires RHEL, and in some cases, we have been able to work around this by installing the prereq packages from the CentOS repo before installing that software.</li>
<li>To add a static route to the private network on RHEL/CentOS: add the route to /etc/sysconfig/network-scripts/route-eth0 so the setting will be persisted, and also run "route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0" to enable it immediately. Google it for details.</li>
<li>OS options are somewhat limited; RHEL 6 will always be the latest version of RHEL SoftLayer supports (currently 6.5).</li>
</ul>
Hope this helps! I welcome your comments below.<br />
<ul dir="ltr">
</ul>
Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com6tag:blogger.com,1999:blog-4454005598309194007.post-89165082721440105172014-05-16T16:48:00.000-07:002014-05-16T16:48:26.009-07:00Bootstrap scripts for SoftLayer: Chef, Eureka, Openstack Devstack, Tomcat<div aria-label="Bootstrap scripts for SoftLayer: Chef, Eureka, Devstack, Tomcat" class="entryContentContainer blogsWrapText" role="article" user_content="true">
<div dir="ltr">
I've published sample scripts that can be
used to bootstrap some popular software packages. The samples are AS-IS open source (not supported by IBM). </div>
<div dir="ltr">
<br /></div>
<div dir="ltr">
The samples out there right now are: Chef
workstation, Chef Solo configuration (solo.rb, node and role files),
Eureka (NetflixOSS), Devstack (OpenStack developers' distribution), and
Apache Tomcat 7. The Eureka and Tomcat bootstrap scripts also use Chef
Solo, while the DevStack script is just a straight .sh script.</div>
<div dir="ltr">
<br /></div>
<div dir="ltr">
If you tweak the scripts a little to use your own
information and provide pointers to them in SoftLayer's "provision
script URL" field, your systems will be automatically configured for you
when create them or re-image them.</div>
<div dir="ltr">
<a href="https://github.com/amfred/softlayerbootstrap">https://github.com/amfred/softlayerbootstrap</a> << See the README.md file for details.</div>
</div>
Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com2tag:blogger.com,1999:blog-4454005598309194007.post-10935572798969901132013-08-21T14:00:00.000-07:002013-08-21T14:09:27.693-07:00How to make life easier for your remote employeesI've already written one post about <a href="http://agile-fall.blogspot.com/2012/01/how-can-you-make-remote-development.html" target="_blank">setting up teams with remote workers</a>. However, I didn't really focus on cultural changes that employees who are in the regular office can accept to make life easier and more efficient for their remote colleagues. Cultural changes are always difficult, but these are worth the effort if you want your entire team to be productive:<br />
<br />
<br />
<ul>
<li><b>Be available for communication.</b> People are going to have questions, and if they're remote, they can't walk down the hall to ask them. Be available via chat, instant message, text message, and phone. Check your email. Respond to messages from remote employees at least as quickly as messages from locals, and remember that remote employees can't just stop by if it's urgent. It's great to set aside a couple of hours each day when you won't be interrupted, but make sure there are plenty of hours when your remote workers can reach you. Set up your "do not disturb" time so it's either a time when your remote employees are not working, or when they are also on their "do not disturb" time.</li>
<li><b>Be available in off hours, especially when working across time zones. </b>Most remote employees will be very respectful of your working hours, and will only call you in off hours if they're stuck, and then they'll keep it quick. If they do not have the freedom to call or text message you in off hours, they will lose hours of productive time on a regular basis. They will learn to stop asking questions, and either spend the time trying to figure it out themselves using the Internet, or implement something that may or may not be what you want, and ask for feedback on it the next day. Worried that you'll end up working too much in off hours? Cross that bridge if you come to it, and allow for flexible schedules.</li>
<li><b>Allow for flexible schedules.</b> If someone ends up working late one night, they should be able to leave early or come in late another day that week. Also, consider whether people in different time zones should work a shifted schedule. For example, I've worked with teams in India that worked from 12-8 PM every day, so they would be at work for at least a few hours when the U.S. employees were at work. Now that I'm living in California, I'll work an hour very early in the morning to sync up with people in Europe and on the east coast of the U.S., then I'll take a couple of hours off to get the kids off to school, then I'll finish up my work day. My "do not disturb" time is in the late afternoon, when my colleagues are not working. I also try to schedule all of my personal appointments late in the day. Sometimes I work from 10-11 PM to sync up with people in China. Flexible schedules are great as long as people are available for communication when the rest of the team is working.</li>
<li><b>Be careful about setting meeting times. </b>Keep meetings short, small, and focused. Set meetings during everyone's working hours, or take turns having meetings during your off hours and during others' off hours. Consider whether it's better to schedule a meeting or pick up the phone.</li>
<li><b>Be careful about who you invite to meetings. </b>Again, keep your meetings small. But if you're having meetings where your remote colleagues have something to add, then invite them and make sure you provide facilities for them to join in (such as a good conference calling phone system, plus screen sharing). Retrospectives and planning meetings, meetings where you set team policies, and town hall meetings are good examples.</li>
<li><b>Avoid using the mute button during conference call discussions.</b> If it's a one-way presentation, then it's fine for everyone else to be on mute to block background noise. But if it's a discussion, don't put people on mute while you have a side discussion. It's disrespectful, and people know it's happening because the sound of the background noise changes.</li>
<li><b>Be very clear in your communication. </b>Write carefully, especially if it's an email message rather than real-time communication. Also, be explicit about work that is required, and work that is optional. Are you assigning a task, or are you tossing around ideas? Do you need a lot of help with something, or do you want to be pointed in the right direction? Is anything blocking you? Are you getting pulled away from your main tasks to work on something else?</li>
<li><b>Be smart about communication modes. </b>Use email when you have to carefully consider what you write, or to report on your status and hand off work at the end of the day. Email and mailing lists are terrible ways to have involved discussions. Use the phone (either an impromptu call or a meeting) when you have much to discuss. Use instant messages or texts for quick questions. If your instant messages or email messages are getting long, it's time to switch to the phone. Use mailing lists for broadcast messages that need to go to a group of people, but if it turns into a discussion, move the discussion to a forum or wiki, send the link to the mailing list, and politely move the discussion off of the mailing list and into the forum/wiki. If discussing a work item (defect, feature, etc.), discuss it via comments on the work item, for future reference. Get all of your tips, setup information, and troubleshooting information into a forum or wiki, or write documentation that stays with the work item.</li>
<li><b>Have blameless retrospectives. </b>Every week or two, get together as a team to discuss what is working and what is not working, in an honest, blame-free environment.</li>
</ul>
<ul>
</ul>
<div>
Anyone have more ideas to add? As always, I welcome your comments!</div>
Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com6tag:blogger.com,1999:blog-4454005598309194007.post-52312865360182069082012-11-14T08:27:00.002-08:002012-11-14T08:35:36.740-08:00Test automation got you down?It's here - my full article on <a href="https://www.ibm.com/developerworks/mydeveloperworks/blogs/devops/entry/making_your_automated_tests_faster1?lang=en" target="_blank">Making Your Automated Tests Faster, on the Enterprise DevOps blog</a>! Thank you again to the dozens of people who contributed their ideas at DevOps Days Rome.<br />
<br />
This reminds me of my earlier post on "death by build", and how a build that takes too long, due in part to slow test automation, can really hamper a project: <a href="http://agile-fall.blogspot.com/2010/03/is-shift-left-agile-and-death-by-build.html" target="_blank">Is Shift-Left Agile? And Death By Build</a>Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com15tag:blogger.com,1999:blog-4454005598309194007.post-35379268788669858932012-11-06T09:18:00.001-08:002012-11-06T11:28:20.066-08:00Interview about DevOps and SmartCloud Continuous DeliveryHere's an interview I recently gave to a fellow colleague of mine, Tiarnán Ó Corráin. This seems like a good time to reiterate that these views are my own, and not the official views of IBM:<br />
<br />
<br />
<b>What is DevOps?</b><br />
<br />
You can think of it as an extension of the principles and practices of Agile. Where the Agile methodology breaks down the barriers between development and business goals, DevOps is breaking down the barriers between development and operations. It's not all about tools; it's about people and processes as well. Both Agile and DevOps have a goal of delivering reliable software to the business more quickly, and ultimately, making more money!<br />
<br />
<b>How does DevOps do that?</b><br />
<br />
Well, traditionally there has been a problem between going from a development system to a production system: installing new machines, installing the software, scripting and so forth. Getting from a working development system to a working production system involved all of these manual steps, and introduced many points of failure.<br />
<br />
In addition, because setting up test machines was so time-consuming and error-prone, developers would often assemble their test machines in a quick and dirty way, on a single server, with the cheapest, simplest components. Production systems, on the other hand, would have multiple servers, configured for clustering and fail-over, with firewalls between some components. This meant that the developers weren't testing the software in an environment similar to the one where it would run in production. And because of that, some bugs were never found until the software was deployed into production.<br />
<br />
One of the primary tenets of DevOps is that you should automate every step in creating production systems. Everything from preparing the machine, to installing the latest software, to starting the services, and testing them, should be fully automated and repeatable. And when creating production systems is automated, you can also use production-like systems for development and test work.<br />
<br />
<b>How does virtualization help that?</b><br />
<br />
When we're working in a cloud environment, deploying a virtual machine is something that can be scripted and automated. We use infrastructure code to automate deploying the machine, installing the software, and (re)starting the services, and then we check that code into our source code repository and version it just like the application code. So effectively the process of deploying a new system becomes part of the development process.<br />
<br />
<b>Presumably that makes testing easier?</b><br />
<br />
Very much so. Our virtualization technology means we can deploy production like environments as part of the development process. It's the way the development process has been trending recently. We already have continuous integration: RTC (Rational Team Concert) triggers automatic builds when changes are submitted, and we run unit tests against those builds.<br />
<br />
Now, take that to the next level with continuous delivery: after changes trigger builds, those builds trigger deployments, and when the deployments are complete, the builds trigger automated tests. What it means is that as part of the development process, we have production like servers running the latest code. This allows us to run automated tests including performance verification against a production like environment as part of the development process.<br />
<br />
<b>Taking Agile to the next level?</b><br />
<br />
Yes. It accelerates development, because it takes away some of the uncertainty about deployment: if I can capture every part of the deployment process in my development and testing process, I have more confidence about what I'm going to deploy.<br />
<br />
Deploying test systems automatically also saves developers and testers a lot of time! On my own development team, it's normal for us to deploy dozens of new servers every day, and delete the old ones just as often.<br />
<br />
<b>Can you tell me a bit about your own role?</b><br />
<br />
I'm on the advanced technology team that works on DevOps. We are driving an internal transformation within IBM, to encourage our own development teams to adopt DevOps principles and practices. In addition, we are creating tools to help IBM's enterprise customers adopt DevOps themselves. The first tool we developed to sell is SmartCloud Continuous Delivery v2.0, and it's shipping to customers this week. SmartCloud Continuous Delivery is currently targeted at customers who want to improve their dev/test cycle. We believe this is the easiest place for our enterprise customers to start taking advantage of these new technologies. We have other tools to help with production deployments, like SmartCloud Control Desk.<br />
<br />
<b>How is it going down in the market?</b><br />
<br />
These ideas are gaining real traction, both within and outside of IBM. Internally, we already have several adopters of continuous delivery for dev/test. For instance, Rational Collaborative Lifecycle Management is using our code, and other teams like SmartCloud Provisioning 2.1 have custom continuous delivery solutions that are very similar to ours. And of course, we're using it ourselves -- SmartCloud Continuous Delivery is self-hosting.<br />
<br />
<b>What would you say to any teams that are interested in this approach?</b><br />
<br />
If anyone would like to evaluate the SmartCloud Continuous Delivery product, please check out <a href="https://jazz.net/products/smartcloud-continuous-delivery/">our website</a>. We have free trials available.<br />
<br />
Even if you're not a good candidate for SmartCloud Continuous Delivery, your team may be able to use several of the DevOps principles and practices. Check out our <a href="https://www.ibm.com/developerworks/mydeveloperworks/blogs/devops/?lang=en">Enterprise DevOps</a> blog for ideas, and feel free to contact me about that as well. IBM even offers DevOps consulting workshops.Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com144tag:blogger.com,1999:blog-4454005598309194007.post-57647159425031184092012-10-08T14:38:00.000-07:002012-10-08T14:38:08.971-07:00DevOps Days Open Space: Making Your Automated Tests Faster<br />
One part of my job is helping other teams adopt DevOps in general, and continuous delivery in particular. But I have a problem: many of them have a suite of automated tests that run slowly; so slowly that they only run a build, and the tests that run in the build, about once per day. (Automated test run times of 8-24 hours are not uncommon.) There are several reasons why this is the case, including:<br />
<br />
<ul>
<li>The artifacts that are produced from the build, and then copied over to the test servers, are very large (greater than 1 GB in size). Also, sometimes the artifacts are copied across continents.</li>
<li>Sometimes there are multiple versions of the build artifacts that must be copied to different test servers after the build. A typical product I deal with will support at least a dozen platforms; a few support around 100 different platforms, when you multiply the number of supported operating system versions times the number of different components (client, server, gateway, etc.) times 2 (for 32- and 64-bit hardware).</li>
<li>Often, the database(s) for the product must be loaded with a large amount of test data, which can take a long time to copy and load.</li>
<li>Many products have separate test teams writing test automation. Testers who are not developers tend to write tests that run through the UI, and those tests are usually slower than developers' code-level unit tests.</li>
</ul>
Running builds and tests often, so developers know quickly when they make a change that breaks something else, is a key goal of both continuous integration and continuous delivery. Ideally, a developer should get feedback on whether their code is "ok", using a quick personal build and test run, within 5 minutes. Anything over 10 minutes is definitely too slow; the developer will probably move on to something else, make more changes, and forget exactly what was changed for that particular test. <br />
<br />
Once the quick tests pass, the developer can run a full set of tests and then integrate the tested changes. Or, in cases where a full set of tests is extremely slow, the developer can integrate his or her code changes once the quick tests pass, and then let the daily build run the full set of tests.<br />
<br />
In this DevOps Days open space session, we brainstormed ways to make automated tests run more quickly. We focused more on quick builds for personal tests, but most of these ideas would make the full set of tests faster too. Many thanks to the dozens of smart people who contributed their ideas. I don't even have their names, but they know who they are. I'm sure we'll use several of these ideas right away.<br />
<br />
Watch for an article with more details on each of these, coming soon...<br />
<br />
<h2>
Fail quickly</h2>
<h3>
<br /></h3>
<h3>
Run a quick smoke test first</h3>
<h3>
Run a small set of tests that fail often next</h3>
<h3>
Run slow tests last, or not at all</h3>
<div>
<h3>
See also: Remove slow tests</h3>
</div>
<div>
<br /></div>
<h2>
Run in parallel</h2>
<div>
<h3>
<br /></h3>
<h3>
Run test buckets in parallel</h3>
</div>
<h3>
Use snapshots of databases or VMs to make it easier to run tests in parallel</h3>
<div>
<br /></div>
<h2>
Break up tests into smaller groups</h2>
<h3>
<br /></h3>
<h3>
Divide your application into components, and test the changed components</h3>
<h3>
Automatically determine which tests to run when code changes</h3>
<div>
<br /></div>
<h2>
Save time on I/O</h2>
<h3>
<br /></h3>
<h3>
Mock responses</h3>
<h3>
Use LXC (Linux Containers)</h3>
<h3>
Move servers and data so they are close to one another</h3>
<h3>
Make your test infrastructure faster</h3>
<div>
<h3>
See also: Use snapshots of databases or VMs to make it easier to run tests in parallel</h3>
</div>
<h3>
Cache what you can</h3>
<div>
<br /></div>
<h2>
Remove some tests</h2>
<h3>
<br /></h3>
<h3>
Remove tests that never fail</h3>
<h3>
Remove slow tests</h3>
<h3>
Replace some UI tests with code-level tests</h3>
<h3>
Replace some tests with monitoring</h3>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com2tag:blogger.com,1999:blog-4454005598309194007.post-76087270549529037282012-07-06T15:11:00.000-07:002012-07-06T15:11:11.788-07:00DevOps Days Open Space: DevOps for Legacy Code and Real ServersI proposed the topic for this OpenSpace: DevOps for Legacy Code and Real Servers. Here are some of the insights I gleaned from this session. <br />
<br />
<b>Legacy servers</b><br />
<ul>
<li>Cloud platforms are evolving to manage real, legacy servers in addition to virtual machines.</li>
<li>Chef, for example, can manage both physical and virtual servers. It can also manage clean OS installations as well as update existing servers. There's a tool called Blueprint that will attempt to reverse engineer Chef automation for an existing server.</li>
<li>It's difficult to re-create systems that weren't automated in the first place. However, it greatly reduces your risk if you invest the time and effort to do that. What if the server was destroyed in a fire or something? </li>
<li>Sometimes people have even lost the source code for applications that are running in production. That is a very risky state to be in.</li>
<li>Another option is to clone the system into a VM first, snapshot it, and then do your exploratory work on the VM.</li>
<li>You can also copy some of your production web traffic to your staging servers. </li>
<li>Or, you can start deploying new applications to VMs, and gradually shift your enterprise code to VMs.</li>
</ul>
<b>Mainframe systems</b><br />
<ul>
<li>Mainframes are the backbone of many legacy systems, and they are not going away. </li>
<li>People who are used to working with mainframes have a different culture and language than people who are used to developing new web applications. There's a communication gap to bridge before they can benefit from DevOps principles and practices.</li>
<li>One option is to just get an enterprise's web applications to adopt DevOps and punt on the mainframe applications. But why can't we do the same thing for mainframe applications?</li>
<li>Mainframes have limited logging and monitoring systems. Why?</li>
<li>Mainframes have limited tooling. Why? </li>
<li>It's very difficult to see what's going in within an application.</li>
<li>It's very difficult to debug applications.</li>
<li>Deployments have to be completed with zero downtime.</li>
<li>LPARs, CICS regions, etc. could actually be considered a type of virtualization. Is there a way we could make them behave more like VMs?</li>
<li>Could mainframe developers take some of the best practices from .Net and Java?</li>
<li>A more open place, like a university, might be more willing to experiment with DevOps first. </li>
</ul>
<b>Universal Principles</b><br />
These principles from DevOps can apply to legacy servers and mainframes just as easily:<br />
<ul>
<li>Source Control Everything (including infrastructure code)</li>
<li>Version Control Everything (including infrastructure code)</li>
<li>Automate Everything (including infrastructure code)</li>
<li>Test Driven Development: Test First, Test Everything</li>
<li>Test for Operational Quality (performance, transaction load, security, etc.)</li>
<li>Agility</li>
<li>Focus on the Business Outcome, not the features or requirements</li>
<li>Improve teamwork between Dev and Ops</li>
<li>Collect metrics so you can find problems earlier</li>
</ul>Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com2tag:blogger.com,1999:blog-4454005598309194007.post-52682382587806038072012-06-29T14:47:00.000-07:002012-06-29T14:47:29.946-07:00DevOps Days Open Space: How Can Ops Teams Give Feedback to Dev Teams?This was another interesting Open Space that I participated in: How Can Ops Teams Give Feedback to Dev Teams?<br />
<br />
Chaos Monkey can teach developers where things might break. You need to couple that with some sort of monitoring tools so you can find bugs of the performance/throughput/overload type as well.<br />
<br />
People in ops would like developers to program more defensively. Developers are not generally taught how to do this. It's also not usually part of their culture. <br />
<br />
One great way to tech developers is by writing tests that fail. Developers are great at fixing tests that fail.<br />
<br />
Another best practice is to embed developers in operations and vice versa. Some companies have done this with teams of people for months or years at at time. Others rotate people between the teams for one day every couple of weeks. Set it up like an apprenticeship, where people can start out with a mentor and gradually become responsible for their own things.<br />
<br />
Operations people can review code!<br />
<br />
Developers can have pagers!<br />
<br />
Product managers need to care about operational constraints and include those in the requirements that they put on the development teams.<br />
<br />
You need to get everyone in the company to think about business value and happy customers. Constantly.<br />
<br />
You need to get everyone in the company to watch dashboards. Give each person a few graphs to watch on a dashboard.Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com0tag:blogger.com,1999:blog-4454005598309194007.post-59232540536089803332012-06-29T12:59:00.002-07:002012-06-29T12:59:43.186-07:00DevOps Days Open Space: Private Cloud Myths, Facts, and TipsI'm at the DevOps Days conference this week, and some of my favorite sessions have been the OpenSpace collaborations. I was the scribe for the session on "Private Cloud: Myths, Facts, and Tips", so here is what we discussed:<br />
<br />
Myth: The cloud will not fail.<br />
Fact: It will fail, so write software that can handle failures. In fact, entire data centers will fail. Have at least two geographically remote clouds that can fail over to each other.<br />
Tip: The Yahoo guys have found that the more technology you have to prevent a data center from failing, the more points of failure you introduce into the system. They've found that it's much more effective to make it easy to fail over to a new data center. In fact, they fail over during peak traffic times on purpose: for practice, to build confidence, and as a way to take servers out of production for a little while to upgrade them.<br />
Tip: Infrastructure automation makes it easy to create a new backup system. <br />
<br />
Myth: The cloud will move my enterprise forward into the future.<br />
Fact: There are issues you have to solve first.<br />
Tip: Teach your developers how to write stateless applications.<br />
Tip: The chaos monkey can help your developers learn how to write stateless applications, and how to handle failures of different services.<br />
Tip: Getting people to write stateless code is a cultural and a management issue.<br />
<br />
Myth: Any workload can run in the cloud.<br />
Fact: Some workloads will not behave in a cloud-like manner. You have to design applications for that.<br />
Tip: A best practice is to tell developers that their application could be moved to the public cloud at any time, without notice. <br />
Tip: A full penetration test on a regular basis is extremely valuable. Especially if the output of that is a set of tests that failed, that your developers can fix!<br />
<br />
Myth: My current ops team can run a private cloud easily.<br />
Fact: You need to train people on how to run a cloud. It's not traditional ops. You have to learn to let go. You have to give people control over their own machines. You have to let them provision their own VMs.<br />
Tip: Sysadmins should learn how to write infrastructure code. Either that or they may be reduced to changing out broken servers on racks.<br />
Tip: One way to prevent people from overloading your cloud capacity is to put fixed leases on the machines. If you have infrastructure as code, you can provision a new system in a few minutes. So, for developer unit test VMs, it's fine to delete those after 24 hours. The developer may manually delete VMs even earlier. Have a way to flag the small percentage of machines that should live for a long time.Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com2tag:blogger.com,1999:blog-4454005598309194007.post-62673331997726206002012-06-28T22:10:00.000-07:002012-06-28T22:10:47.533-07:00The Women at DevOps DaysI recently posted a tweet about DevOps Days Mountain View 2012, and I feel the need to explain it a bit: https://twitter.com/DukeAMO/status/218541579219116032/photo/1<br />
<br />
First of all, why did all of the women there that evening decide to get together, in a cage of all places? Well, DevOps Days was held in a very odd sort of place, the Silicon Valley Cloud Center. It's a former data center, apparently, and the evening dinner and party was basically in the garage. For some reason, the garage is partitioned into several large cages, surrounded by chain link fences. We thought it would be fun to get together, and the easiest place to do that was in one of the cages, because there were tables in there. The fact that it was slightly subversive to do that was part of the fun.<br />
<br />
Did we do that to make some sort of political statement? No, not really. The fact of the matter is, when you're a woman at an operations conference, it's very obvious that you're in a minority. I haven't done any scientific studies, but I'm going to guess that the ratio of men to women attendees at both DevOps Days and the larger Velocity conference this week was around 95% to 5%. It's a shame that more women aren't attracted to this field, because it's quite rewarding, and there's no reason for women to stay away. But the women who do buck the trend are rare birds, and we don't mind being a little bit different. We also get used to it. On a day to day basis, I don't even notice that the vast majority of the people I work with are men. It's normal.<br />
<br />
We had our little toast together, and had a great time talking for the rest of the evening. By the time I left, there were just as many men at our table as there were women, and we were happy to have them.<br />
<br />
Here's to the rare birds!Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com1tag:blogger.com,1999:blog-4454005598309194007.post-4112730904251448572012-05-24T13:41:00.001-07:002012-05-24T13:42:32.550-07:00Are you treating your servers like snowflakes or rubber stamps?This is more of an agile blog, but I thought I'd cross-post this from our DevOps team blog, just in case anyone's interested: <br />
<br />
Are you treating your servers like snowflakes or rubber stamps? <br />
<a href="https://www.ibm.com/developerworks/mydeveloperworks/blogs/devops/entry/snowflakes_or_rubber_stamps?lang=en">https://www.ibm.com/developerworks/mydeveloperworks/blogs/devops/entry/snowflakes_or_rubber_stamps?lang=en</a><br />
<br />
I also submitted this proposal for DevOps Days Mountain View 2012.<br />
<br />
How IBM is using DevOps to build DevOps tools <br />
<a href="http://devopsdays.org/events/2012-mountainview/proposals/How%20IBM%20is%20using%20DevOps%20to%20build%20DevOps%20tools/">http://devopsdays.org/events/2012-mountainview/proposals/How%20IBM%20is%20using%20DevOps%20to%20build%20DevOps%20tools/</a><br />
<br />
Comments on both are welcome!Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com0tag:blogger.com,1999:blog-4454005598309194007.post-54450105843630065422012-01-27T11:42:00.000-08:002012-01-27T11:46:23.554-08:00How can you make remote development work?I have a few years of experience with working from home, and several years of experience with working with cross-site teams. There are a few things that are necessary for a remote worker. A remote worker needs to be:<br /><ul><li> Motivated</li><li> Independent</li><li> Self-managing</li><li> Responsible</li><li> Good at time management</li><li> Excellent at communications (oral and written)</li><li> Honest and trustworthy</li><li> Good at ignoring distractions</li><li> Willing to ask for help if needed</li></ul><br />While most of these are disciplines that you can get better at with practice, some are harder to learn.<br /><br />Working from home can be great, but it's a special skill, because it's easy to get distracted with things that need to be done around the house. Also, there's no one there to see if you're goofing off, so you have to motivate yourself to focus on the task at hand. It's easier to focus and be productive if you have specific tasks that are due on a regular (daily) basis. Scrum helps with this by providing a quick daily checkpoint meeting.<br /><br />Ideally, remote workers should be in the same time zone as each other, or at least close to the same time zone. I've found that working with people 5 hours away (the UK, for example) is workable. You just learn to block off your mornings for communication and meetings, and do independent work in the afternoons and evenings. Working with people 7 or more hours away is painful. You end up losing days of work because you can't find someone who can help you get past a blocking issue. Perhaps you send an email describing a problem and asking for help... then the person on the other end misunderstands something... and now you're into a second lost day. We work with people in India, but they're actually on a 12-8 PM working schedule, so we at least have a couple of hours of overlapping time in the Eastern US. While you *can* call people early in the morning or late in the evening to keep things moving along, nobody likes to do that on a daily basis.<br /><br />One other tip, especially for software engineers, is that it really helps to bring people into the office for a week or so, once or twice a year. First of all, face to face communication is high-bandwidth, so you can get a lot of work done in a short amount of time (planning, design work, strategy sessions, training, etc.). People tend to trust and respect each other a little more once they've met in person. They're also more comfortable asking questions, or asking each other for favors, which means people are more productive when they get home. Finally, it builds a stronger team and improves morale.<br /><br />A year or two ago, multinational development teams (even within one scrum team) were the norm here. But we're trying to consolidate projects locally as much as we can now. My current project is probably 75% local and 25% remote.Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com0tag:blogger.com,1999:blog-4454005598309194007.post-26240439955684352822010-08-16T08:06:00.001-07:002010-08-16T08:33:33.787-07:00Doing Planning Poker Remotely?Has anyone out there worked out a clean way to do Planning Poker remotely? We have development teams with people in multiple locations, so we conduct our planning sessions via conference call and screen sharing.<br /><br />If you're not that familiar with Planning Poker, it's a quick way to size stories without getting down to the level of person-days. Story point sizings are based on your past experience - an 8-point story should take rougly as much design/dev/test/doc effort as an 8-point story you've finished in the past. You talk for a few minutes about a story, and then everyone in the planning session does a "secret ballot" where they choose a story point sizing. Then you discuss the largest and smallest sizings, and come to a consensus on what the sizing should be. You can use the story point sizings to plan how much work you can do in an entire release, across several sprints.<br /><br />There's a Planning Poker plug-in for Sametime. The plug-in was OK - you could set up a planning session with a group of people, and enter the stories to be sized. Then when you selected a story, each person could submit their estimate, and it wouldn't show the estimates until everyone had submitted. But it's often a problem for people to get the plug-in installed and working. For example, if people had the wrong version of Sametime, or if they were using a Mac, it wouldn't work for them.<br /><br />Another thing we have done is a standard group chat where everyone just types in their estimates, and everyone can see them right away. There are a couple of problems with that - for one thing, some people (mostly newbies) will refuse to submit their estimates until other people (usually the team lead / scrum master) submits theirs. Also, once a few people submit their estimates, all of the other estimates seem to magically become the same number. Which means that people are changing their answers based on what they already see on the screen. Sigh.<br /><br />A third thing I've seen scrum masters do is set up a second display where they have an individual chat session with each person, and no one can see the sizing estimates except for the scrum master. That works resonably well, but it's still kludgy.<br /><br />Can anyone recommend a better way to do this? What do you do?Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com5tag:blogger.com,1999:blog-4454005598309194007.post-89577000210460898372010-06-04T07:09:00.000-07:002010-06-07T13:21:18.944-07:00Working with a remote teamI'm having a little adventure this month - we just hired five contractors in Bucharest, Romania to work on my current project. They're new to our product, and we have to get them to a point where they're doing productive work very quickly. It's a challenge for sure.<br /><br />I've worked with remote teams before, and this experience is re-enforcing a few rules of thumb that I have:<br /><ul><li><span style="font-weight: bold;">Time zones matter.</span> Romania is 7 hours ahead of us, which means that my day starts at 9 AM and theirs ends at 10 AM. So, we've blocked off 9-10 each day to talk to each other. This is significantly easier than working with teams in India or China, where the difference is close to 12 hours. (Meetings at 8 PM? 7 AM?) However, it's significantly more difficult than working with teams in Brazil, which are only 1 hour off, or even teams in Ireland, which are 5 hours off. The Romanian team is working while I'm asleep, so I have to make sure they all have enough to do before I stop working the night before. Also, if they get stuck on something, it's easy to lose a day. So to keep them productive, ideally each of them should have a few things they could be working on. That way if they're blocked on one task they can work on another until we sync up again.<br /></li><li><span style="font-weight: bold;">You should be in the same building</span><span style="font-weight: bold;"> as the people you work with most closely.</span> Yes, you can talk to someone 4-5 times a day over the phone, but it's significantly more efficient if you can talk in person. Especially if your work sometimes involves drawing things, like design diagrams or GUI prototypes. It's also very helpful if your work involves making difficult decisions, because you can read people's body language in person. Reading body language can help you see if someone is sure or unsure of what they're saying.<br /></li><li><span style="font-weight: bold;">Teach the remote team to be as independent as possible.</span> If they get stuck on something, they need to be able to work past the problem on their own, so you don't lose a day. Also, it helps if they're working on their own set of files/features, so they're not interfering with people who they can't communicate with immediately.<br /></li><li><span style="font-weight: bold;">Language skills are critical.</span> One of the worst feelings is explaining something to a technical team on the phone, and hearing crickets in response. Did they understand what you said or not? Are they talking to each other on mute? Even a thick accent can make communication difficult, because you may have to ask someone to repeat themselves over and over. After a few tries, people may give up and try to guess what the other person is saying. That can lead to misunderstandings. You can mitigate this somewhat by speaking more slowly and carefully, and by rephrasing what you heard the other person say. But this just slows down communication, so it's best to hire people you can communicate with easily in the first place.<br /></li><li><span style="font-weight: bold;">Meeting people in person, even once, helps a lot.</span> There are people that I spent a week with in training sessions three years ago. I still go to them first with a problem, and they are more likely to help me than someone I've never met. And vice versa. In some ways, I feel like I know them better than people I've worked with for years over the phone. Meeting people face to face naturally creates a certain comfort level and familiarity that's very difficult to develop otherwise.</li><li><span style="font-weight: bold;">Training takes a lot of time and money.</span> This sounds obvious, but it takes a long time to get a new set of people up to speed. On our product this ranges from a month to a year or more, depending on the specific assignment. The hidden cost is that the people doing the training are not able to spend as much time on their other assignments. This is an argument for retaining existing employees whenever possible! Also, when searching for contractors or new hires, you should put a high premium on people who already have a skill set close to what you need, even if they cost more. One other tip in this area - the time to train five or six people at once is similar to the time required to train one or two, so you should try to add people to a project in batches, not one at a time.</li><li><span style="font-weight: bold;">Record your training and demo sessions.</span> We record most of our training sessions and demos (end of sprint demos and customer demos). We use Lotus Live/Sametime Unyte/Webdialogs, or Camtasia Studio, to record screen sharing along with the sound from the conference call. We post them to a shared server and link to them from our development wiki. We use those recorded sessions over and over again. They come in handy for training new developers and testers, creating beta scenarios, training sales and support teams, creating new training classes, etc. We get pretty good reuse out of the slide decks too.<br /></li></ul><br />Anyone else have some favorite tips? Maybe you could help me!Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com1tag:blogger.com,1999:blog-4454005598309194007.post-46879747403845118392010-04-07T10:53:00.000-07:002010-04-07T11:16:56.263-07:00Struggling with the Product BacklogOne area that our team consistently struggles with is the product backlog. This is not a new problem; even before we adopted agile development processes we had problems with this.<br /><br />Our legacy system for tracking customer requirements is a Notes database. Notes does some things well, but this particular database is often a black hole. Here are my complaints with it:<br /><ol><li>The search functionality is terrible. We have literally hundreds of requirements, so a search feature is really important. I can't get a full text search to work at all; maybe it's not enabled.</li><li>It doesn't provide an easy way to prioritize requirements to get the most important to bubble up to the top.</li><li>Almost anyone who works with customers can open requirements. Some people are not particularly good at writing up requirements. So, they may think that they've done their job by opening a requirement, but the product leadership has no idea what they're really asking for. Some people also don't fill in contact information for the customer requesting the feature, so it's difficult to go back to the source for more details.<br /></li><li>There's no good way to find very similar or overlapping requirements. We might have five separate requirements for the same thing, and no way of seeing that.</li><li>It's not managed well. Nobody goes through the painful process of looking at all of the requirements to see which ones no longer apply, or which ones have become more important since they were opened.</li></ol>We have started using Rational Team Concert (RTC or Jazz) for managing requirements. However, we still have this legacy database to deal with, and I'm pretty sure no one has the time or inclination to copy everything from Notes into RTC. Besides, would that even be a good idea? RTC at least has a good search function, and it's easy to prioritize things. But RTC doesn't fix problems 3 through 5.<br /><br />So, what happens when the product backlog is poorly managed? We start planning a release and ask the product owner what should go into it. Some items are obvious and get added added to the release backlog right away. Then the blank stares begin. How can you even tackle the task of looking through the rest of the requirements to find the best and most important ones?<br /><br />We end up developing whatever the product leadership has been hearing about the most in the past few months. Sometimes this is good enough, sometimes not. For example, if the support team is consistently hearing the same complaints, but not bringing them forward to product management, those complaints might not be addressed.<br /><br />Something like an idea cloud would be nice. It could show us the keywords that are coming up most often in the requirements. That would help us find common themes, and overlapping requirements. But I have no idea how we could get that working with Notes or RTC.<br /><br />Getting people to categorize their requirements when they open them could help too. At least that would help us group similar requirements together. RTC lets you enter keywords on stories, but that's freeform, so people may use different keywords for the same idea.<br /><br />I would love to get some advice on how to improve this situation. Especially problems 3, 4, and 5.Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com4tag:blogger.com,1999:blog-4454005598309194007.post-34684988696273342932010-03-22T11:21:00.001-07:002010-03-22T18:05:55.454-07:00Is Shift-Left Agile? And Death By BuildSometimes people confuse "shift-left" and "agile" software development practices. They both involve testing earlier, and talking to customers earlier, but that's pretty much all they have in common.<br /><br />"Shift-left" practices are ones that help you find and fix problems earlier. You can think of the software development life cycle as a time line, with gathering requirements on the left, then design, code, test, and field maintenance to the right. The earlier (and farther to the left) you find a problem, the less it will cost you to fix it. It takes extra time and effort to find problems earlier, but it's worth it in the long run. For example, if you create JUnit test cases for most of your new code, you will invest more time up front, but you'll almost certainly find some bugs even before the code is handed off to the testers. <br /><br />Showing your code-in-progress to customers on a regular basis is also a "shift-left" practice, because you may find out that you are not delivering what the customers want early enough to change your design.<br /><br />"Shift-left" practices can help make your project more agile, because if you catch bugs earlier, you don't need to have such a long test cycle at the end of a release. If you automate your tests and run the automated tests with every build, you can also find newly-introduced bugs in old code more easily, and cut down on the amount of regression testing and bug-fixing you have to do later.<br /><br />But you can slow down a project with "shift-left" processes too. I know of a project where their build process takes 30+ hours. I'm assuming it takes so long because they're trying to do too much automated code analysis and automated unit testing as part of the build process. (My product is quite large, requiring several CDs, and it only takes about an hour to build everything.) Then, after the 30-hour build, they start their BVT (build verification test), which also takes many hours. The goal of testing during the build process is to catch bugs that escaped unit test before they get to the test team, and "shift-left". But because the build process is so slow and painful, they only run it once a week! So if a code change misses a build, it's delayed by an entire week. And if someone's new code breaks the build (heaven forbid), the entire project is delayed by a week. <br /><br />So what did they do to avoid accidentally delaying the project by a week? They implemented a heavy process of code reviews to make sure that no one integrates code that will break the build. Code reviews and inspections are another "shift-left" tool. So now you have to make your code changes by Wednesday. On Wednesday you have to find a team leader to review your code changes and approve them so they're ready for the Thursday build. Then the build and BVT processes run on Thursday, Friday, Saturday, and code is ready to be tested on Monday. <br /><br />This is the opposite of an agile process. Yet, strangely enough, this team claims that they are following agile software development processes, specifically Scrum. I'm sure the people who designed their development process had the best intentions. However, one of the more important tenets of agile software development is that you have to get your code into the hands of testers, and customers, as soon as possible. When it can easily take a week or more just to get new code to the test team, you're losing many of the benefits of agile development processes.<br /><br />Don't get me wrong, shift-left and agile processes can work together quite nicely. But they can also work against each other if you're not careful!<br /><br />Does anyone else have any shift-left horror stories to share?Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com6tag:blogger.com,1999:blog-4454005598309194007.post-45590171983669617182010-03-17T06:52:00.001-07:002010-03-17T08:41:14.278-07:00Open Source SoftwareThanks to Husain for this topic suggestion...<br />"What Software Development Methodology can best fit in delivering an open Source Software??? After doing a little research I found no better than the Agile method!!"<br /><br />Open Source (and Community Source) software is special, in that much of it is developed in tiny pieces by people working independently. It does lend itself to agile development, in that the requirements are not all collected in advance. Open source development cannot be done in a waterfall fashion, where all of the design for a release is done at one time, then all of the development, then all of the testing.<br /><br />The one open source project I worked on used a project backlog, where people would pick up the next high-priority work item (story) that interested them. Someone would write a bit of code, test it, document it if needed, and submit it. Then the changes were approved and integrated by the project owner, and released to the public within a few weeks. So, adding a new feature/story was like a mini-sprint. However, the project did not use time-boxed iterations with fixed start and end dates. Each person was on their own schedule. It was very agile.<br /><br />I think this highlights the fact that Agile software development methodologies are not one-size-fits-all. What works for a typical open source project would not work for my current project. <br /><br />Our product consists of several components that depend upon each other. To implement my new feature, I may have to ask someone from another component team to write some new code for me. And someone else may use the new code I'm writing in their component. Because we are adding major new features across the board, we usually need to install the same version of every component, or they will not work together. We also need to plan some features a few months in advance: component A will implement the first piece, then component B will use that new feature and implement the second piece, and then component C will use features from A and B to implement the third piece. Our customers expect A, B, and C to all work together, so we need some time to stabilize the code and test all of the pieces together. So we have a couple of months at the end of a release where no one is allowed to make any major changes, and only bug fixes go in. We also use this time to test the code on additional platforms, and in different languages. This is less agile, but it's not realistic to expect us to ship a complex product like this without some dedicated time to stabilize everything and clean things up.<br /><br />I have some stories about a team I know of that calls themselves agile, but really isn't. It's pretty silly what they do, really. More on this later...Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com3tag:blogger.com,1999:blog-4454005598309194007.post-32949186844022040672010-03-08T10:32:00.000-08:002010-03-08T11:06:09.526-08:00When is a story "done done"?When exactly is a story "done done", ready to be checked off the product backlog?<br /><br />On the one hand, saying that a story is "done" too soon will leave your project with hidden debt. If you're still writing code for a story after you've said that it's done, then it's not done! And if you're still writing code in the next sprint for something that was theoretically done in the previous sprint, then a few bad things are happening:<br /><ul><li>Your velocity for the previous sprint will look too high.</li><li>Your velocity for the current sprint will look too low.</li><li>You may or may not be "in sync" with the test team and product management on what's really done and what still needs to be tested.</li><li>You lose the benefits of time-boxed iterations.<br /></li></ul><br />But what about things like <span class="blsp-spelling-error" id="SPELLING_ERROR_0">Bidi</span> <span class="blsp-spelling-error" id="SPELLING_ERROR_1">enablement</span>, visual design clean-up, or extensive logging? I would argue that much of this code hygiene work can and should be done toward the end of the release. Get the new function and risky changes out there so they have time to mature. Then save a sprint or so at the end for clean-up work.<br /><br />I believe it's reasonable to create a story like "Add <span class="blsp-spelling-error" id="SPELLING_ERROR_2">Bidi</span> support to the following areas: ...". It's testable, and it's new function. Plus, you get yourself into a <span class="blsp-spelling-error" id="SPELLING_ERROR_3">Bidi</span>-<span class="blsp-spelling-error" id="SPELLING_ERROR_4">enablement</span> mode, and you can make a single pass through the code making the same changes everywhere. This is good because it decreases the amount of context-switching your brain has to do.<br /><br />On the other hand, I believe that logging/tracing, <span class="blsp-spelling-error" id="SPELLING_ERROR_5">JUnit</span> testing, and factoring out text for translation should be done before a story is "done". Logging/tracing make it easier to debug the code from the beginning, so you don't waste time finding where the bugs are. <span class="blsp-spelling-error" id="SPELLING_ERROR_6">JUnit</span> testing finds bugs early, so you can fix them more quickly and easily. And factoring out messages is easier to do while you're in the code; you'll inevitably miss some translatable text if you try to do it all later.<br /><br />Performance testing is a tricky one. You can't leave it until the end because you may find that you need to make significant changes to your algorithm to improve performance, and then everything will need to be re-tested. But you could ship a product with new function that takes too long to run, if you ran out of time for performance testing. If a new feature warrants performance testing, I would recommend doing the performance testing one sprint after the new code goes in.<br /><br />In my current project, we say a story is "done done" when all of the function test scenarios have been executed, and it has no severity 1 (the function doesn't work, and there's no work-around), severity 2 (the function has bugs that you can work around), or must-fix (as determined by the testers) defects that are still not closed. It may have severity 3 (small problem) or 4 (annoyance) defects. It also has to be demoed at the end of the sprint.<br /><br />I know someone who works on a project where there is a long list of criteria to be met before a story is "done done done" as they call it. In addition to code hygiene, it also has to go through system test, translation, <span class="blsp-spelling-corrected" id="SPELLING_ERROR_7">accessibility</span> testing, and so on. As a result, no story is marked as "done" until the product is about to go out the door. This makes their story points meaningless. It's overkill!<br /><br />On the other hand, it's not appropriate to say that a story is "done" just because you've completed unit testing on it. If the testing is not completed by the last day of the sprint, the story needs to be moved to the next sprint as debt. I've seen people try to fudge their way out of this... "well, this story only has one open defect, so we should get credit for it". We need to be at peace with sprint-to-sprint debt. The good news is that the stories that are almost done should be closed quickly at the beginning of the next sprint, so your velocity for the next sprint will be higher. Over time your story point velocity will average out to an accurate number.<br /><br />I'd love to hear what some other teams are using as their "done done" criteria.Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com0tag:blogger.com,1999:blog-4454005598309194007.post-30213934466594215822010-03-03T11:06:00.000-08:002010-03-03T11:29:17.683-08:00Using Story Points for Sprint Planning?Our team is still trying to get the hang of using story points. Until very recently, we weren't using them at all. I think this is for a few reasons:<br /><ul><li>Story points are relative to each other, not to the calendar. This makes them more abstract than person-days, so they're a little hard to wrap your head around at first.</li><li>We weren't in the habit of playing planning poker to put points on our stories. On the rare occasion we did put points onto stories, they were really just bastardized person-days (1 story point = 1 person-day).</li><li>Because we didn't have story points on things we had already completed, we couldn't use previously completed stories as a reference to put points onto new stories.</li><li>Because we didn't have any historical data on how many story points we closed in previous sprints, we had no idea what our velocity was, so we couldn't use story points or velocities to help plan the next release.</li><li>Story points aren't detailed enough for sprint planning. If you know that you have two weeks to code, and 3 people writing the code, how many story points can you commit to in the sprint? It doesn't have any meaning.</li></ul><p>It's a vicious cycle: story points aren't useful if you haven't used them before... so people don't try to use them... so you don't gather the historical data to make them useful...</p><p>The problem is that we were spending too much time doing sizing estimates. As soon as you try to size something in terms of person-weeks, that implies that you have a pretty good idea what the low-level tasks are, and how long it will take to complete them. Honestly, it wouldn't be unusual for us to invest a person-week into sizing a story, when you consider how many people were pulled into each sizing effort for a few hours apiece. So we were told that we had to start using story points, and stop spending so much time on the sizing estimates. In planning poker, you rarely spend more than 2-3 minutes sizing a story.</p><p>People scoffed at the idea that you could plan a sprint without getting down to the task level and estimating how long it would take to implement each story, though.</p><p>Then someone found this article:<br /><a href="http://blog.mountaingoatsoftware.com/why-i-dont-use-story-points-for-sprint-planning">http://blog.mountaingoatsoftware.com/why-i-dont-use-story-points-for-sprint-planning</a> </p><p>Now, I think we will be using story points for release planning (meaning, planning far into the future), and task hours/person-days for planning each individual sprint.</p><p>This means that we'll have to take a leap of faith. We have to start assigning points to stories before we start working on them. Once we've done that for a couple of sprints, we'll have some idea what our velocity is. Then we can start to use that information for planning at the release level. We'll see how it goes.</p><p>So, what does your team do with story points and sprint planning? Is it working?</p>Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com1tag:blogger.com,1999:blog-4454005598309194007.post-30239600011349557242010-03-03T07:07:00.000-08:002010-03-03T07:26:38.964-08:00What is AgileFall?<span id="SPELLING_ERROR_0" class="blsp-spelling-error">AgileFall</span> is a tongue-in-cheek term for a software development model where you are trying to be agile, but you keep falling into waterfall development habits. For example, a team practicing <span id="SPELLING_ERROR_1" class="blsp-spelling-error">AgileFall</span> may say:<br /><ul><li>We have sprints, but they are four or six weeks long.</li><li>We try to do most of the development at the beginning of the sprint and most of the testing at the end of the sprint.</li><li>We have a product backlog with priorities, but we start working on a release with a long list of features already committed to the business.</li><li>We know our product release dates several months in advance. There's a long list of target dates that must be met before the release date.</li><li>We test new features as they are developed in each sprint, but we also have a <span id="SPELLING_ERROR_2" class="blsp-spelling-error">lonnnng</span> system / globalization / translation / integration test cycle at the end of the release.</li><li>We sometimes spend too much time up front designing the software before we even start prototyping it.</li><li>We do our initial <span id="SPELLING_ERROR_3" class="blsp-spelling-error">sizings</span> in person-weeks, because we're not comfortable with story points and velocities yet. </li><li>When management asks us for a rough sizing, we might spend days working on that sizing effort, because we feel like we need to really understand the tasks involved before we "commit" to a sizing.</li><li>People get upset (or defensive) when stories are not completed in the sprint they were planned for.</li><li>We demo our software in development to customers, but only after we're happy with it. By the time we get around to having a demo, it might be too late to change much.</li></ul><p>Is your team practicing <span id="SPELLING_ERROR_4" class="blsp-spelling-error">AgileFall</span>? In what ways?</p>Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com11tag:blogger.com,1999:blog-4454005598309194007.post-54338324258055403642010-03-03T06:59:00.000-08:002010-03-04T06:20:43.924-08:00IntroductionWelcome to my AgileFall blog! I've been creating software since 1995. I have a Bachelor's of Science in Computer Science from Duke, and a Master's of Science in Computer Science from UNC, where I focused on Distributed Computing and Software Engineering. I studied software enginnering theory, including the waterfall and agile models, in school. For the first 10 years or so of my career at IBM, we followed a strict waterfall development model. Then about 2 years ago we started switching to an agile development model. Our development and support team has also become increasingly global. This blog is about our successes and failures.Anonymoushttp://www.blogger.com/profile/07439703069777995137noreply@blogger.com1