One part of my job is helping other teams adopt DevOps in general, and continuous delivery in particular. But I have a problem: many of them have a suite of automated tests that run slowly; so slowly that they only run a build, and the tests that run in the build, about once per day. (Automated test run times of 8-24 hours are not uncommon.) There are several reasons why this is the case, including:
- The artifacts that are produced from the build, and then copied over to the test servers, are very large (greater than 1 GB in size). Also, sometimes the artifacts are copied across continents.
- Sometimes there are multiple versions of the build artifacts that must be copied to different test servers after the build. A typical product I deal with will support at least a dozen platforms; a few support around 100 different platforms, when you multiply the number of supported operating system versions times the number of different components (client, server, gateway, etc.) times 2 (for 32- and 64-bit hardware).
- Often, the database(s) for the product must be loaded with a large amount of test data, which can take a long time to copy and load.
- Many products have separate test teams writing test automation. Testers who are not developers tend to write tests that run through the UI, and those tests are usually slower than developers' code-level unit tests.
Once the quick tests pass, the developer can run a full set of tests and then integrate the tested changes. Or, in cases where a full set of tests is extremely slow, the developer can integrate his or her code changes once the quick tests pass, and then let the daily build run the full set of tests.
In this DevOps Days open space session, we brainstormed ways to make automated tests run more quickly. We focused more on quick builds for personal tests, but most of these ideas would make the full set of tests faster too. Many thanks to the dozens of smart people who contributed their ideas. I don't even have their names, but they know who they are. I'm sure we'll use several of these ideas right away.
Watch for an article with more details on each of these, coming soon...