Continuous Integration Builds Are Not Good Release Builds

Something’s been bothering me for a while. According to what I’ve read, lots of folks out there seem to be releasing their [tag]Continuous Integration[/tag] builds. A lot of tools even seem to have this notion built in. For a while now I’ve been assuming that sooner or later we’ll switch to doing things this way, but now that I come round to giving it some proper thought, I’m not convinced it’s the right thing to do. Maybe someone out there will change my mind on this one.


The problem, as I see it, is that by releasing CI builds, I’m giving up control on what I hand over to the customer. Or more precisely, if I want to maintain that kind of control, I have to pay for it by limiting what developers are checking in to the Main/Head of development. At the base of all this is the way that we use [tag]CVS[/tag] and [tag]TFS[/tag] labels. Developers here use release labels to indicate what it is that they intend to release. (Often these release labels are applied on release branches, but that isn’t really important here). Our SCM team use the label to extract, build and ship the various applications we’re building. The fact that developers can label what they wish to be delivered gives them fine-grain control over what goes out, and doesn’t interfere with their decisions to commit work for the next release.

Using CI to create releases means that it is [tag]Cruisecontrol[/tag] (or [tag]Luntbuild[/tag] or whatever CI engine you want to use) that does a build label. One of these CI builds gets labelled with a release label and shipped. What gets labelled depends on what happens to get checked in since the last CI buld. What do you do if you want most but not all of a given group of changesets, perhaps along with a little of the following group? No CI build label exist to represent this! If I want to correlate the CI build labels to release labels, I have to coordinate the checkins that developers are doing. Besides being impractical, this is eerily reminiscent of the kind of heavyweight bureaucracy we have been shunning for almost the last three years.

Now I know there are a lot of clever people out there who release their CI builds, so I’m just curious how it works for them. How do you square this circle?

– brendan lawlor –

  1. #1 by Slava Imeshev on December 1, 2006 - 9:28 pm

    That’s because CruiseControl may be a bit underpowered for build management tasks. And I agree, releasing from the head of the code base is not a good idea.

    Our Parabuild the software build management server solves this problem by providing manual and scheduled builds in addition to CI builds. This way you can pick any build you like, run it as a release candidate using build parameters, publish results, run the result through extensive testing and run an official release build, all in fully managed, automated fashion.

    Hope this helps.

    Regards,

    Slava Imeshev

  2. #2 by Martin Perez on December 2, 2006 - 9:56 am

    Perhaps the problem lie in using the same CI system used by developers. Just I was talking about that this week with a colleague. Normally companies use a single CI system to perform the builds, but as you say, it´s clear that those are developer builds.

    When the complexity grows then an approach that performs better is to split the system and use several CI systems. A master CI system will take the ouputs of those subsystems as inputs and generate a release. Following the same approach, you could create a master CI system for performing customer builds.

    Regards,
    Martin

  3. #3 by Brendan Lawlor on December 4, 2006 - 9:07 am

    Slava,
    I wasn’t aware of your product, so thanks for the information. At the moment we’re switching to Luntbuild, as Cruisecontrol just wasn’t working out in a scaled up environment.

    -brendan

  4. #4 by Brendan Lawlor on December 4, 2006 - 9:09 am

    Martin,
    That’s an interesting point of view. I’m not sure what the function of the customer-build CI engine would be however. That build would be a once-off, and would only be different over time if the label had been ‘slipped’ on one of the files – something we avoid as much as possible.

    – brendan.

  5. #5 by Martin Perez on December 5, 2006 - 10:27 pm

    Brendan,

    My believe is that a process like build – test – release only works with predictable systems even when they can can be complex (frameworks like apache ones, ides like Eclipse, etc. ). But with complex systems that require a heavy integration testing phase, I don´t think that simple approach would be enough.

    An alternative approach can be having a CI (or several is your team is big enough) to build and unit test the different parts of your system. Then, if nothing has failed, the function of the customer-oriented CI would be something like grabbing those outputs from the other CIs, and perform integration tests against a live system; even you can go further and run performance and load tests against that or other system, or even health tests if you have a good instrumentation layer. If some of those tests fails, you have that the product hasn´t enough quality to go outside.

    Obviously, it´s not an easy approach 😉

  1. DSI and MTUG host Agile with Team System and DB Pro event at DeCare Systems Ireland Blog

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: