There is no such thing as a continuos integration server

Of course the title is reference to the “There is no such thing as a free lunch" adage, also known as TANSTAAFL, but really it is about the fact that people think they have the advantages of Continuous Integration when all they have is a build-server.
Of course this is just another instance of semantic diffusion, but IMHO there really is a huge opportunity wasted by not following the concept of continuous integration.

The original Continuous Integration

When I first came across the idea of continuous integration it was in the context of eXtreme Programming (XP). It was just a practice that required a lot of discipline, a finely tuned set of tests, a sound system architecture, capable developers and a good source code management system.

Low-tech is key

James Shore wrote a piece about “Continuous integration on a dollar a day” back in 2006, which in my regards still holds true even today.
The point here is in both cases – in the original description as well as in the James’ article –, that no task is done until it is incorporated in the “main development line” and it is shown that this main development line is proven to be as error-free as can be at that point in time. And the developer(s) who signed up for that task take it on as their responsibility to make it happen. To ensure that this is handled in an efficient way, integrations are serialized and don't happen concurrently. (James employs a nice token to ensure that)
Simple enough – not much technology needed.

Using this approach you end up with a product that always includes all the work completed at that point in time in a way that could be shipped or installed instantly.

That may sound nice in theory, but in our case...

• ... the tests run too long
• ... our tasks are so small, we would have way to much overhead
• ... out team is too big for that
• etc.

Fair points – let me address them one at a time:

The tests run too long?
That is a very good indicator to make your tests faster and perhaps more expressive. Or change your architecture in such a way that you have more, smaller, independently testable components.

The task are to small for that?
Create slghtly(!) bigger tasks

The team is too big for that?
Your team is too big. Period. Change that!

etc.?
If the CI-approach is not feasible because of «X» it is almost always a good indicator that you have a problem with «X» - even though the case of long running tests deserves a seperate discussion.

The Problem with CI-Servers

Don't get me wrong – I'm a big fan of automated builds and build servers. But my point ist that they just can't provide continuous integration.
being serious about continuous integration means you can never have a red build on the deliverable after a task is completed and integrated. After all making sure that the main line is “clean” is essential to the very definition of continuous integration.

The point of the original CI-concept is: As a developer your job is not done until the main line reflects your work

The point of the so-called ”CI-Servers“ is: “Just commit your current work an start on something new – I‘ll let you know some time in the future if the test still show that the software is okay or if there are any clashes with contributions from your co-workers.”

Therefore build-servers actually promote starting on new tasks before the seemingly finished tasks are completely integrated – that's exactly what they are made for...

And the problem gets worse if your tasks are small and the test are long-running... Then you end up with huge build queues that grow during the day and get cleared up at night. And it takes until the next morning until you get feedback on whether your code is really integrated with the system or you still have to do rework.

So yes, please use a build server – but only as a safety net. And don't call it continuous integration just because you have a server performing your build-runs and unit-tests for you.

’till next time
  Michael Mahlberg

scm & build: Levels of configuration

Here's the first little part of the (to be - or not -) series on configuration management and build management.

Although I wanted to start with some clarification on "Task level commits" I actually concentrated on different levels of configuration in a build environment. Here we go...

The levels of configuration

One of the biggest differentiators between a one-man-show and a team-effort-project are the different levels of configuration that have to be managed – and this is also a point where the quality of the whole build process can be heavily influenced.

Basically – unless the application-to-be is monolithic – there are four levels of abstraction: Machine dependent, user dependent, purpose dependent and (last but not least) project specific configurations. Each of this has to be managed separately and consciously to avoid (to much) manual intervention. Talking about indirection let me cite (once again) David Wheeler to whom the phrase “Any problem in computer science can be solved with another layer of indirection. ” is attributed. As he stated in the second part – which is often omitted – “[But] this usually leads to another problem” so let's have a look at the relative pros and cons of this fine distinction. To start of lets examine each level a bit closer.

Remarks:

By the way: Of course there are at least two Dimensions involved in this topic as well: run-time configuration and build time configuration. For the sake of this argument I'll postpone this discussion towards the [[build]] topic.

Purpose dependent configuration

Let's start with the purpose dependent configuration since this is a concern covered in most modern environments. The purpose I'm talking about is also known as build type or target environment or something similar to that. Typical purposes are “Test”, “Debug”, “Release” or – a bit less frequent – “Integration”. Depending on the purpose of the build there usually are a number of things that differ. For “Test” there might be some hard-wired shortcut to circumvent server-roundtrips or a “don't really send to printer”-entry or some other special behaviour that is meant to make testing easier (or even possible) without imposing side-effects on already installed systems. If you're building for “Debug” – one of the most commonly differentiated purposes – you'll certainly want to include debug information into your code, something you probably don't want to ship (although that could be disputed, but that is another story). “Release” of course is the purpose with which you build the shippable product once all test and QA-work has been done. The necessity of an “Integration” purpose arises only in projects where you need to integrate several sub-products and usually has rather project-specific configuration needs.

And of course there are some things (e.g. logging) that need to be configured differently for each of these levels. But speaking of logging we encounter another type of configuration that should not be mixed with the purpose specific configuration: the project specific configuration of components. While I'll go deeper into those in the next paragraph, the important part with respect to things like logging is to be aware of the fact that some thing have both – a project specific configuration and a purpose specific one. Trying to manage both in the same way can create real nightmares (I guess, everybody who has tried to keep Log4J configuration files useful for an extended period of time without that conceptual distinction knows what I'm talking about)

Project specific configuration

This usually is the first configuration option you come across. Almost any project nowadays uses some reusable libraries. Those of course have to be adapted to the specific needs of the project and thus the first level of configuration indirection comes into existence.

Although these configurations are applicable on many levels – from configuration information specifying a windows' layout to the much mentioned log-file configurations – at least they have a clear association. They are “just another kind of source code” and thus relatively easy to handle.

Machine dependent configuration

This one strikes as soon as there is even one more developer! The path which used to point at /usr/bin has to point to /usr/local/bin, the drive for intermediates that used to be C: has to be E: and the monitor resolution goes from 1024x768 to 1600x1050. Consequently some things have to be configured somehow – and here we definitely need a distinction between build-time and run-time.

User dependent configuration

The distinction between user dependent configuration and machine dependent configuration is a bit hard to make in a time where the correlation of people:machine moved from n:1 to 1:n. But even now – where lot's of people have more than one computer the real relationship is more like n:m since some computers are still shared. Especially build and integration machines are prone to sharing. Now, even on the same machine, the configuration might differ in paths, desired screen resolutions and mounted network shares, so there is basically the same set of configuration information as there is in the machine dependent part, but it needs to be managed in a separate space.

To summarize: We have the purpose specific configuration which is a central [[build]] topic, the project dependent configuration that correlates to source code, the machine dependent configuration that correlates to hardware configuration management, and the user specific configuration that somehow correlates to profile information. All of these should have traceable connections to identify possible configuration errors.

After I have raised all these questions of course I should also answer them – I'll do so some time in the future and will provide a follow-up link in this post...

I think that even the concept to have different levels of configuration enables people to create more stable build environments.

Cheers
_MM_