The difference between bugs and requirements – and how to handle them

What's seems evidently different for some people might appear identical to others. Let’s see how different bugs and requirements really are.

There are different ways to look at this question...

By definition

If you look up the definitions of at the terms “bug” and “requirement” they may seem only vaguely similar, clearly distinct

Bug: “A [...] bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.” Software Bug, Wikipedia

 

Requirement: “1) A […] capability needed […] to […] achieve an objective.
2) A condition […] that must be met […] by a solution component to satisfy a contract, standard, specification, or other formally imposed documents.
3) [omitted]” IEEE, via Wikipedia

By point in time

But if you look at them by the point in time when they become manifest, they become a bit more similar.

A thing (like maybe some ‘feature’) that is not ‘available’ now, is just a requirement: “Cars should have visible (blinking) indicators that show where they want go” was a requirement before indicators were invented.

A thing that has been there once, but subsequently has disappeared, sometimes is a bug "That car's indicator should blink" is a bug report for a car on the road today.

So in some cases whether you look at a bug or a requirement just depends on the point in time you're at.

By reaction

When one looks at bugs or requirements with regard to the possible ways to handle them it somehow pays off to be sure who is looking. For now lets focus on the customer (product owner, product manager etc. – the person who pays for the things that the development team spends time on).

I don't see any reason at all to react differently – except with regard to further context.

Given that a so-called ‘bug’ surfaces in an already published piece of software that is used regularly the fixing of a bug has to have some budget allocated. And obviously it needs to be prioritized. From a business point of view this means that fixing a bug just is fulfilling ’a condition that must be met to satisfy the formally documented desired behavior’ for this bug ... sounds familiar?

If, on the other hand, the bug surfaces during the acceptance phase of a feature (or ‘system capability’) the situation is straightforward. Since the error wasn't present in the previous version of the software the new feature can't be accepted. The instruments of the respective process should be used to remedy the situation. That could be blocking the work-item, swarming, stop the line etc. – each way the error has to be fixed before the new feature can be accepted. Not a bug – just a requirement that has not yet been met.

By common (agile) sense

Now if we believe in test driven development (tdd) we start implementing each thing by writing some tests that tell us when we are done. If we follow behavior driven development (bdd) we might be able to map requirements directly to one or more bdd-specifications. Which we write first, before we implement the capability.

The first thing we do with a bug is to check whether we can reproduce it. Some people then write a test that demonstrates the bug. (according to the rule, that you just don‘t write any code that isn't required by some broken test)

So bugs and requirements are both firstly described by a test that is ‘red’ (or breaking) and then implemented – not much of a difference here.

By work item type

Now this a thing that even in my opinion justifies to differentiate between a bug and a requirement – or isn’t it?

I think in some cases bugs (from a system that is already in use) are different from other requirements in the ratio between size and value. So while most system capabilities that can be described by user stories are comparatively expensive while bugs sometimes are cheap to (e.g. with little effort) fix and yield a high value by making the user experience much more enjoyable (for example if they don’t have to confirm erroneous error messages any longer).

There is no bug

In the end a bug just is special kind of requirement. And when and whether to fix it should be a business decision just like any other prioritization of requirements. For a web-based shop-system that has the bug “’stay logged in’ doesn't work” and a feature request “Add the capability to use the most commonly requested credit card” the decision what to do next might very well favor the feature over the bug. From a business perspective it might be acceptable for a while if users have to re-enter their credentials every time, even though they activated the “stay logged in” box while not having the capability to pay by credit card can be a real problem for the business.
Some bugs are quite easy to live with, while it’s really a nuisance to live with some unmet requirements.

So – how do you handle your so-called bugs?

till next time
  Michael Mahlberg

The Diagnostics Plug – a missing abstraction in most systems

Another car analogy – this time with iPhone-Support...

As the website tom's guide - tech for real life tells us

“... Every car sold in the U.S. since 1996 features a built-in engine control computer that can be accessed with the right tools. This is called On Board Diagnostics-II (OBD-II), ...”

And actually this information can be accesses all the time – even while the car is driving.

Nowadays it is actually pretty simple to access the diagnostics information from the car. All you've got to do is buy an odb-II bluetooth adapter and an application like DashCommand or Engine Link or the like and you easily can see all the information from the diagnostic sub-system of your car. Some application also allow for the extraction of the error codes and error log. If you use one of these applications you can find out what's really wrong with your car as soon as you notice that something is off – way before an appointment at the garage would be possible.

Having a running (software) system is much like having a running car - it gets harder to work on the parts when there is more load on the system. If the database load is at 80% it becomes hard to run an additional query to find out how many stale entries are in a certain table. And if the server is maxed out, it is not so easy to just run a second instance of the system just to find out if the supposedly optimized css really is optimized. Some systems (like Apples OS-X) have built in mechanics to enable the collections of diagnostic information with a relatively low penalty for the overall performance of the system. The same holds for some web-servers and database systems – but what about your business application? Is it possible to easily find out how many logical errors occurred in the last half hour? How many searches returned too many or too few results? If that isn't possible, you might not yet have had the necessity to start exploring these types of questions.

If you do have the information available it is quite often the key to effective (end probably also efficient) bugfixing and code optimization.

In his work Betrand Meyer introduces us to the concept of “design by contract” and the related concepts of pre- and postconditions as well as invariants. While Betrand Meyer originally thought of aborting the program whenever any of the expectaions (assertions) was not met, I found it a sensible approach – especially for “legacy” systems that have been running for a serious amount of time – not to stop the whole system, but only to make a “note” of the violation and make these notes available via the diagnostics port for everyone who is invested in the health of the system.

By analogy, I think it's a good idea to make the diagnostic information of you system (any system) available at runtime too.

Disclaimer: “Just” make sure that this diagnostics plug is not also providing access to internals that should not be available from the outside... like credit-card information ad the like.

Till next time
  Michael Mahlberg

Testing: How to get the data into the system

Even though the correct term for a lot of the “testing” going on would be verification let‘s just stick with “testing” in the titles for the time being...

General verification workflow

The general way to verify that a piece of software does what it is meant to do seems quite simple:

• Formulate the desired outcome for a defined series of actions
• Put the system in a known state (or the sub-system or the “unit” – depending on your testing goal)
• Execute the aforementioned defined actions
• Verify that that the desired outcome is actually achieved
• [Optional] Clean up the systems [1]

While this process sounds simple enough, there are enough pitfalls hidden in these few steps to have spawned a whole industry and produce dozens of books.

In this post I want to tackle a very specific aspect – the part where the system is put into a “known state”.

Putting the system into a known state might involve several – more or less complex – actions. Nowadays, where it's possible to automate and orchestrate the whole creation and setup of machines with tools like vagrant and puppet it is even possible to set up the whole environment programmatically.

You might not want to that for each unit test, which brings us to the question of when to setup what wich I will try to address in some future post.

The problem with the data

However big or small the test-setup is, one thing that is very hard to avoid is providing data.

The state of the system (including data) if often called a fixture and having those fixtures – known states of the system with reliable, known data – is a fundamental prerequisite for any kind of serious testing - may it be manually or automated.

For any system of significant size if there are no fixtures, there is no way to tell if the system behaves as desired.

Getting the data into the system: Some options

In general there are three ways to get the data into the system

• Save a known state of the data and import it into the system before the tests are run.
  In this scenario the important question is “which part of the data do I load at which time“ because the tests might of course interfere with each other and probably mess up the data – especially if they fail. Consider using this approach only in conjunction with proper setups before each test, amended by assertions and backed up by “on the fly” data-generation where necessary.
• Create the data on the fly via the means of the system.
  Typically for acceptance tests this means UI-interaction – probably not the way you want to go if you have to run hundreds of tests. Consider implementing an interface, that can be accessed programmatically from outside the system, that uses the same internal mechanisms for data creation as the rest of the software.
• Create the data on the fly directly (via the datastore layer).
  This approach has the tempting property that it can be extremely fast and can be implementing without designing the system under test specifically for testability. The huge problem with this approach is that it duplicates knowledge (or assumptions) about the systems internal structures and concepts – a thing that we usually try to avoid. Consider just not using this approach!

So, do you actually have fixtures? And how do you get to your data?

’til next time
  Michael Mahlberg

---

[1]

(One can either put the effort in after the test or in the setup of the test - or split the effort between the two places, but the effort to make sure that the system is in the correct state always has to go into the setup. Cleaning up after the test can help a lot in terms of performance and ramp-up time, but it can not serve a substitute for a thorough setup.

Testing in Production?? Of Course! No Never! … ?!?

Recently I’ve come across a number of discussion on testing in production and whether this is good or bad.

Misunderstandings all the way down

Of course it all depends on your perception of what “testing in production” means. If it means delivering products that ripen at the client (what is called “Banana Software” in Germany) that’s quite different from when it means “being able to probe the running system without (too much) disturbance of vital funtions”

How do other professions handle it?

A little while ago I elaborated a bit more on the subject of testing and I also think most of the ideas from this earlier article are still valid. Testing should contribute to better, and more reliable solutions. Whether this requires testing at creation time, build time, roll-out time or during production, testing at the right level with the right approaches is a great thing – of course!!

What do you think?

’till next time
  Michael

D is for Design ... and T for Verification

That is, if we talk about software development and the acronym TDD.
Often translated as Test Driven Development the acronym has been around since the late 1990s and especially the book by Kent Beck – rightfully at that time in my opinion – made people think of TDD as a development technique.
Now when you ask different people what "development" is, a lot of them might argue that it is about coding only – while others take a much broader view. This leads to many people (including Kent Beck if I recall correctly) pointing out that the second D in TDD is very strongly about the design aspect of development.
Since I came into closer contact with some of the proponents of Exploratory Testing (ET) a couple of years back I can't help but wonder if the whole term is misleading. Of course it is about a certain aspect of testing, but those engineers who "really" do testing in the hardware sector (e.g.: with cars, planes, elevators etc.) would consider such tests only as checks or verifications, which don't require a specialist in testing to perform them. After all, everything that is done in these "tests" is to check whether a certain assumption by the developer is met by the system. (Or if the axle distance is really what the designer specified, or if the elevator cable really is capable of holding the specified weight, or... you get the picture)

The (hardware) testers I know on the other hand do something different – and usually only leave a pile of scrap metal when they are through with their tests. They test how much weight is necessary to break the elevator cable, at which speed or lateral acceleration the torsion changes the distance of the axles (which usually doesn't bear to well with the car) and so on.

And TDD simply doesn't give us that kind of tests. Those tests that look for the unexpected or un-specified. And that is is why we still need a lot of non-automated (e.g. exploratory) testing.

So please bear in mind that in a true cross-functional team TDD has it's place in development but so does true testing know-how besides TDD.

Untill next time
  Michael Mahlberg