Feed on
Posts
Comments

My blog post the other day about giving up on test-first development attracted a lot of attention, not least from ‘Uncle Bob’ Martin, an agile pioneer who wrote an entertaining riposte to my comments on his ‘Clean Code’ blog. He correctly made the point that my experience of TDD is limited and that some of the problems that I encountered were typical of those starting out in TDD.

1. I said that TDD encouraged a conservative approach because developers (at least those who think the same way as me) were reluctant to break a lot of the developed tests. Bob suggested that the problem here was that my tests were too tightly coupled with the code and that if tests are well-designed then this shouldn’t be too much of a problem. Looking again at my tests, I reckon that they are too tightly coupled to the code and they can be redesigned to be more robust.

So, I think that Bob’s right here  –  this is a problem with my way of thinking and inexperience rather than something that’s inherent in TDD.

2. I made the point that TDD encouraged a focus on detail because the aim was to write code that passed the tests. In fact, one of the things I read when getting started with TDD, was ‘Uncle Bob’s three rules of TDD‘:

  • You are not allowed to write any production code unless it is to make a failing unit test pass.
  • You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
  • You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

If this isn’t advocating a focus on detail, then I don’t know what it’s saying.

Bob says that ‘Code is about detail; But this doesn’t mean you aren’t thinking about the problem as a whole‘. But how do you think about the problem? Maybe Bob can keep it all in his head but I think about problems by developing abstractions and denoting these in some way. I don’t like notations like the UML so I do it as a program. So how do we think small AND think big without writing code that isn’t about passing tests? Or have you changed your mind since writing your three rules of TDD,  Bob?

3. I made the point that TDD encouraged you to chose testable designs rather than the best designs for a particular problem. Bob was pretty scathing about this and stated unequivocally

“ Something that is hard to test is badly designed”

But we know that systems made up of distributed communicating processes or systems that use learning algorithms are hard to test because they can be non-deterministic – the same input does not always lead to the same output.  So, according to Bob,  system designs with parallelism or systems that learn are badly designed systems.  Bob, I reckon you should take this up with the designers of AlphaGo!

4. I said that TDD didn’t help in dealing with unexpected inputs from messy real data. I don’t think I expressed myself very well in my blog here – obviously, as Bob says, TDD doesn’t defend against things you didn’t anticipate but my problem with it is that proponents of TDD seem to suggest that TDD is all you need. Actually, if you want to write reliable systems, you can’t just rely on testing.

Bob suggests that there’s nothing you can do about unanticipated events except try to anticipate them.  To use Bob’s own words, this is ‘the highest order of drivel’. We have been building critical systems for more than 30 years that cope with unexpected events and data every day and carry on working just fine.

It’s not cheap but we do it by defining ‘a safe operating envelope’ for the software then analyse the code to ensure that it always will operate within that envelope, irrespective of what events occur. We use informal or formal arguments supported by tools such as static analysers and model checkers to provide convincing evidence  that the system cannot be driven into an unsafe state whatever events occur.

That’s how we can send systems to Mars that run for years longer than their design lifetime. Accidents still happen but they are really very very rare when we put our mind to building dependable systems.

Just a final word about the space accidents that Bob quotes. I don’t know about the Apollo 1 fire or the Apollo 13 explosion but the Challenger and Columbia disasters were not unanticipated events. Engineering analysis had revealed a significant risk of a catastrophic accident and engineers recommended against the launch of Challenger in low temperatures. But NASA management overruled them and took the view that test results and operating experience meant that the chances of an accident were minimal. These were good examples of Dijkstra’s maxim that:

Testing shows the presence but not the absence of bugs

I think that TDD has contributed a great deal to software engineering.  Automated regression testing is unequivocally a good thing that you can use whether or not you write tests before the code. Writing tests before the code can help clarify a specification and I’ll continue to use the approach when it’s appropriate to do so (e.g testing APIs).  I don’t intend to spend a lot more time learning more about it or consulting a coach because when it works for me, it works well enough to be useful. And, as a pragmatic engineer, when it doesn’t work for me, I’ll do things some other way.

Understandably, TDD experts promote the approach but they do  themselves a disservice by failing to acknowledge that TDD isn’t perfect and by failing to discuss the classes of systems where TDD is less effective.

We can only advance software engineering if we understand the scope and limitations as well as the benefits of the methods that are used.

18 Responses to “Responding to Uncle Bob’s comments on test-driven design”

  1. Ian, I appreciate the chance to interact with you on on this topic. I’ll respond point by point.

    1. No argument.

    2. You are correct that the three laws cause a focus upon detail. But then so does the writing of any particular line of code. In the end, we cannot escape the detail of that code. The three laws are nothing more than the discipline that I use for addressing that detail.

    This is one of the great paradoxes of engineering. Engineering is the art of organizing an impossible amount of detail into a cogent, operational whole. And of all engineering practices, software is the most extreme in this regard.

    One again I’ll quote Ron Jeffries advice: Act locally, think globally. The three laws of TDD is how I act locally.

    3. It is true that complex software systems can exhibit quasi-non-determinism. But every complex system is made out of deterministic parts. Those parts are perfectly testable. Moreover, it is the role of the designers to discover and suppress any non-deterministic behavior in the system as a whole. After all, another word for non-deterministic behavior is: “bug”. Therefore, any system with a design that is untestable due to non-determinism is, by definition, buggy, and therefore badly designed.

    There is no escape from this. Non-determinism is a bug. Designers must be able to demonstrate that their systems are deterministic. That demonstration involves tests.

    Let me put it this way. The Forbin Project was a fun movie; but Forbin was a very bad designer.

    4. I mostly agree with this point. I quite agree that proponents of TDD who claim that TDD is all you need are peddling snake oil. You need a lot more than just testing in order to build high quality, high reliability systems. Among those other things are good coding skills, good design skills, high order systems thinking, deep analysis of all potential inputs and confounds, and a chair for Murphy at the breakfast table.

    As a TDD expert (self proclaimed, but willing to defend the title against all comers) I agree that it is just as important for me to identify those areas where TDD is not effective, as those areas where it works well. And, indeed, I have done so in my blogs, tweets, and books. TDD is not a panacea, a magic bullet, or the keys to the kingdom. It’s just a discipline. A good discipline that I will aggressively defend; but a discipline nonetheless.

    I also agree with the operating envelope argument. Of course that’s what we try to do. We try to define the extremes, add a safety margin, and then live within that margin. Our problem is that we sometimes suck at doing this.

    Apollo 1. We did not identify risk of locking three men into a capsule and pumping in 20 PSI of pure O2. OK, hindsight is 2020; and, I mean, DUH! But we simply missed that part of the operating envelope.

    Apollo 11. During the descent of the LEM, just before landing, the computer system exceeded its processing load envelope several times. The engineers on the ground simply moved the envelope by telling the astronauts to ignore the alarm. Ooops.

    Challenger. Yes, the engineers at Thiokol knew that the envelope had been breeched; and they tried their damnedest to get the managers to listen. This is a case where the engineers had an operating envelope, but the management procedures did not. A failure of the overall system.

    Columbia. The engineers never anticipated that a small block of insulation foam, shaken loose by the vibration of launch could achieve relative velocities sufficient to punch a hole in the orbiter. They improperly defined the operating envelope of that part of the launch system.

    We sometimes suck at this.

    Finally, a word about Dijkstra’s statement that tests can only prove the presence of bugs, not their absence. Dijkstra was trying to define a way to mathematically prove programs correct. That was the ultimate goal of “Structured Programming”. He failed. Formal proofs, while possible if you limit your control structures as Dijkstra recommended, are entirely impractical.

    Dijkstra had hoped that over time a library of Euclidian style of Theorems would be built up from simple postulates. His dream was the creation of a mathematical superstructure that programmers could use to quickly prove their programs correct. The five decades since have unfortunately shown that to be a pipe dream.

    But all is not lost. Indeed, though we cannot rely on Euclid, we can turn to Bacon, and Galileo. We can use the hypothesis-experiment cycle of the scientific method to “prove” our programs correct. After all, the scientific method is precisely subject to Dijkstra’s complaint. Experiments can only prove an hypothesis false; they can never prove it correct. And yet we have built up a superstructure of physical laws, based upon this _falsifiability_, that we are blithely willing to bet our lives upon.

    Every time we get into a car, or a plane we are betting our lives that Newton was correct. Why? Because in the frame of reference that our cars and planes operate the Newtonian test suite passes.

  2. Monica says:

    (Ian and Bob: it is sooooooooooooooooooooo good to hear two sensible people having a respectful disagreement on the internet. Bravo, sirs!)

    Bob, I respect that you are a TDD expert, so I ask again: where are all the links to open source TDD code out there?

    You must defend TDD a dozen times before breakfast: have you not yet compiled a list of links to open source TDDed software that you are happy to offer up as good design for the rest of use to examine and learn from?

    Wouldn’t that be the ultimate argument FOR TDD? What am I missing here?

    And if TDD doesn’t work for open source projects, why do you think this is?

    • Kevin Roche says:

      Have you tried looking at github: reckon you’d find many hundreds, probably many thousands of examples?
      What makes you think open source projects are so much better an indicator of the credibility of TDD than any other type of development?

      • Monica says:

        Hi Kevin,
        Searching on github for “tdd java” gives 538 hits, but these are overwhelmingly TDD tutorials. I was rather hoping to find entire non-trivial (>5000 lines) projects produced not to explicitly demonstrate TDD, yet whose superb design nevertheless showcases TDD’s benefits incidentally.

        And I don’t think open source projects are a better indicator of credibility, it’s just that I can’t ask to see closed source projects because they’re … closed.

        • Kevin Roche says:

          Hello Monica,

          Your final sentence “And if TDD doesn’t work for open source projects, why do you think this is?” seemed to imply that open source projects are necessarily superior in quality to non-profit open source; my misunderstanding sorry.
          I agree if you type “tdd java” you will likely just get tutorial, but I meant that you should look at some actual projects; I suspect many of them will contain TDD code.

    • Monica, the first to look at is FitNesse. See fitnesse.org. Also see https://github.com/unclebob/fitnesse.

      Next check out Keith Braithwaite’s articles comparing TDD project with non-TDD projects. That’s very revealing. https://sites.google.com/site/unclebobconsultingllc/the-quality-of-tdd

      • Monica says:

        Thank you, Robert, for pointing me at that. I had no idea that –
        and I highly commend you for – putting your own software out there as
        a TDD example.

        However I’m more puzzled than ever. I find Fitnesse to be highly
        coupled. Can you please show me how I’m wrong?

        Run JDepend and point it at your fitnesse-standalone/fitnesse
        directory (I downloaded the latest from
        http://fitnesse.org/FitNesseDownload).

        Here’s an example of good coupling: fitnesse.http depends on package
        util (I’ll leave out the leading “fitnesse.”), and package util
        depends on no other fitnesse package, which is great (especially for a
        utilities package).

        But take almost any other package and you find they depend on nearly
        everything else.

        For example, root package fitnesse depends on authentication. But
        authentication depends on fitnesse (immediate circular dependency),
        components (which also depends back on fitnesse), and
        template. template depends back on fitnesse, on components (hence back
        to fitnesse), on html (which is fine) and on wiki. Bit wiki depends on
        components, util and parser. parser depends on (again) fitnesse, html
        and tables. tables depends on slim which depends on converters which
        depends on slim, etc.

        Chose any package and see if it doesn’t lead to an explosion of
        dependencies that couples it to most of the rest of the
        packages. reporting depends on components and so circularly back to
        fitnesse. responders seems to depend on everything straight out of the
        gates.

        (testsystems.fit, however seems well-designed.)

        If TDD leads to “much more decoupling” and “forces you to create
        better, less coupled, designs” then how do you explain this? Why
        didn’t TDD give you feedback on this coupling?

        • Pablo says:

          What a nice way to say that fitnesse code is… ugly. Well, it is, is really really ugly.

          Too bad is the only open source project that Robert can point to.

          Thanks Monica!

      • Daniel says:

        Monica made her case very well. It would be wonderful to see an Uncle Bob response. This kind of share is what teach us how to face real world problems when following such disciplines and what show us why we can’t just give up when those problems appear. Sadly, till this day (09-07-2016) we have no answer of Uncle Bob yet.

        Please, Uncle Bob, give us an answer to those questions.

  3. […] Responding to Uncle Bob’s Comments on Test-Driven Design (via David Whittaker) […]

  4. Philip Schwarz says:

    Being a TDD expert doesn’t preclude pragmatism nor its advocacy.
    Kent Beck for example: see https://www.facebook.com/notes/kent-beck/when-tdd-doesnt-matter/797644973601702
    I made a summary slide, attached here: https://twitter.com/philip_schwarz/status/714948590553186304

  5. Miguel Gouveia says:

    If the majority of open source projects don’t use TDD it doesn’t mean that it is a bad discipline.
    Maybe TDD have not reach the necessary maturity, maybe it is not teach well enough to the young developers out there.
    Open source project are not perfect. And it’s hard to have a open source project that forces a no so well understand discipline. There are a lot of successful open projects out there that have pour design and have bugs. Maybe we must start thinking a way to effectively teach TDD to open source developers. Bob Martin is helping, but is not easy.

    • Monica says:

      Hi Miguel,
      I agree with every point you make.

      If the majority of open source projects don’t use TDD, this is certainly NOT an indicator that TDD is a bad discipline.

      But Uncle Bob once wrote that, “Indeed, if you follow the three rules [of TDD], you will find yourself doing much more decoupling than you may be used to. This forces you to create better, less coupled, designs.”

      I was just hoping there’d be a authoritative cannon of such better, less coupled designs known to the TDD community, which would serve to demonstrate the power of TDD to us struggling initiates.

  6. Philip Schwarz says:

    @Monica open source TDD code examples?

    The Spring Framework, JUnit, the ECLIPSE IDE, plus more – see ‘What are examples of substantial open-source software written using test-driven development (TDD)?’ https://www.quora.com/What-are-examples-of-substantial-open-source-software-written-using-test-driven-development-TDD?srid=Wf2

    • Monica says:

      Hi Philip,
      Wonderful! Thank you so, so, so much for this link! It could hardly be more perfect.

      JUnit! By the FATHER of TDD itself! (Though he humbly talks about “rediscovering” it, I see). What a perfect, perfect first project to analyze. I’ll rev up my JDepend and examine its coupling, along with Uncle Bob’s SOLID principles. And Spring, too (I imagine it’s quite small). Eclipse, presumably, is monstrously large, but I’ll also study it, in time.

      I can’t believe I didn’t think of JUnit. Being written by Mister TDD himself, it must be a textbook example of the “better less coupled design” that Uncle Bob encourages. And not only that, but Mister Beck HIMSELF offers it as an example of good design produced by TDD.

      You’ve made my day, Philip. I’ll post a super-super-brief analysis-summary back here (unless Ian’s getting tired of people using his comment sections as a soapbox – sorry about this, Ian, but you did start one helluva fantastic discussion!).

  7. Monica says:

    I’ve spent some time studying JUnit’s design (rev 4.12).

    I see one very positive and one very negative issue.

    The positive is that JUnit has beautifully small methods, classes and packages. Taken individually, they’re delightful, with methods less than 20 lines long, classes less than a few hundred lines long, and packages with simple class interactions. Kent has used TDD to drive the design towards an excellent application of Single Responsibility Principle.

    The negative is that JUnit’s design is extremely coupled. If you open the JDepend Graphic UI, it shows JUnit’s 30 packages their “Depends Upon” (Efferent) dependencies. Clicking on them shows the packages than any particular package depends upon. For a well-designed system, clicking on any package will show the small number of packages depended upon, eventually leading to packages that depend upon no others. But for JUnit, clicking on almost any package leads to ever-regressing hierarchies of depended-upon packages because so many packages are coupled to one another in package-dependency cycles. Kent has used TDD to drive JUnit’s design towards a very poor application some of the rest of the SOLID design principles.

    In his Dependency Inversion Principle, Uncle Bob writes:

    “A piece of software that fulfills its requirements and yet exhibits any or all of the following three traits has a bad design. It is hard to change because every change affects too many other parts of the system. 2. When you make a change, unexpected parts of the system break. 3. It is hard to reuse in another application because it cannot be disentangled from the current application.”

    Given JUnit’s prolific coupling, it is hard to see how it would not exhibit to all three traits.

    Why is it so heavily coupled on package level? It seems that there are several core classes that are depended upon by many others and that themselves depend upon many others, in many other packages. Classes like ParentRunner, JUnitCore, Assert, Categories all tie together their parent packages, without a clear delineation between service-user and service-provider. As Uncle Bob says, “HIGH LEVEL MODULES SHOULD NOT DEPEND UPON LOW LEVEL MODULES. BOTH SHOULD DEPEND UPON ABSTRACTIONS.” But it’s unclear in this design what is high-level, what is low-level and where abstractions should be defined.

    JUnit displays a coupling that should not be encouraged. (It would be instructive to analyze why its tests failed to give feedback of this coupling.) Uncle Bob’s claim that by doing TDD you will spend more time decoupling leading you to “create better, less coupled, designs” seems, in this instance, false.

    I’ll look at Spring next.

  8. Huy Hoang says:

    Now I understand why they named you Uncle Bob. Your points are awesome!

  9. owen says:

    I am still doing my research but TDD seems to be trying solve problems that arise because of OOP overuse; writing programs in classes that eat their own dog food until you don’t know your head from your tail. Round and round you go.

Leave a Reply