Feed on

This post has been prompted by a discussion about wind power that I had recently. I have made clear that I am totally opposed to the construction of wind farms on ‘wild’ land. However, I am not opposed to wind turbines in principle. I think there is a place for a (relatively small) proportion of our energy supply to be met by wind energy. After making this point, it was recommended that I look at a number of articles which, it was claimed, ‘proved’ that wind power could never be economic.

I looked at these articles and the thing that I found most striking was that the authors looked at wind power in isolation rather than as part of a wider power generation system. They either were ignorant of or deliberately chose to ignore, critical factors that, in practice, meant their analyses are of no real value.

The energy supply system in a developed economy is a complex system. A complex system is one where there are a large and dynamically changing number of components with relationships between these components. Because of the dynamic nature of these systems, it is theoretically impossible to have complete knowledge of them so we can never produce a completely accurate mathematical model of their behaviour or other characteristics, such as whole life-cycle costs.

Complex systems generate enormous volumes of data. This data is rarely consistent and is frequently contradictory. Because of this, commentators with an axe to grind can usually cherry pick this data to support their own views. Hence, there are pro/anti nuclear articles which appear to be based on objective data analysis. When the alternative perspective is pointed out to those with one set of views, their reaction is often to ignore it or to rubbish the alternatives.

To provide a national energy supply, we cannot simply make decisions based on the cost of a generation technology – we must also consider at its availability (can it deliver power when needed) and its political and environmental risks. Availability is a key issue as continuity of supply is essential for the functioning of our society.

For example, we need to take into account the possibility of a nuclear accident, the possibility that political factors will cut off imported gas supplies, the possibility that weather conditions will mean that there is a widespread still weather so wind turbines don’t work and so on. There are lots of other issues and risks – some of which we DO NOT KNOW.

We can consider all of these other factors as RISKS – things that might happen which will lead to additional costs. A conventional economic approach tries to work out these costs and includes them in cost computations. However, if you try and do this in a large complex system such as energy supply, there are so many uncertainties that the resulting conclusions are completely unreliable.

In complex systems, there are things which are unknown and actually unknowable. Many classes of risk fall into this category. So, any assessment of the risk that e.g. Russia will cut off gas supplies to Western Europe so causing a huge spike in gas prices is no more than a guess. Therefore, because we cannot reliability assess risks or the consequences if these risks arise, we simply CANNOT accurately model the whole life-cycle costs of ANY power generation technology.

But we need to build a reliable power generation system so what do we do? We use two techniques that are fundamental to building any reliable system – redundancy and diversity. In power system terms, this means building more capacity than we need because we know we will have to cope with outages and using different technologies to generate electricity. This means that if there are problems that affect one kind of technology, then we don’t lose everything.

This need for diversity means that the Scottish Government policy that a very large proportion of our energy needs will be met from wind energy is highly risky and means that Scotland sometimes has to rely on imported energy. It also means that France’s reliance on nuclear energy is unwise, although less so because so much work has been done in understanding nuclear risk. It means that government energy subsidies for e.g. nuclear energy may make sense even although there seems to be ‘cheaper’ generation alternatives.

This post is already long enough and I don’t want to go into my opinions of what technologies we should be using or to go into more detail of complex systems issues. But there are two takeaways from this:

  1.  Most so-called economic analyses of power generation technologies are incomplete and only consider the price of these technologies. These should be an input to the decision making process but should never be the sole driver of that decision making.
  2. There is no such thing as an objective view of a complex system. We all can only have incomplete views that are biased by our existing knowledge and prejudices. We cannot ‘prove’ that people who argue for/against wind power, nuclear power, etc are right or wrong. They see the system in different ways and simply don’t listen to contrary arguments because these do not fit in with their world view.

It is perfectly valid to write articles, papers and blogs which discuss the costs of power generation technologies. But, some people who write such articles then go on to draw more general conclusions about these technologies in a broader power generation system e.g we should never build wind turbines. If these articles don’t acknowledge systemic uncertainties and risks and take these into account, then I advise you to treat their conclusions with the contempt that they deserve.

So what do I know?
I worked in the engineering of reliable, safety-critical systems (e.g. air traffic control systems) from 1990 to 2005. From about 2003 until I retired in 2014, I studied and analysed large scale, complex socio-technical systems. My father, as a consulting engineer, worked on every large and many small power stations built in Scotland between 1950 and 1975. We had lots of conversations about power stations as I was growing up and I think he hoped I would be a power engineer. To his disappointment, I followed the siren song of software engineering. Sadly, he didn’t live long enough to realise this was probably the right decision.

Tags: ,

In an article entitled ‘From Software Development to Software Assembly’ (currently accessible w/o login) in the most recent issue of IEEE Software, Harry Sneed and Chris Verhoef argue that increasing software maintenance costs combined with skill shortages mean that organisations have to move from original software development to much more extensive use of pre-packaged, off-the-shelf software. Fewer people are required to maintain such systems and overall costs are reduced.

Now I have been reading articles like this which advocate universal software reuse for at least 30 years and it is so self-evidently sensible that the key question we have to ask is why hasn’t this happened?

Well, to a significant extent it has happened. Fears of a ‘millennium bug’ in 2000 prompted many organisations to ditch their existing software and replace this with packages from companies such as SAP and Oracle. Increased use of these ‘enterprise systems’ has continued . My experience is that in sectors , such as healthcare and education, where software provides an essential service but is not an integral part of business innovation, use of packaged software for new systems is now pretty well universal.

However, the authors are right that there is still a lot of software that COULD be standardised but which is being developed as ‘original programs’ rather than using off-the-shelf packages. Given the benefits of standardisation set out in the IEEE Software article, why does this practice continue?

I think that there are four main reasons for this:

  1. Packaged solutions include embedded assumptions about business processes and organisations have to adapt their processes to fit in with these assumptions. In some case, this adds cost rather than reduces cost. I worked with a project where £10 million was spent on a packaged solution that, after 4 years, delivered less functionality that the systems it replaced. The key problem here was the mismatch between the process assumptions in the software and the processes used. Trying to change the processes used to fit the software was practically impossible as it would have required major regulatory change.
  2. The move towards agile development is not really consistent with ERP system procurement. The message that organisations have picked up from the agile community is that requirements and software to implement these requirements can be developed incrementally. So, there may never be a ‘complete’ requirements specification for the system or, if there is, it isn’t available until the software is finished. This precludes expensive packaged software as it is essential to understand almost everything that the software is required to do before committing to an ERP system. I don’t know how easy it is to use modern ERP systems for agile development but I suspect that agile approaches are not widespread amongst ERP system users.
  3. The short-term model of business where shareholders expect very rapid ROI means that managers, in many businesses, are not rewarded for savings that might take years to accrue. In fact, they resist risky package software because they know they’ll be blamed for cost overruns but will be doing some other job by the time any costs are saved. This has been the principal reason why there has never really been widespread investment in engineering for maintainability.
  4. Increasingly, businesses and other organisations rely on software for innovation. Gaining a competitive edge is harder when everyone is using the same package and it’s easy to replicate what competitor’s have done. Therefore, even if there are long term costs, the ability to create new software that’s different may lead to significant short-term benefits.

There are also other social and political reasons why standardised software is rejected by both developers and managers but I don’t think these are as important as the reasons above.

Will the situation, driven by skills shortage, change as Sneed and Verhoef suggest? Maybe skills shortages will be the catalyst for change but I am not optimistic about this. Software-driven innovations will become increasingly important for all industries and this factor alone means that we’ll be developing and not simply assembling software for many years to come.

Yesterday (August 16th 2016), Ford set out their goal of producing a high-volume, fully autonomous vehicle for ride sharing by 2021. This led to a number of follow-up articles including this one, which suggested that Ford’s vision would be counter-productive and would lead to a major drop in the number of privately owned vehicles and hence major problems for large car makers.

This is consistent with Uber’s long term goal of replacing private transport with on-demand cars and, certainly, fully autonomous vehicles would significantly reduce Uber’s costs. But it seems to me that the notion that taxi services such as Uber will make a huge difference to the number of private cars is based on a technical rather than a socio-technical analysis.

Let’s start with the technical analysis. The utilisation of private cars is incredibly low – mostly cars are parked and according to the above analysis the capacity utilisation is about 3%. It makes no sense economically to have a car if services like Uber are readily available with the low costs that fully autonomous vehicles will allow. So, your car will take you to work but instead of parking will then go off and take someone else to work.

BUT, if you live in a city with reasonable public transport and taxi services, it already makes no sense economically to have a car. The costs of car ownership significantly exceeds the costs of using public transport and taxis. Yet people still insist on having a car. Why?

  1. A significant number of people don’t live in cities but in suburbs or outside of the city. The problem with services like Uber is that in less populated areas, there isn’t enough demand for local vehicles so you have to wait a while for a car to arrive (typically, where we live on the outskirts of a city, we have to wait 10-20 minutes for a taxi). People who can afford it have a car because they don’t like the inconvenience of waiting. This will not change with driverless cars.
  2. People who live in a city often have leisure interests that take them outside of the city – they go walking in the hills, sailing, take their kids to the beach and so on. Taking public transport is a pain and, while renting a car is possible, it precludes last-minute decision making. Maybe if there was widespread coverage from companies like Uber outside the cities people would use them but I suspect they would still prefer the convenience of their own vehicle.
  3. A key benefit of car ownership is that it allows you to be reactive. You discover that you are lacking a vital ingredient for a recipe you are cooking, so you jump in your car to get it from the local shop; your son calls saying his football game has finished early and needs to be picked up earlier than planned; the sun is shining so you decide to drive to a local forest for a walk in the woods. The key point here is once you make a decision, you don’t want to wait and you want to be sure that transport is available – something that an external service will never be able to guarantee.

These reasons are all about convenience – we are willing to pay significantly for convenience and Uber-like services, autonomous or not, won’t change this.

Some time ago, I was involved in discussions about designing a computer-based system for so-called congestion pricing for road usage. The idea was to have a dynamically changing price for driving so that people changed their behaviour and staggered their journeys to work. We concluded it would not work for one simply reasons – kids all start school around about the same time.

While, for some jobs, the hours of work could certainly be staggered, this is not really an option for kids going to school. Many parents drive their children to school on their way to work or leave for work after their children leave for school. They don’t particularly want staggered working hours because they have to fit their working time around school days. So, we reckoned that while congestion pricing would theoretically work, its effect in practice would be quite limited.

The same issue affects demand for Uber-like services. To meet the peak demand in the morning and evening, car services would have to hugely over-provision and have an immense amount of spare capacity sitting around most of the time waiting for a call. The economics of this don’t make any sense so there will never be enough taxis to meet the personal transport demands at peak times. So people will continue to have a car because they can’t take the risk of not getting an Uber – their kids would be late to school. And, if they have a car, they will use it if possible.

For a number of reasons, I think their will be a falling demand for personal vehicles in the developed world. Cities, increasingly, are actively discouraging car ownership and better connectivity makes it possible for more people to work from home. Some people would not be significantly inconvenienced without a car and more and more of them will give up their vehicle. But this is not dependent on vehicle autonomy.

I reckon Ford have identified a market opportunity for driverless vehicles for ride sharing although I suspect that, while the technology may be available in 2021, the regulation simply won’t allow this. But, when it happens, I doubt if it will really make much of a dent in private car ownership.

Tags: ,

As I have said in previous posts, I am convinced that the most pressing educational challenge for the 21st century is to develop effective and efficient ways of adult continuous education. Without these, we will consign an increasing number of our citizens to the ‘digital scrapheap’ as they are replaced by ever-more powerful digital technologies.

To provide this type of education, I see no alternative to using digital learning technologies to provide the educational experience that people need. Face to face delivery simply does not scale to this challenge.

UK Universities have, disgracefully in my view, largely disassociated themselves from both the delivery of adult continuous education and research in this topic. There are, of course, obvious exceptions to this such as the OU, but in general the ‘market-driven’ approach to higher education has driven universities to focus almost exclusively on full-time education for people under 25.

There seems very little research in the general area of adult continuous education so we have to turn to more general research on digital education to try and get some clues how to tackle this problem. A research team that is active in this area is the digital education group at Edinburgh University. They have been researching digital education issues for a number of years and have recently updated their ‘manifesto for online teaching’, which was first proposed in 2011.

Manifestos, by their nature, have to be quite generic (OK, waffly) but I think this is (or could be) a useful document that provides a framework for people working in this area to reflect on what they are doing and plan to do.

However, as I suggest in the title of this post, what has been produced is a bit of a mixed bag. Some statements get it absolutely right, some are, in my experience, quite wrong and some are really just a bit daft.

I won’t go through this manifesto statement by statement but will pick out some examples of the good, the bad and the bonkers.

“There are many ways to get it right online. ‘Best practice’ neglects context.”

This is spot-on and really important. When we are dealing with heterogeneous groups of learners, I don’t think ‘best practice’ makes much sense. Even less so when this ‘best practice’ is derived from work with children and young adults and we try and apply it in an adult learning context. However, I would add ‘and culture’ to this statement.

“Aesthetics matter: interface design shapes learning.”

This is so important and so hard to do. Sadly, the designers of the digital learning systems that I have seen for higher education don’t seem to understand this. Unfortunately, good interface design is very expensive and time consuming and clashes somewhat with agile, iterative systems development that has become the norm for this type of software.

“Contact works in multiple ways. Face-time is over-valued.”

I have no idea what the first clause here means but I think that the second is absolute nonsense. Face time is certainly NOT over-valued although it might be argued that ‘face time’ spent delivering material is of dubious value. But when students have understanding problems, simply talking face to face and understanding their concerns beats all digital interactions. Of course, it doesn’t scale – but that’s not the same thing.

“A routine of plagiarism detection structures-in distrust.”

This is wrong and, in fact, using plagiarism detection promotes trust in the fairness of assessment (I taught courses with plagiarism detection for more than 10 years). The vast majority of students do not plagiarise and resent their peers who do and are rewarded for doing so. Plagiarism detection reassures this larger group that there is a level playing field for everyone. I don’t recall any student ever complaining about plagiarism detection, even those who it detected.

“ Algorithms and analytics re-code education: pay attention!”

This strikes me as complete gibberish – I don’t know of any educational practitioner – teacher or university lecturer – who would have the faintest idea of what this means?

“Automation need not impoverish education: we welcome our new robot colleagues.”

I don’t know if the author here has swallowed some of the more ludicrous statements about AI but really, were are a very long way from having robot colleagues in the sense that I understand the word ‘colleagues’. Automated systems can make some data-driven decisions but they can’t understand the concepts of policy, culture, empathy, tolerance and politics that are an inherent part of both working with colleagues and involving them in learning processes.

In themselves, these odd statements are not that harmful. However, the danger with including such statements is that educational practitioners who read this manifesto will simply dismiss this as the eccentric ramblings of academics who are divorced from the real world of learning. This would be a pity as digital education can only succeed if we bring together (sceptical) practitioners and researchers at all levels.

I have been thinking for some years about how we can use digital learning effectively for adult learners, specifically on how to learn computer programming. This manifesto, in spite of its flaws, was helpful as it articulated some of the issues that I have struggled with. I don’t have any answers here but when I finally get round to creating something, I will turn to it as a checklist to help me assess how I’m getting on.

Tags: ,

I’ve written this post from a Scottish perspective and specifically discuss the digital skills shortage in Scotland. However, I think that the problem is much wider, government responses everywhere are equally unimaginative and that innovative approaches to continuous adult education are required across the world.

It is generally accepted that there is a worldwide shortage of people with ‘digital skills’ . Specifically in Scotland, it has been suggested that, in a tech sector that employs about 84, 000 people, there is a requirement for about 11, 000 more people in the near future. Whether or not this number is accurate I don’t know  (I suspect it’s a bit high) but for sure we need more people who are trained to work in the tech industries.

However, in Scotland and I think elsewhere in the world, both the government and industry response to this shortage has been profoundly unimaginative and based on ill-informed guesswork rather than an analysis of the real requirements of companies and a ‘joined up’ view of skills education.

One problem that I have with the current response is that the term ‘digital skills’ is bandied about without ever really saying what this means. It is generally taken  that ‘digital skills’ equates to programming. However, if we look at ‘digital skills’ as a whole, there’s a lot more to it than simply programming. What is the balance of requirements for people with conventional programming skills (e.g. Java), real-time systems engineering (which needs hardware as well as software understanding), web development, digital marketing, systems architecture, etc.? Maybe such information exists (I haven’t seen it) but if it does, those who simply equate digital skills and programming don’t seem to have paid much attention to it.

The most visible response of governments to the ‘skills shortage’ is to promote the teaching of computer science in schools. This has led to a new computer science curriculum and most recently in Scotland, an announcement that a new fund is to be made available to widen access to extracurricular technology activities.  As a software engineer, both of these seem to me to be absolutely good things – the more young people that understand computers and how to program the better.  However, I do not believe for one minute this will make much difference to the digital skills shortage.

I don’t know what it’s like in other countries but the Scottish government has NOT introduced any programmes to encourage computer science graduates into teaching; teachers’ salaries have been effectively frozen for the past  years and fall far behind the salaries that CS graduates can attract in industry.  Why should a graduate spend another year in training for a starting salary that is 25% or more less than they can immediately after graduation.

So – who will teach computer science in schools? Undoubtedly, existing teachers who will do their best but who will have, at best, some in-service CS training and nothing like the knowledge of a computer science graduate.  We could see a situation where poorly-trained teachers put off more students than they inspire.

In Scotland, Scottish university students do not pay fees but the number of local students funded by the government is capped. There has been no significant increase in the number of places in computer science degrees for Scottish students (in spite of a rise in applications) so, if the school initiatives enthuse students, where are they to go?  It’s not enough to stimulate new applications, applicants need to believe that there will be a university place for them.

Across the world, there is a reluctance for young women to sign up for computer science degrees. I’ve written before about this problem and I believe that this is the most pressing issue that we should be trying to address.  The usual response to this problem seems to be simply promoting role models for women to encourage more women into the tech industry. This can’t do any harm but has so far had very little effect and I doubt if it really will make much difference. What could make a difference is making it easier for students who start in some other discipline to switch to computer science and to encourage a wider range of ‘joint degrees’ where students study CS alongside some other subject. But this needs dedicated funding that will encourage universities to provide such courses and encourage women to take them.

All of this exemplifies the lack of joined up thinking in governments who simply do not understand that education has to be looked at as a whole; tinkering with one bit without considering the big picture is unlikely to have the effects desired.

Furthermore, education is not just a stage in life between the ages of 4 and 22. If we are to adapt to a rapidly changing world, education HAS to be a lifelong process with opportunities for people to learn new skills throughout their career.

In this respect, I had some hope that the Scottish Government might just understand the problem when they announced £6.6 million pounds of funding in 2014 to help tackle the digital skills shortage.  This was an opportunity to develop an imaginative response to the skills shortage but, sadly, it was not to be. What did we get? Another coding bootcamp, that offers intensive courses to train about 150 people per year to develop web applications in Ruby.  Hardly an innovative or imaginative approach.

Maybe this will address the market need for Ruby programmers? But what about the systems engineering skills that manufacturing companies need, what about mainstream software development, what about maintaining the millions of lines of existing code, which is a nightmare for all companies?  We have missed an opportunity to understand what skills (not just programming) that are really required by industry and to develop a world-leading approach to meet these requirements.

Effective adult education needs to recognise that education and work go together.  A full-time process for adult education is neither practical nor effective. To make a real difference to the number of people with digital skills, we need a different approach. We need to innovate and to provide a vehicle that blends electronically-mediated and face-to-face learning to allow ‘continuous’ education’ – to bring people into the tech industry and to update the skills of people already there.

While I applaud the efforts to widen the teaching of computer science in schools, I think that industry and government are kidding themselves if they believe that this will solve the digital skills shortage. We need a committed programme of adult education to encourage and support people who want to develop new skills. We need to abandon the idiotic notion that only young people can program and provide opportunities for all ages to learn digital skills. Industry needs to develop approaches to recruitment based on testing the skills of applicants and not simply taking the easy way out of rejecting those without formal qualifications.

The digital skills shortage is a short-term problem but the solutions we need for this problem will have longer-term applicability.  The potential of AI to destroy jobs has, in my view, been over-hyped but, for sure, AI-driven automation will lead to major changes in the professional jobs market. We should be planning for re-education now not waiting for the problem to happen.

We need a new approach to digital skills education – the old ways neither work well nor scale to the number of people who we need and who will want tech education in the future.  I don’t have easy answers to the problem but I am convinced that the place to start looking is in the open-source movement.  Open source has revolutionised software and I think that there is the same potential for ‘open-source, continuous education’. This is the only practical way of addressing the short-term problems of a digital skills shortage and a longer term goal of reskilling people as traditional jobs are automated.

I’ll write more about this in future posts.


My blog post the other day about giving up on test-first development attracted a lot of attention, not least from ‘Uncle Bob’ Martin, an agile pioneer who wrote an entertaining riposte to my comments on his ‘Clean Code’ blog. He correctly made the point that my experience of TDD is limited and that some of the problems that I encountered were typical of those starting out in TDD.

1. I said that TDD encouraged a conservative approach because developers (at least those who think the same way as me) were reluctant to break a lot of the developed tests. Bob suggested that the problem here was that my tests were too tightly coupled with the code and that if tests are well-designed then this shouldn’t be too much of a problem. Looking again at my tests, I reckon that they are too tightly coupled to the code and they can be redesigned to be more robust.

So, I think that Bob’s right here  –  this is a problem with my way of thinking and inexperience rather than something that’s inherent in TDD.

2. I made the point that TDD encouraged a focus on detail because the aim was to write code that passed the tests. In fact, one of the things I read when getting started with TDD, was ‘Uncle Bob’s three rules of TDD‘:

  • You are not allowed to write any production code unless it is to make a failing unit test pass.
  • You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
  • You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

If this isn’t advocating a focus on detail, then I don’t know what it’s saying.

Bob says that ‘Code is about detail; But this doesn’t mean you aren’t thinking about the problem as a whole‘. But how do you think about the problem? Maybe Bob can keep it all in his head but I think about problems by developing abstractions and denoting these in some way. I don’t like notations like the UML so I do it as a program. So how do we think small AND think big without writing code that isn’t about passing tests? Or have you changed your mind since writing your three rules of TDD,  Bob?

3. I made the point that TDD encouraged you to chose testable designs rather than the best designs for a particular problem. Bob was pretty scathing about this and stated unequivocally

“ Something that is hard to test is badly designed”

But we know that systems made up of distributed communicating processes or systems that use learning algorithms are hard to test because they can be non-deterministic – the same input does not always lead to the same output.  So, according to Bob,  system designs with parallelism or systems that learn are badly designed systems.  Bob, I reckon you should take this up with the designers of AlphaGo!

4. I said that TDD didn’t help in dealing with unexpected inputs from messy real data. I don’t think I expressed myself very well in my blog here – obviously, as Bob says, TDD doesn’t defend against things you didn’t anticipate but my problem with it is that proponents of TDD seem to suggest that TDD is all you need. Actually, if you want to write reliable systems, you can’t just rely on testing.

Bob suggests that there’s nothing you can do about unanticipated events except try to anticipate them.  To use Bob’s own words, this is ‘the highest order of drivel’. We have been building critical systems for more than 30 years that cope with unexpected events and data every day and carry on working just fine.

It’s not cheap but we do it by defining ‘a safe operating envelope’ for the software then analyse the code to ensure that it always will operate within that envelope, irrespective of what events occur. We use informal or formal arguments supported by tools such as static analysers and model checkers to provide convincing evidence  that the system cannot be driven into an unsafe state whatever events occur.

That’s how we can send systems to Mars that run for years longer than their design lifetime. Accidents still happen but they are really very very rare when we put our mind to building dependable systems.

Just a final word about the space accidents that Bob quotes. I don’t know about the Apollo 1 fire or the Apollo 13 explosion but the Challenger and Columbia disasters were not unanticipated events. Engineering analysis had revealed a significant risk of a catastrophic accident and engineers recommended against the launch of Challenger in low temperatures. But NASA management overruled them and took the view that test results and operating experience meant that the chances of an accident were minimal. These were good examples of Dijkstra’s maxim that:

Testing shows the presence but not the absence of bugs

I think that TDD has contributed a great deal to software engineering.  Automated regression testing is unequivocally a good thing that you can use whether or not you write tests before the code. Writing tests before the code can help clarify a specification and I’ll continue to use the approach when it’s appropriate to do so (e.g testing APIs).  I don’t intend to spend a lot more time learning more about it or consulting a coach because when it works for me, it works well enough to be useful. And, as a pragmatic engineer, when it doesn’t work for me, I’ll do things some other way.

Understandably, TDD experts promote the approach but they do  themselves a disservice by failing to acknowledge that TDD isn’t perfect and by failing to discuss the classes of systems where TDD is less effective.

We can only advance software engineering if we understand the scope and limitations as well as the benefits of the methods that are used.

In my recent blog post on TDD, several people suggested  that some of the problems that I was having with TDD were that I may have a system with a tightly coupled UI and that I was trying to test through that UI.  They suggested that TDD is more effective when the UI is made as ‘thin’ as possible.  The problems of testing GUIs using TDD are well-known so the general idea is to build systems where a simple UI interacts with the underlying system through an API.  Most of the testing required is API testing rather than interface testing.

I’d like to argue here that this notion is only applicable to one class of system and, even when it is possible to structure systems in this way, it means that we cannot use the affordances of the digital medium to provide a more effective user experience.

If we step back and look at what a user interface is, it is essentially a way of rendering and interacting with an underlying object model (data and operations). If you have a loosely coupled object model where interactions with one object have minimal effects on other objects, a small number of operations and a sequential interaction model then you can develop a ‘thin’ interface. Objects data is presented in forms, operations as buttons or menus and the sequence of objects presented depends on the the business process supported.  Quite a lot of business processes can be represented in this way (they started out with paper forms after all) so its not too hard to build web-based interfaces. These interfaces are rarely aesthetically satisfying and often pretty user-unfriendly, especially for occasional users,  but they are sort-of tolerable and people put up with them because they have no alternative.

However, if we want to render object models, such as an image, and present users with richer, perhaps more natural, forms of interaction, then the idea of a ‘thin interface’ has to be discarded. Take a simple document editor as an example. You can render the document using characters in a single font, with markup used to show how it is displayed. You can use an editor like emacs to access this and memorise a range of keyboard commands to do so. On the other hand, you can render the document as an image where you show how it looks (fonts, colours, spacing, etc.) and allow the user to directly manipulate the document through the image – adding text, changing styles and so on. This needs an awful lot more UI code and you need to find a way of testing that code.

For some applications, such as a visualisation application, the base application object model may be pretty simple and easy to test (using TDD or any other approach). The majority of the application code is in the UI and if TDD is difficult to use in this situation, then it really isn’t surprising that TDD isn’t that effective for this type of application.

If we want to make our systems human-centric then we need to have more natural models of interaction with them and it may make sense for interfaces to adapt automatically to user characteristics and actions.   Our ‘thin’ interfaces are still mostly based on form-filling and menu selection because to make them ‘thin’ there has to be very little between the user and the system object model. It’s a complete failure of the imagination of system designers to think that we can’t do better than this. Thin interfaces may make TDD easier but it is really quite disappointing that we are still building interfaces that, frankly, haven’t really moved on from the interfaces that were presented on 1980s VT100 terminals .

My last post on top-down development attracted a lot of attention from the Twittersphere and lots of comments. The vast majority of these were constructive, whether they agreed with me or not.  I am delighted that the post sparked such a response because we can only improve approaches through challenge and discussion.  It’s well worth looking at Robert Martin’s Clean Code blog where he has taken the time to rebut the points I made (Thanks, Bob).  I think he has some things wrong here but I’ll address them in a separate post.

As I make clear in Chapter 8 of my book on software engineering, I think TDD is a an important step forward in software engineering. There are some classes of system where it is clearly appropriate, one of which is web-based consumer-facing systems  and I believe that the use of TDD in such circumstances  makes sense. I think that the key characteristics of  ‘TDD-friendly’ systems are:

  1. A layered architecture. A point made by several commentators was that, even when GUIs are hard to test, a layered architecture overall simplifies the testing process. Absolutely right – when you can structure an architecture with the presentation layer, the application logic layer and the data management layer, these can be tested separately.
  2. Agreed success criteria. When the stakeholders in a system agree on what constitutes success, you can define a set of tests around these criteria. You don’t need a detailed specification but you do need enough information to construct that specification (and maybe represent that as a set of tests)  as you are building the system.
  3. A controllable operating environment. By this, I mean an environment where you don’t have to interact with other systems that you can’t control and which may, by accident or design, behave in ways which adversely affect the system you are development.  In this situation, the problem is designing for resilience and a deep program analysis is much better for this is better than (any kind of) testing.

I started with TDD on a class of system that met these criteria and I liked it. It worked well for me. Then, I moved on to a system which was concerned with visualizing complex linked structures. Now, the thing about visualization is that (a) it’s often much more difficult to have clearly separate layers – the UI is the program and (b) it’s very hard to have pre-defined success criteria – you basically have to program by experiment and see what works and what doesn’t.  TDD didn’t work. Of course, this may be due to my inexperience but I think there is more to it than this. Essentially, I think that if a system does not have the above characteristics, then TDD is inherently problematic.

It is unrealistic to think that all systems can be organised as a layered model. For example, if you are building a system from a set of external services, these are unlikely to fit neatly into layers. Different services may have different interaction models and styles and your overall UI is inevitably complex because you have to try and reconcile these. If you have a system that involves rich user interaction (e.g. a VR system), then most of the work is in the UI code. I’ll discuss the myth of a ‘thin’ UI in a separate post.

It is equally unrealistic to think that we can always have agreed success criteria for a system, just as it’s unrealistic to have complete program specifications. Sometimes, stakeholders who have significant influence choose not to engage in system development but don’t like what they see when they get it. Some problems, like visualisation, are often problems where you work by trial and error rather than around a definitive idea of what the system should do. If you are not sure what you are trying to test, then TDD is challenging. In those circumstances, you build planning to throw at least one away. And, maybe if you finally get agreement, you can use TDD for the final version.

The problem of a controllable operating environment is one that isn’t often mentioned by software engineering commentators.  When you put software into a complex system with people and other hardware devices, you will get lots of unexpected inputs from various sources.  The classic way of handling this is to force a UI on system users that limits the range of their interaction and so the software doesn’t have to handle inputs it doesn’t understand. Bad data is eliminated by ignoring anything that doesn’t meet the data validation criteria defined by the system. So far, so good. You get frustrated users who can’t do what they want through the UI (think of the limitations of e-banking systems). Unexpected data from sensors just gets ignored.

This is all very well until you are faced with a situation where ignoring data means that your system breaks the law; where ignoring sensor data means that systems fail in catastrophic ways and kill or injure people; where stopping users interacting with the system means that they can’t respond to system failure and limit the damage caused by that failure.

So, sometimes, you simply have to deal with ‘unknown unknowns’. By definition, you can’t test for these and if you can’t test, how can you use TDD? Deep program analysis and review is the only way that you can produce convincing evidence that unexpected events won’t move the system into an unsafe state.

I don’t believe that such thing as a universal software engineering method that works for all classes of system.  TDD is an important development but we need to understand its limits. I may have missed it, but I have never read anything by experienced TDD practitioners that discusses the kinds of system where it’s most effective and those systems where it might not work that well.

Test-first or test-driven driven development (TDD) is an approach to software development where you write the tests before you write the program. You write a program to pass the test, extend the test or add further tests and then extend the functionality of the program to pass these tests. You build up a set of tests over time that you can run automatically every time you make program changes. The aim is to ensure that, at all times, the program is operational and passes the all tests. You refactor your program periodically to improve its structure and make it easier to read and change.

It is claimed that test-first development improves test coverage and makes it much easier to change a program without adversely affecting existing functionality. The tests serve as the program specification so you have a detailed but abstract description of what the program should do.

I deliberately decided to experiment with test-first development a few months ago. As a programming pensioner, the programs I am writing are my own personal projects rather than projects for a client with a specification  so what I have to say here may only apply in situations where there’s no hard and fast program specification.

At first, I found the approach to be helpful. But as I started implementing a GUI, the tests got harder to write and I didn’t think that the time spent on writing these tests was worthwhile. So, I became less rigid (impure, perhaps) in my approach, so that I didn’t have automated tests for everything and sometimes implemented things before writing tests.

But, I’m not giving up on TDD because of the (well-known) difficulties in writing automated tests for GUIs. I’m giving up because I think it encourages conservatism in programming, it encourages you to make design decisions that can be tested rather than the right decisions for the program users, it makes you focus on detail rather than structure and it doesn’t work very well for a major class of program problems – those caused by unexpected data.

  • Because you want to ensure that you always pass the majority of tests, you tend to think about this when you change and extend the program. You therefore are more reluctant to make large-scale changes that will lead to the failure of lots of tests. Psychologically, you become conservative to avoid breaking lots of tests.
  • It is easier to test some program designs than others. Sometimes, the best design is one that’s hard to test so you are more reluctant to take this approach because you know that you’ll spend a lot more time designing and writing tests (which I, for one, quite a boring thing to do)
  • The most serious problem for me is that it encourages a focus on sorting out detail to pass tests rather than looking at the program as a whole. I started programming at a time where computer time was limited and you had to spend time looking at and thinking about the program as a whole. I think this leads to more elegant and better structured programs. But, with TDD, you dive into the detail in different parts of the program and rarely step back and look at the big picture.
  • In my experience, lots of program failures arise because the data being processed is not what’s expected by the programmer. It’s really hard to write ‘bad data’ tests that accurately reflect the real bad data you will have to process because you have to be a domain expert to understand the data. The ‘purist’ approach here, of course, is that you design data validation checks so that you never have to process bad data. But the reality is that it’s often hard to specify what ‘correct data’ means and sometimes you have to simply process the data you’ve got rather than the data that you’d like to have.

So , I’m going back to writing code first, then the tests.   I’ll continue to test automatically wherever it makes sense to do so, but I won’t spend ridiculous amounts of time writing tests when I can read the code and clearly understand what it does. Think-first rather than test-first is the way to go.

PS. I’m sure that TDD purists would say that I’m not doing it right so I’m not getting the real benefits of TDD. Maybe they are right. But I have never found zealots to be very convincing.

Resigning from the BCS

I have been a Member and a Fellow of the British Computer Society for about 30 years. I joined because I thought that it was important to support professional bodies in computing and software engineering. The degrees of the university I worked for at the time (Lancaster) were accredited by the BCS and, rightly, they expected staff in these institutions to support them.

I never thought that the BCS accreditation process was particularly good or contributed much to the quality of education, nor because of our geographical location, did I manage to attend many BCS meetings. Nevertheless, I continued my membership because of my general feeling that this is something that a professional should do.

However, I have now decided that enough is enough and that it’s time to leave the BCS. There are several reasons for this:

  1. Subscriptions seem to go up every year by more than the rate of inflation and I get very little for this subscription. The BCS’s magazine for members (IT Now) is lightweight and, in my view, not worth reading. Normally, it goes straight into the recycling, unread. Most group meetings, which sometimes sound interesting, are in London and the BCS seems to make no effort whatsoever to cater for members outside the south of England.
  2. As I have written previously, I don’t think that the BCS accreditation process is fit for purpose and I have no wish to support this. In my view, it is now damaging the quality of computer science education.
  3. The BCS Academy was supposed to be a ‘Learned Society’, with a focus on more academic computer science. It doesn’t seem to be particularly active, its ‘Research Symposium’ seems to have died a death and those events which take place are mostly in London or the south of England. I can’t really see anything in this for me.
  4. Most of all, I dislike the BCS’s refocusing on ‘Information Technology’. I am an engineer but the BCS insists on calling me a ‘Chartered Information Technology Professional’. I never asked for this and, like I suspect the vast majority of the population, I have no idea what it means. What is seems to me is a ‘dumbing down’ of the whole discipline – exactly the opposite of what’s needed in my view. To be respected, we need to focus on science and engineering not wiffly-waffly ‘Information Technology’.

So, it’s time to leave. With some regret, as I do approve of the principle of professional institutions. The BCS lost its way some years ago when it rejected the opportunity to merge with the IEE (now the IET) to create a software engineering institution and I wonder if it really has any future?

Older Posts »