and why it matters.
It’s one of the knowledge frontiers. Universities all over the world are trying to understand the emerging science of complexity better, because our world and the challenges it’s presenting us, are complex. Things are interconnected making them hard to understand, even harder to influence and impossible to predict. Conventional reductionist approaches – breaking problems down into their component parts, which have dominated problem-solving and scientific methods for 400 years1, are still essential, but are no longer sufficient. Grappling with 21st Century complexity requires contemporary thinking.
WHAT IS COMPLEXITY?
‘Complexity’. It’s part buzz-word, part relatively new branch of scientific enquiry embodying concepts like complex adaptive systems, feedback, nonlinearity and dynamical systems, and part everyday expression for something difficult to deal with. What’s common across all usages of the word is the idea that something is complex when it is hard to understand, hard to influence and hard to predict2.
Even in the academic community where the study of complexity has been a burgeoning area of scientific and mathematical research for over fifty years, there is no single agreed definition of complexity3. But we all know intuitively when we come across it – it’s a term we use to describe a ‘wicked’ problem – a problem that’s contested, that has no right answer, and that crosses several subject areas (or disciplines). It’s hard to make progress, and it’s hard to figure out what to do.
Complex things are complex because many parts interact with and influence each other to produce ‘behaviour’ that is difficult to understand, influence and predict. It’s hard for us to get our heads around all the parts that interact with each other, what the interactions are, and the complex behaviour that the interactions cause. Blood pressure is a useful example. Many parts of the human body (blood, veins, arteries, hormones, and many others) interact to produce someone’s blood pressure (the behaviour). The medical community is still trying to understand exactly how and why blood pressure is ’caused’ – it’s a complex phenomenon. Complexity manifests itself at seemingly all scales – from the minuscule (genetics) to the grand-scale (societies, economies, the universe). And complex systems interact with each other. Cellular systems interact. A human’s endocrine system interacts with the pulmonary system, humans individuals interact with communities, communities interact with economies and the natural environment, education opportunities interact with employment opportunities, health with happiness. And so on ad infinitum, and not just in pairs, but in messy, nested feedback loops where things happen because of mutual and multiple cause-and-effect relationships, not the simple, linear ones we usually explain the world with. For example, literacy levels are not just driven by class size, they are influenced by a whole range of other ‘things’, but also, critically, by how those things interact. The literacy level of a child is partly influenced by their teacher, and their parents, but also by how the teacher and parents interact, and how they interact with the child. Yes, it’s complex already and we haven’t even considered the multitude of other drivers of literacy and how they interact. And that’s just literacy.
Despite the ‘hard’ factor, there is a lot that researchers have found out about complexity over recent decades.
We know that complex behaviour can emerge from the interaction of fairly simple parts. For example, colonies of ants (each one a relatively simple organism) perform eye-opening collective enterprises such as building bridges and efficiently finding food sources4. And we know that concepts such as feedback, non-linearity and adaption are ubiquitous in our world and are the mechanisms of much of the complexity we observe.
Despite their ubiquity, few of us are even aware of them, let alone understand what they mean. So let’s take that diversion (albeit briefly).
Linearity refers to how two things change relative to each other. They have a linear relationship when the relationship between them can be drawn with a straight line. For example – the amount of medicine given to a child (generally) varies linearly with child weight. But the amount of medicine given to a child and its effect on the child, varies non-linearly. Up to a certain amount has almost no effect, then within a certain range, has a safe and helpful effect, then above that amount has a damaging effect5. Non-linear systems are so common that a mathematician, Stanislaw Ulam, likened the study of non-linear systems to the study of non-elephant animals (ie. most systems are non-linear, just as most animals are ‘non-elephant’. But we tend to think about how the world works, overwhelmingly, in linear ways. The result is that we are often surprised by the effects non-linearity has. For example, few governments believed the predictions for the spread of the AIDS epidemic in the 1980s6, and the spread of noxious weeds can take farmers by surprise, because they appear to be a minor problem for many years, but because of their exponential growth, can became a major problem very quickly and catch them by surprise. It’s a version of the proverbial lilypad story. If a noxious weed doubles its coverage of a paddock every year, and it will completely cover the paddock in ten years, how long will it take to cover a quarter of the paddock? Many people give the incorrect answer (2.5 years), which is linear thinking. The correct answer is 8 years, but by that time, the seeds are literally sown, and it’s too late to get the weed under control. That’s a relatively straight-forward example, but the real world has many non-linearities acting concurrently and in combination. This can cause the system behaviour to change radically from one trajectory to another – for example, many climate scientists are worried about the melting rates of ice at the poles (and other climate trends) – nonlinearities mean that these can accelerate (or, decelerate) radically, rather than stay on a linear, predictable trajectory.
People are generally speaking not very good at thinking in terms of non-linearity. We are also typically not very good at thinking about causality. We tend to think in open-loop or ‘one-way’ causality7. For example ‘A’ and ‘B’ are drivers of ‘C’ which may in turn cause ‘D’. Except for simple, everyday cases, we tend to ignore that ‘C’ might also ’cause’ ‘A’ and/or ‘B’, forming a closed loop, called a feedback loop. There are two main type of feedback loops. One, often-cited with the example of the thermostat, is the balancing loop. These loops acts to restore a system to a certain value, just like a thermostat takes the current room temperature and uses that to control the air conditioning system to get the actual room temperature to the required room temperature. The other type is the reinforcing loop. This type can act to grow something (for example, the principal in a bank account depends on the principal itself (and the interest rate), or, to decrease something (if educational outcomes are poor, then employment opportunities are poor and standard of living is poor. If standard of living is poor, then educational outcomes are generally poor, and so the feedback loop continues. Reinforcing loops are commonly referred to as vicious or virtuous cycles.
Self-organisation and adpation
Self-organisation refers to a characteristic of complex systems where the parts interact to do something without any central control, like an ant colony building a bridge or finding food in sophisticated ways without being ‘told’ to. Adaptation refers to the ability of a complex adaptive system to adapt to changes in its environment and context. Ants can adapt to an obstacle in their path by going around it. People and communities adapt to natural disasters. Economies adapt to fluctuations in markets. Human bodies adapt to invading viruses.
Non-linearity, feedback and adaptation occur concurrently in real-world systems, which makes them…. complex. But, concepts such as hierarchy8, system archetypes9, system dynamics, and the mathematics of complexity have revealed patterns in complexity, a ‘simplicity on the other side of complexity’ (a concept that we’ll delve into more deeply in a later article and seminar series) that allow us to improve how we understand complex systems.
WHY UNDERSTANDING COMPLEXITY IS IMPORTANT
The growing body of concepts, language and tools to improve our understanding of complex things helps to reveal insight about how to influence them – to alleviate problems and grasp opportunities. This can be done in very sophisticated ways and also in ways that are accessible to us all every day.
At the sophisticated end of the scale are examples like Professor Kimberly Thompson’s team that uses modelling of the polio virus and epidemics to demonstrate (among many findings) that a policy to eradicate polio, despite having a high up-front cost, would more likely result in a better outcome (fewer overall cases of polio and lower cumulative cost) than a more gradual ‘control’ approach, because the latter would result in significant polio resurgent outbreaks from time to time10.
The everyday applications of complexity concepts are numerous. They include the recognition that complex systems are transdisciplinary (crossing many subject and topic areas), so to understand and influence them requires input from a wide range of people with expertise across all the relevant areas. We also recognise that complex systems are dynamic and often adaptive – they change over time and adapt to their environment, so any strategy to influence it needs to be dynamic and adaptive as well, to keep up.
There are others too – complexity science and some of its spin-offs such as network science have made us aware that cause-and-effect is a bit more complicated that we used to think of it. We should no longer think of simple, single ‘drivers’ as causes for things, but look for several things – interactions between ‘drivers’ , feedback loops, time delays, information flows and second-order influences as contributing ’causes’ of something.
However, it could be argued that while our research communities are rapidly understanding complexity better day by day, many other fields like reporting and journalism, public discourse and policy-making, are, generally speaking, still catching up. Its a missed opportunity, but easy to understand why – the new ways of understanding the world help, but it requires effort to do. It’s still easier to defer to partial analysis, commentary, or focus on the political issues at the expense of the policy discussion. But the implications of glossing over or ignoring interconnectedness in problem-solving, policy and strategy, of any kind, can be significant – unintended consequences, wasted time and money, and unmet outcomes.
Sometimes debate on issues can be superficially polarised to black or white scenarios, when in reality there are many shades of grey. Or, issues are spoken about in very certain terms, when in reality, uncertainty is rife. The nuances associated with major policy decisions are often not well understood or discussed – how they might play out, the possible flow-on effects, the potential unintended consequences, the implications for different stakeholder groups.
Not many of us can cite the lower-order flow-on effects of raising the GST, or deregulating the tertiary education system.
Part of the problem is the ‘24/7’ media cycle, our (arguably) decreasing attention spans due to information being conveyed in soundbites and tweets, seemingly continual polling, short election cycles, the decline of longform journalism, and the attraction of tabloid topics over ‘more serious’ ones. We don’t seem to have the time or space to delve deeply into matters.
Another part of the problem is that the vast majority of us don’t have the language or the understanding of the concepts to think about complexity (who can blame us – many of the important concepts were only discovered or defined in niche scientific circles since the 1960s, and most are still a long way off from entering mainstream thought or our education systems).
There are some exceptions. The United States has programmes to incorporate systems thinking into their K-12 school curriculums11, and New Zealand has includes critical thinking skills in theirs12. Countries such as Norway, Poland and South Korea perform well in standardised tests that check for higher-order thinking skills because of their curriculums’ emphasis on real-world problem-solving and not just memorisation of facts13.
Without awareness about complexity, we do what we’ve been doing for hundreds of years – we oversimplify things. We take a reductionist and linear approach by breaking problems down into their component parts so we can make sense of those. We tend to view cause-and-effect in simple, ‘linear’ terms – ‘A’ causes ‘B’, which fails to recognise what ‘A’ also causes, the other ’causes’ of ‘B’, and what ‘B’ might lead to, or, that ‘B’ might also cause ‘A’ or more or less of itself, via feedback. It’s hard to think in non-linear and feedback terms. It’s not intuitive that more police on the streets and ports to intercept drug supplies can increase the health and justice burden on society (by incentivising drug dealers to mix pure substances with impure additives and to increase prices, in turn exacerbating health problems and crime respectively). Neither is it intuitive that giving food aid to developing countries can increase overall levels of malnutrition (by initially decreasing the death rate, which then leads to an increase in population and fertility rates, and then higher rates of malnutrition as food aid can’t stretch far enough)14. Or that firing staff would decrease profitability (by decreasing morale and then productivity). We also tend to adopt a ‘linear’ or sequential approach to strategy and policy implementation – we tend to design a strategy, implement it and evaluate it (often in terms of whether the implementation timeline is on track, and less in terms of whether the strategy is actually working or not and why)15. At this point there is often little room to change the strategy. What’s been approved has been approved, and there is usually little opportunity (authority, time and resources) to adapt the strategy as we go – taking into account things that change and what we learn about what works and what doesn’t.
Almost all of the time a reductionist approach is a necessary and useful approach (there are after all limits on time, our cognitive abilities and available information), but unfortunately its also usually insufficient – it’s only part of the story. We also need to take a ‘systems’ view of the world and understand the bigger picture – how all the component parts interact and behave together, and how they change over time. It’s no use breaking a complex problem down in to its constituent parts and improving those, without thinking about how they all fit and work together. Just as we wouldn’t expect a mini-moke to win a Grand Prix if it had a Formula One engine fitted (most of the rest of the car would fall apart before it left the pits), neither should we expect domestic violence to be ‘fixed’ by focusing solely on gender inequality16, or educational outcomes for indigenous kids by focusing solely on increasing school attendance (they might physically be in the school grounds but without considering what and how they will spend their time while there, there is no guarantee learning outcomes will improve). Or improving economic security for women in later life by focusing solely on superannuation. You need to re-design the system, not one part of it (unless that one part is particularly high-leverage and allows the whole system to perform as we want it to).
It’s hard to imagine the day when we won’t need to simplify things to understand them, but over-simplifiying things to the point where we forget that there is complexity lurking is often counter-productive. Sometimes the outcomes we hope for don’t materialise or worse, negative unintended effects occur instead. Some problems become entrenched and some opportunities remain elusive, despite effort and resources of massive proportions.
The often cited example of drugs policy illustrates the point compellingly. In an attempt to curb the high burden of illicit drug use on the health and justice systems, intercepting drug importation at the ports, and dealers on the street is a common policy response. Unfortunately, the lower supply incentivises drug dealers to mix the limited pure substance they still have with impurities. This can result in worse impacts on a user’s health. Addicts are incentivised to turn to crime to keep up with an increasingly expensive habit as low supply forces street prices up. The policy can have the opposite effect on the health and justice systems than it intended17.
Often we muddle through and sometimes that’s OK, but when issues are time critical or severe, it’s not. It’s not to say that suddenly thinking in terms of feedback, nonlinearity, multiple stakeholder perspectives and other concepts related to complexity will suddenly fix all the world’s problems. But proponents of these concepts agree that our progress towards these important goals will be better18.
For many, the alarm bells have been ringing for some time. There are calls for less policy based on ideology, “policy-on the-run” and short-term thinking, and more evidence-based policy, longer-term thinking and thinking things through.
Some call for deliberation and deeper thinking of important issues by decision makers19. Al Gore and Australian journalists Bernard Keane and Helen Razer, call for more reason and logic in the public sphere20. Malcom Turnbull argues for improved explanation of policy challenges and trade-offs21. Barry Jones calls for more sophisticated argument and analysis of evidence22. Andrew Leigh, for less commentating on events and more making sense of them in their context23. Ken Henry for more substantive debate on bold policy measures24, Geoff Mulgan for policy based on careful diagnosis of the problem and well-designed solutions (among other things)25, and President Barack Obama for having ‘a serious go at’ problems we want to alleviate26. More generally, Sir Ken Robinson, in his most-watched TED talk of all time, places creative thinking alongside literacy and numeracy, as the skills we’ll need most for problem-solving in the future 27. And Amanda Ripley, in her book, The Smartest Kids in the World and How They Got That Way, argues that the skills we need to thrive in the world are not the ability to regurgitate facts, but higher-order thinking – the ability to use those facts for problem-solving28.
As with most things, not quite everyone agrees on the need for strategy. Some critics point to bad and failed strategies as proof of it’s futility. Others argue that some issues are so complex, any effort to understand them reveals more and more complexity – things are so unpredictable and uncertain, we are better off improvising at the time29. Our counterpoint would be that strategy done well – strategy that is comprehensive, rigorous, adaptive and based on a rich understanding of the issue (as good as we can get, and improving all the time), is more likely to result in the outcomes we’re seeking than other approaches. Humanity is still a long way off fully understanding the human body, or the dynamics of the earth’s atmosphere but good strategy, drawing on interdisciplinary expertise and recognising it will have to be developed iteratively, has resulted in immensely positive outcomes in medicine and sustainability – the polio example mentioned above, and the Montreal Protocol to stem the damage to the ozone layer30, are some of many examples.
For those who agree with the need and value of thinking more deeply about complex and important issues, there is little guidance around on what, exactly, that means in practice. There are compelling speeches, papers, newspaper articles, conference talks and treatises which plead for, and describe at length, concepts and ideas and frameworks for deeper thought and and more purposeful discussion, but they tend to be somewhat abstract and discursive. There is remarkably little guidance on what it is that we actually need to do, to put it all into practice. What, specifically, it is that we should do, to think strategically and to deliberate wisely? Where do we begin? And where do we go from there?
Grappling effectively with complex problems requires both an understanding of what ‘complexity’ actually means, and being able to apply a broad range of skills in order to be any sort of match for it – the ability to think strategically, critically, analytically and creatively are high on the list.
Happily, there is already a lot that we can draw on to deal with complexity better than we do now – from Aristotle’s logic dating from circa 350BC to relatively new disciplines such as system dynamics, behavioural economics and complexity science itself. One barrier is that the knowledge is distributed – fragmented across multiple disciplines and across the private, public, non-government and academic sectors – here and there are many useful techniques and concepts, but there isn’t a handy compilation of them or any consistency in how or when to use them.
That’s not to say that well-known techniques for problem-solving and strategy development don’t exist. Where would we all be without SWOT analyses and the 80/20 rule among many others? They’re just not used consistently. But more importantly, they are not sufficient for making progress with contemporary, complex issues.
TECHNIQUES FOR GRAPPLING WITH COMPLEXITY
At Ponder we think a guideline for grappling with a complex issue in more of a systematic and comprehensive way, has the potential to radically improve how we think about, talk about and go about strategy and policy development. So we compiled one. It can be used for any endeavour. It can be used by anyone – from citizens to our most senior-decision makers. And, it applies for the entire duration of the policy lifecycle (from the very first day to the very last day that a policy issue is on our radar). It doesn’t provide ‘the answer’, or do the thinking for us, or prescribe how to develop a strategy or policy. What it does do, is prompt our thinking, it reminds us of the most important things we should be considering by providing the questions we should be asking (there are 20 of them), and the techniques to help answer them. Used properly, it helps to build a deeper understanding of an issue, to find clever courses of action to achieve an outcome, to think them through, to understand and explain to others how and why the policy should work, and to deal with uncertainty by adapting the strategy to things that change and to what we learn along the way.
Around 350BC Aristotle saw the benefit of making logical thinking systematic. That’s when he conceived of our current systems of deductive and inductive logic. Two and half thousand years later, why aren’t we more systematic about problem-solving?31
So that’s how our guide is structured – a systematic approach for navigating complex issues, structured in a way to provide very practical guidance on what to consider – 20 Questions for policy and strategy design (and techniques to help answer them). There is a pattern to how we problem-solve – the same questions seem to apply to most endeavours. We just haven’t fully recognised that pattern and captured it. The 20 Questions is a first attempt at that – to improve our problem-solving, to save ourselves time, and to make it easier to grapple with complexity, by providing an outline of an approach for doing it. An overview of the 20 Questions appears at the bottom of this article.
The 20 Questions are most useful as a checklist and as a guideline for rigorous and comprehensive strategy development. Atul Gawande popularised the checklist as a way of managing complexity when he, Peter Pronovost, and others saw it’s potential to help medical staff cope with the myriad of tasks they are required to coordinate and execute for critically sick patients. Gawande’s original article in the New Yorker32 led to a subsequent book – The Checklist Manifesto33. which extended the idea that a checklist can help humans manage the complexity of intensive-care medicine to dramatically improve outcomes for very sick patients, to other complex tasks. We’ve extended it to strategy development in complex areas of public policy. As Gawande himself points out, a checklist has two main benefits – it helps remind experts what they should be doing (not easy when dealing with a myriad of competing concerns), and, it establishes the minimum expectations for doing something. Applied to public strategy then, the 20 Questions reminds us of the important things we should be thinking about. Just as forgetting to clean a patient’s skin before inserting a line, risks infection and consequently patient death, so too does forgetting to identify our assumptions that underpin our strategy (and 19 other things) risk a strategy’s efficacy. The 20 Questions is useful even to the extent that a strategy document could be structured into five sections, and 20 sub-sections.
While the 20 Questions (and the techniques to help answer them) can also usefully be used as a toolkit (draw on number 15 – Compelling Communication when you are working on a speech or report), they are significantly more effective if they’re considered as a package deal – it’s necessary to consider all 20 Questions. As mentioned above, not considering one or more of them, puts the strategy at risk. But, there is flexibility in the extent that you consider them – it might be relevant for one problem to consider Consequences and trade-offs extensively, but another problem may require more attention to Compelling Communication.
If such a guide were to ever enjoy widespread use, the benefits would be significant. People accountable for policy would know the potential questions coming their way (from their bosses or Boards, from journalists, from their electorate) and they would need to be prepared to answer them. They would direct their teams to explore those avenues to be prepared with the responses. The policy discussions would become more purposeful.
The 20 Questions are versatile, in three ways – they can be asked of anyone, they can be applied to any endeavour, and they can be asked at any time – from the first day of dealing with an issue to the very last day we think about the issue.
The 20 Questions can, and should be asked of anyone – the electorate can ask their government representatives, public officials can ask the Ministers, Ministers can ask their departments, policy and decision makers can ask their teams, and strategists and analysts can ask these questions of themselves – it’s at the same time, both a way to ensure there is accountability for good (at least better) policy making, and, a proforma for designing and thinking through a strategy.
It’s systematic and generic – we can re-use the same questions when thinking about pretty much any issue. The detail and context for each specific situation come through in the answers.
The 20 Questions can and should be asked in the early days of understanding a problem through to the final days of implementation. The questions remain the same. The emphasis placed on the questions and the answers are what changes over time.
The 20 Questions bring together trusted and proven techniques such as logical reasoning, and engineering design principles, together with more recent but equally influential techniques from fields such as behavioural economics, systems thinking, complexity science and design thinking.
The 20 Questions are an alternative to us all taking classes on ‘strategic thinking’, ‘critical thinking’, ‘creative thinking, ‘analytical thinking’, elementary ‘system dynamics’ and ‘complexity science’, among others. The 20 Questions (and the techniques to help answer them) have it all embedded. Put another way, if you are asking these 20 questions, and answering them properly, then you are thinking strategically, analytically, critically and creatively. We are doing evidence-based policy, we are considering the short, medium and long term, we are doing risk management, and we are engaging the range of necessary people in meaningful ways.
Why 20? It sounds like a lot (and admittedly some of the 20 have more than one concept packed in), but jamming all the necessary points into a list of seven (humans can more easily remember lists of this length)34, hides the level of granularity we need to properly approach complex problems – we run into the over-simplifcation problem again. Compiling the important considerations of strategy design into 20 was somewhat arbitrary, but the number seemed to result in reasonable groupings of like considerations, and 20 seemed to be somewhat less overwhelming than anything larger.
The main critique of this approach is likely to be that it ‘codifies’ or ‘prescribes’ a process that claims it can be applied to a wide range of topics, and that any sort of codification risks eliminating the specifics of each issue. To a very limited extent that’s true. The 20 Questions aims to be a systematic guide that prompts thinking on just about any issue. Yes, it advocates for careful consideration of all 20 questions, but in no particular order, and going as deep into each as the problem would benefit from. This level of codification aims to be specific enough to avoid incomplete, superficial approaches, and generic and flexible enough to be tailorable to any endeavour. Until we get instinctively and routinely better at understanding and thinking about complex issues, the negatives of this systematic and rigorous approach can be managed and are far outweighed by the potential benefits (in our view).
Another potential critique might be that the 20 Question approach seems overly rational, leaving little room for other routes to effective policy such as creativity and intuition, and ignoring the very real role of perception and emotion in most endeavours. This critique is less valid – creativity, intuition, perception and emotion are central considerations to many of the questions and play an important role in most of the others as well. The 20 Question approach is itself systematic and rational, but only at the ‘meta’ level – the approach specifically builds in considerations and techniques spanning the hard-soft spectrum and the creative-analytical spectrum.
The articles that will follow this one will ask the 20 Questions of specific issues (and start to answer them). The aim is two-fold: to contribute a way of improving how we navigate complex challenges, and to build a deeper understanding of specific issues that matter.
It is an experiment, tweaking will be necessary, and it won’t be perfect. We’ll never be able to cover each topic entirely (probably an impossible task), but we do aim to go broad and deep so that you get a balanced and fairly comprehensive understanding of each topic, so readers can form their own, informed judgments.
Please contribute additional arguments, counterpoints, and comments to articles (via in-line commenting, or send us an email via the contact page). Our commenting requirements are on the Guidelines page.
The following references and experts were consulted during the development of this article:
- 1 Complexity: A Guided Tour Melanie Mitchell; Oxford University Press, 2009, p.1
- 2 Complexity – A Guided Tour,Melanie Mitchell, pp.1, 38
- 3 Complexity – A Guided Tour, Melanie Mitchell, p.xii, 48
- 4 Complexity – A Guided Tour, Melanie Mitchell, p.4, 176, 195-6
- 5 New England Complex Systems Institute concept map – non-linear
- 6 The Art of Public Strategy: Mobilising Power and Knowledge for the Common Good, Oxford University Press, 2009, p.34
- 7 Thinking about systems: student and teacher conceptions of natural and social systems, Linda Booth Sweeney and John D. Sterman, System Dynamics Review, January 2007
- 8 The Architecture of Complexity, Herbert A. Simon, Proceedings of the American Philosophical Society, Vol. 106, No. 6, December 1962
- 9 Systems Thinking: A Primer, Donella Meadows, Chelsea Green Publishing, 2008
- 10 Polio Eradicators Use Integrated Analytical Models to Make Better Decisions, K.M. Thompson, R.J. Tebbens, M.A. Pallansch, S.G.F. Wassilak and S.L. Cochi, Interfaces, Vol. 45, No.1, Jan-Feb 2015, pp.5-25
- 11 System Dynamics in K-12 Education: Lessons Learned, Debra Lyneis and Lees N. Stuntz, 2007
- 12 The NZ Curriculum, see Key Competencies
- 13 The Smartest Kids in the World and How They Got That Way, Amanda Ripley, Simon & Schuster, 2013
- 14 The Fifth Discipline, Peter Senge, p. 59
- 15 Systems Thinking in the Public Sector, John Seddon
- 16 For more discussion see article from The Guardian
- 17 System Failure: Why Governments Must Learn to Think Differently, Jake Chapman, DEMOS, 2nd edition, UK
- 18 The Art of Public Strategy: Mobilising Power and Knowledge for the Common Good, Geoff Mulgan, Oxford University Press, 209
- 19 For example – Evidence-based Policy: A Practical Guide to Doing it Better, Nancy Cartwright and Jeremy Hardie, 2012
- 20 The Assault on Reason, Al Gore, 2007, and, A Short History of Stupid: The decline of reason and why public debate makes us want to scream
- 21 The George Winterton Lecture, 2012, University of Western Australia (and many other speeches since)
- 22 Stupidity is on the rise in our age of enlightenment, Sydney Morning Herald, August 9, 2012,
- 23 The Naked Truth? Media and Politics in the Digital Age, University of Canberra Public Lecture Series, 1 August 2012,
- 24 Ken Henry slams Australian politicians, Business Spectator, 10 May 2013
- 25 Public Strategy: Mobilizing Power and Knowledge for the Common Good, Oxford University Press, 2009
- 26 Interview – President Barack Obama and Sir David Attenborough, The White House, 2015
- 27 TED talk, Do Schools Kill Creativity?, Sir Ken Robinson, 2006,
- 28 The Smartest Kids in the World and How They Got That Way, Amanda Ripley, Simon & Schuster, 2013
- 29 The Art of Public Strategy: Mobilizing Power and Knowledge for the Common Good, Geoff Mulgan, Oxford University Press, 2009, pp.29-31
- 30 Systems Thinking – A Primer, Donella Meadows, Chelsea Green Publishing, 2008
- 31 We recongise that several high-level problem-solving frameworks have been proposed, for example the general process: Conduct research; Identify options; Evaluate options; Implement chosen option), however the ones we are aware of are too abstract to adequately account for complexity.
- 32 The Checklist – If something so simple can transform intensive care, what else can it do?, Atul Gawande, The New Yorker, December 10, 2007
- 33 The Checklist Manifesto, 2011
- 34 The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information, George A. Miller, 1956, Harvard University