The Australian Government Guide to Policy Impact Analysis discusses the importance of keeping risk in perspective. Our role as policy makers is to provide advice to governments about acceptable levels of risk—taking into account the possible consequences—and how much it will cost the community to reduce or eliminate that risk. An effective approach to the analysis of risks requires you to develop a thorough understanding of the nature of the risks. This can be achieved by soundly applying risk analysis and economic evaluation principles.
Given the importance of keeping risk in perspective, the purpose of this guidance note is to advise you on how you can approach the evaluation of regulation aimed at managing risks in Impact Analyses.
Contents
- Defining risk and uncertainty
- What is the policy problem you are trying to solve?
-
What is the likely net benefit of each option?
- Assess the impact of the proposal on risk
- Quantify the risk
- How are risks and hazards measured in practice?
- What if risks cannot be quantified?
- Assess the impact of each option by applying cost–benefit analysis
- Consider potential unintended consequences of each option
- Dealing with uncertainty in risk analysis
- Tools for dealing with uncertainty
- How will you implement and evaluate your chosen option?
- Assistance
Government policy rarely deals with certainties, and is often designed to reduce the likelihood of harmful or hazardous events occurring. Examples include change to:
- reduce the incidence of workplace accidents
- reduce public health hazards (for example, food standards)
- reduce risks from faulty consumer products (for example, product safety standards)
- reduce the risk of financial institution failure
- reduce the risk of terrorist attacks.
The Australian Government Guide to Policy Impact Analysis (the Guide) discusses the importance of keeping risk in perspective. Our role as policy makers is to provide advice to governments about acceptable levels of risk—taking into account the possible consequences—and how much it will cost the community to reduce or eliminate that risk. An effective approach to the analysis of risks requires you to develop a thorough understanding of the nature of the risks. This can be achieved by soundly applying risk analysis and economic evaluation principles.
Given the importance of keeping risk in perspective, the purpose of this guidance note is to advise you on how you can approach the evaluation of change aimed at managing risks in Impact Analysis.
Defining risk and uncertainty
When talking about risk, a distinction is often made between the terms ‘risk’ and ‘uncertainty’. Risk usually refers to situations in which the likelihood of a hazardous event occurring is reasonably well known and can be reasonably estimated. Uncertainty refers to a situation in which the likelihood of a hazardous event occurring cannot be reasonably (or reliably) estimated. In practice, this distinction is often difficult to draw because, while probabilities can usually be assigned to most events, there is seldom complete certainty about the size of all risks. Even when a risk can be quantified with some level of certainty, there will be uncertainty associated with the measure of risk.
Risk analysis is an important part of the Government’s Impact Analysis requirements and, where relevant, must be incorporated into the development of the documents. The purpose of risk analysis in Impact Analysis is to shed light on sources of uncertainty about the possible impacts of proposed policy on outcomes. Risk analysis should not be seen as a distinct step in the Impact Analysis process but should be considered throughout each step of the Impact Analysis process, with particular emphasis on defining the problem and considering the impacts.
What is the policy problem you are trying to solve?
As discussed in the Guide, Impact Analysis Question 1 should clearly define the problem you are trying to solve and demonstrate why it is a problem. In the case of change aiming to mitigate risks or reduce harms, the problem definition should identify the underlying cause of the problem, its seriousness and the Government’s capacity to deal with it. The Impact Analysis should clearly identify whether there is a case for government intervention. This means that the Impact Analysis should contain relevant information on the likelihood of the risk as well as the consequences of the risk (discussed below). The problem section should also clearly identify who bears the risk and highlight how the risk is currently being dealt with.
Actual versus subjective risks
The Impact Analysis should focus on objective risks rather than ‘perceived’ risks. Perceptions about risk can be founded on bias and misinformation about the true magnitude and severity of risks (Viscusi et al. 1995). Individuals can often perceive a risk (or harm) to be much greater than it actually is—especially when there is a lack of information about the risk or strong perceptions about the size of the risk. To avoid this error, you should focus on evidence about actual risks and seek to quantify the actual risk.
Perceptions about risk, however, should not be entirely overlooked. If there is inadequate information about a risk, or public misperception about a risk, that may call for government action to reduce the degree of misperception about the risk. For example, there is often considerable public misperception about many health risks. A possible response would be to inform the public through appropriate campaigns aimed at reducing the extent of misinformation.
Likelihood versus consequences
The size of a risk is usually characterised by the likelihood of an event occurring (that is, the probability of the adverse event or harm occurring), and the consequences should the event occur.
Measuring risk can be a difficult task but can be achieved using reliable sources of information. Quantifying the magnitude of the risk is an important first step because it will inform the assessment of the likely net benefit of each option (and cost–benefit analysis) at a later stage of the Impact Analysis process. Sources of information to identify the likelihood and consequences of risks include:
- accident and incident data (such as fatality rates and incidence statistics)
- coroners’ reports
- actuarial estimates
- medical research data (such as medical studies and epidemiological studies)
- expert panel surveys
- targeted surveys.
You can use information from these sources to inform stakeholders and decision makers about all the relevant facts so that they can make an informed decision. When using data from secondary sources as evidence, it is important to ensure that the estimates used appropriately characterise the risk associated with the problem being considered.
Where perceived risks are deemed important, the Impact Analysis should also inform stakeholders about general risk perceptions.
Some risks, however, will be very difficult to quantify; in these cases, sound qualitative assessments can be used to supplement quantitative analysis. Discuss the size of the risk by reference to the likelihood of the event occurring and the consequences of the occurrence. This assessment may be supported by a discussion of the factors that contribute to the likelihood of the risk and by reference to the impacts of similar events occurring in Australia or overseas.
The nature of the risk
The Impact Analysis should describe the nature of the risk and the adverse outcomes that could eventuate in the absence of government action. Is it a risk that is straightforward to estimate (actuarial-type risks) or is the risk characterised by random events (such as flood, earthquake or terrorist attack)?
The evaluation and analysis should consider whether there is a long delay between an adverse event and the consequences (latent risks). The adverse consequences associated with climate change, for example, will occur gradually over time. This means that the impacts of the associated consequences of climate change will most likely happen slowly over time. Some risks, such as the risk of terrorism, tornados, floods and some disease outbreaks, are characterised by random events that occur instantaneously and may require a rapid response to deal with the impacts.
Who bears the risk?
The Impact Analysis should discuss which groups will bear the consequences if an adverse event occurs (that is, the parties affected). The distribution of the risk may have important consequences for efficiency (some parties may be able to bear the risk at lower cost than other parties) and equity (it may be more acceptable to society for some parties to bear risk than others). The Impact Analysis should outline whether the distribution of risk is an important consideration.
What is the likely net benefit of each option?
As discussed in the Guide, Impact Analysis Question 4 should clearly assess the likely net benefit of each regulatory option. This section of the Impact Analysis should give the decision maker a fair and balanced assessment of the costs and benefits of each option. This means that you should assess, where significant, the economic, social and environmental costs and benefits as well as how those costs and benefits are likely to be distributed.
Assess the impact of the proposal on risk
You need to examine the impact of each option and how each option will manage the risk. Your analysis of the impacts should clearly spell out how each option will affect the size and distribution of the risk. For each option being considered, this involves answering the following questions:
- Will the option reduce the size of the risk (for example, using standards or providing information)?
- Will the option remove the risk altogether (such as by banning certain activities or products)?
- Will the option transfer risk from one party to another (for example, through mandatory insurance)?
A proposal may aim to reduce risks by imposing specific standards, such as food standards or product quality standards, to improve product quality. In some instances, the proposal might aim to mitigate risk by directly banning activities, such as cigarette smoking in vehicles or certain public spaces. Actions can also be proposed to transfer or redistribute risks (for example, bank deposit guarantee schemes). You should analyse each option clearly in the light of its impact on risk reduction, transfer or elimination, and in terms of its impacts on all those affected (consumers, producers, governments and regulators).
Quantify the risk
Quantify the risks as far as possible.1 The Impact Analysis should consider whether the reduction in risk resulting from the proposed policy has an economic benefit that will be felt in the community. This means that the Impact Analysis should contain your detailed assessment of the size of the actual risk. The analysis should be based on existing sources of relevant evidence about risks or based on specific studies to uncover the size and magnitude of the risk. The analysis in this section of the Impact Analysis will flow directly from the analysis given in the problem section of the Impact Analysis. To apply cost–benefit analysis (CBA), you will need to be informed about:
- the likelihood of the risk (an estimate of the actual risk)
- the impact of the proposed change on probable future risk.
The latter will require that you make forecasts about how the proposed change or option will affect the future size and magnitude of the risk.
How are risks and hazards measured in practice?
Risks and hazards are often measured by the ‘rate’ or ‘average number of occurrences’ for an event of interest per 100,000 persons per period of time. For example, the expected annual number of cases of salmonellosis from seed sprout consumption in Australia was estimated at 924.2 persons per year (Table 1).
Take care to ensure that all estimates used are reasonable estimates of the probability of the actual hazards that you are evaluating. Consult data from different sources to compare and contrast estimates if such data is available.
Decision makers can use evidence on risks and hazards to assist them to prioritise their responses. Understanding the level of risk is important for the impact assessment. For meaningful impact analysis, it is important to understand and form expectations about how each option will reduce risks into the future.
| Annual cases of salmonellosis in Australia | Expected % of annual salmonellosis cases associated with seed sprout consumption | Potential number of cases in the year (nj) | Probability of observing corresponding number of casse in that year (pj) | Expected number of cases |
|---|---|---|---|---|
| A | B | C = (A x B) | D | E = (C x D) |
| 84,056 | 0.75% | (n0) 630.4 | 1.0 | 630.4 |
| 84,056 | 0.33% | (n1) 277.4 | 0.25 | 69.3 |
| 84,056 | 1.67% | (n2) 1,403.7 | 0.1 | 140.4 |
| 84,056 | 5% | (n3) 4202.8 | 0.02 | 84.1 |
| Total (N) | 924.2 |
Note: The number 84,056 is based on the number of attributed cases for 2010 (12,008) in the NNDSS 2010 summary on the Department of Health website multiplied by 7 as per the recommendation of Hall et al. (2006) to take into account the level of underreporting.
Source: Proposal P1004—Primary Production and Processing Standard for Seed Sprouts Regulation Impact Statement, Food Standards Australia New Zealand.
What if risks cannot be quantified?
When sound quantitative evidence is not available, qualitative discussion based on sound qualitative evidence can also be used to inform the Impact Analysis. However, you should attempt to quantify all risks as far as possible, noting that regulatory costs must be quantified in line with the Regulatory Burden Measurement framework. When risks cannot be quantified, various approaches can be used to analyse their potential size and impact. Qualitative tools for risk analysis include:
- using opinions based on expert panels
- using a risk categorisation matrix (prioritising or categorising according to informed judgements about likelihood and severity)
- comparative analysis (for example, comparing the effectiveness of risk management approaches across different jurisdictions).
Assess the impact of each option by applying cost–benefit analysis
Decisions about risk management strategies should be informed by consideration of the costs and benefits of policy. Risk management strategies will impose costs and benefits on members of society and hence will support different efficiency and distributional outcomes. In practical CBA, your aim is to quantify who will be affected by the change and by how much. The key objective is to assess alternative risk management options in terms of their relative efficiency outcomes. This means quantifying all costs and benefits associated with a risk management strategy.
Consider potential unintended consequences of each option
Risk management policies can sometimes elicit adverse changes in the behaviour of economic agents (consumers, suppliers and other actors in the economy). In particular, agents may engage in riskier behaviour if they face a reduced likelihood of adverse consequences of that behaviour or if the consequences are borne by another party. This is known as the moral hazard problem.
For example, proposals aimed at guaranteeing bank deposits may elicit excessive risk-taking behaviour by financial institutions, which increases the probability of financial loss (Davis 2004).
Similarly, care should be taken in designing regulatory penalty regimes. The effectiveness of regulatory penalties will depend not just on the quantum of the penalty, but also on the extent to which individuals perceive that they are at risk of detection and enforcement action. The deterrence value of the penalty, in turn, can be influenced by relevant social norms; the timeliness and salience of sanctions; and the extent to which individuals or businesses can understand the law. The effectiveness of a given penalty may vary from one person to the next.
When evaluating the impact of government intervention, it is important to consider the potential undesirable consequences on the behaviour of all stakeholders affected. This can be taken into account by applying sensitivity analysis within the CBA framework.
Dealing with uncertainty in risk analysis
Many risks are uncertain, which means that their size can be difficult to quantify in practice. In such cases, you can take uncertainty into account by applying sensitivity analysis to your main assumptions. Various quantitative techniques could be useful where the probability distribution of risk is known. Where it is unknown, sensitivity analysis based on ‘best case’ and ‘worst case’ scenarios can be used. When the analysis of the risks and/or uncertainty is not possible, the Impact Analysis should include a qualitative discussion about sources of uncertainty of the risk, and how this might affect the likely outcomes of the policy.
Tools for dealing with uncertainty
Uncertainty in CBA can be taken into account by using various quantitative tools and techniques. Boardman et al. (2006) provide a comprehensive account of different techniques that can be applied to deal with it, including the following:
- Sensitivity analysis can be used when there is considerable uncertainty about the values of key variables used in CBA. When used in combination with probabilistic modelling, sensitivity analysis can provide a powerful tool to gain useful insights into the basis of ‘worst’ and ‘best’ case outcomes. It is often applied to assess the impact of changes in a key variable (a single parameter) on the net benefit estimate. Sensitivity analysis can also be used to check the overall robustness of the CBA results for changes in globally sensitive assumptions (such as cost and benefit estimates and risk estimates).
- Probabilistic modelling can be applied to gain useful insights into the statistical distribution of uncertain variables used in CBA. Monte Carlo analysis can be used to evaluate the effects of changing various assumptions in the CBA and assess the impact on net benefit estimates. This method allows analysts to estimate the degree of ‘variation’ associated with key CBA parameters simultaneously and assess the impact on key CBA results.
How will you implement and evaluate your chosen option?
As discussed in the Guide, Impact Analysis Question 7, given that many risks are uncertain (that is, the likelihood of them occurring is poorly known) and there is the potential for unintended consequences, it is important for the Impact Analysis to contain a detailed discussion of the policy implementation and evaluation plan.
Implementation plan
In the implementation plan section you identify implementation challenges, timeframes and project phases. Your discussion should identify key risks and threats to success that could prevent the effective delivery and implementation of the proposed change or other option. These factors may include resource requirements, information gaps, administrative and compliance issues, and enforcement requirements. A risk management strategy should be developed and maintained in conjunction with stakeholders.
Evaluation plan
The Impact Analysis should highlight the key details of how the selected option will be evaluated during and after implementation. This is particularly important where there is uncertainty about the risks involved because the optimal strategy may change as new information is gathered. The Impact Analysis should identify the most important information gaps and highlight how they will be overcome.
Assistance
If you have any questions about this guidance note, email OIA at helpdesk-OIA@pmc.gov.au or call (02) 6271 6270.
Further information on the Impact Analysis process is in the Australian Government Guide to Policy Impact Analysis.
- Any administrative, substantive compliance and delay costs must be quantified. For further information on these requirements, see the Regulatory Burden Measurement framework guidance note. All other costs must be quantified where possible.Return to footnote 1 ↩