Post-implementation reviews

July 2024

On 1 July 2023, responsibility for post-implementation review (PIR) evaluation and monitoring was transferred to the Australian Centre for Evaluation (ACE). For broader advice on the management of ‘ex-post’ evaluations in the Australian Public Service, please refer to the Commonwealth Evaluation Policy and Toolkit, and contact the ACE on evaluation@treasury.gov.au.

Back to top

Summary

A post-implementation review (PIR) evaluates whether the implemented policy is operating as intended and is effectively and efficiently meeting the Government’s objectives in addressing the original problem.

Australian Government agencies need to undertake a PIR for all policy changes that have a substantial or widespread impact on the economy.

A PIR also needs to be prepared when a policy change is made without Impact Analysis that has been through a final assessment by the Australian Centre for Evaluation (ACE). Furthermore, a PIR is required where a policy change is made based on Impact Analysis that has been assessed as insufficient.

A PIR should outline the original problem and the Government’s objectives, provide evidence about impacts of the policy, analyse the impacts, present findings from consultations, and make a conclusion. The impact analysis for a PIR should include information about the actual impacts of the policy, including the regulatory costs. Stakeholder consultation is essential and is a key part of a PIR.

A PIR’s conclusion should provide an assessment, based on the available evidence, of whether the policy remains appropriate and of how effective and efficient it has been in meeting its original objectives.

Regular reviews are useful for evaluating the ongoing performance of policy, and agencies are encouraged to review policy following implementation even if a PIR is not required.

Back to top

Purpose of a post-implementation review

The purpose of a PIR is to assess whether the policy remains appropriate, and how effective and efficient it has been in meeting its objectives. The PIR requirements apply to all agencies that review or make policies that have an impact on businesses, community organisations or individuals, including those agencies with administrative or statutory independence.¹

The Australian Government Guide to Policy Impact Analysis seeks to ensure that policy has efficient outcomes, by achieving its objectives with the least possible impact. For this reason, if a policy change was not subject to scrutiny under the Impact Analysis process at the pre-decision stage, it is necessary that it be subject to a PIR. It is also important for all policy decisions to be assessed within the same framework to ensure the ongoing quality of policies.

Intended audience

A PIR is intended for both decision makers (for whom a balanced assessment of the performance of the policy is critical) and stakeholders (who have a right to accurate, timely, transparent and accessible information about government decisions affecting them).

The PIR is provided to the relevant portfolio minister and the Prime Minister after being assessed by ACE, and is then published on the Department of the Prime Minister and Cabinet’s website at Published Impact Analyses.

Back to top

When to prepare a post-implementation review

It is a requirement to prepare a PIR if any of the following criteria apply:

• The Office of Impact Analysis (OIA) has assessed the proposal as having a substantial or widespread economic impact.
• Impact Analysis that had been through a final assessment by OIA for the policy change was not prepared for the final decision and the change was neither minor nor machinery in nature.²
• A change was granted a Prime Minister’s exemption from the need to prepare Impact Analysis.

An agency can be exempted from the obligation to undertake a PIR where a previously implemented policy is no longer in force (for example, policy has been removed or has expired) or the previously implemented government policy is no longer in practice.

Evidence that a policy is no longer government policy could be in the form of a Cabinet decision, a Ministerial decision or determination, or Federal Executive Council decision that would demonstrate the intent to remove the policy, together with evidence that substantial progress has been made to implement the decision, for example, through preparation and passage of legislation.

Where an exemption from undertaking a PIR is being sought, the relevant Minister writes to the Prime Minister seeking an exemption from the requirement to complete a PIR, following consultation with ACE.

The scenarios included in Appendix 1 to this Guide provide further guidance on how the PIR requirements can be fulfilled.

Back to top

Policies subject to review by another process

In some cases, a policy is subject to a statutory review as it is provided for under legislation. Alternatively, the policy may be subject to another comprehensive and rigorous review, close to the time when the PIR is required.

You can draw on the work done for these other type of reviews to meet some or all of the PIR requirements. In some cases, these other type of reviews can be assessed by ACE as meeting the PIR requirements without needing further work. However, the alternate review will still be assessed against the criteria for a PIR as set out in the Australian Government Guide to Policy Impact Analysis.

If you want ACE to assess an alternative review as part of or in lieu of a stand-alone PIR, you should advise ACE as soon as possible. Because PIRs need to be certified by the agency head or deputy head, your request should be accompanied by an appropriate approval.

Back to top

Post-implementation review process

Start working on your PIR as soon as your policy is implemented. Do not wait until just before the PIR deadline to start the work.

You are responsible for gathering necessary data and consulting stakeholders during and after the implementation of the regulation. Start gathering data from the date the regulation is implemented.

ACE will advise you of any areas in the PIR that need to be improved to adequately inform the relevant minister on the efficiency and effectiveness of the policy and whether the regulation remains appropriate. Once completed and assessed as adequate by ACE, the PIR needs to be sent to the relevant portfolio minister and the Prime Minister. The PIR will then be published on the Department of the Prime Minister and Cabinet’s website (Published Impact Analyses). The PIR process is complete, and your PIR is compliant with the Government’s PIR requirements, only after it has been published on this website.

Some judgement is required in deciding when to start writing the PIR. You need to balance the requirement to gather sufficient data to be able to make robust conclusions against the need to identify and address inappropriate, non-performing or inefficient policy through the review in a timely fashion. ACE can provide technical assistance to help you determine whether or not you have enough information for an informative Impact Analysis. As a guideline, preparing the PIR report should take around three to six months once the relevant data and information has been gathered.

Consultation with stakeholders is an essential source of information for the PIR. The timing guidelines do not prevent you from establishing strong stakeholder consultation arrangements at the earliest opportunity. Ideally, those arrangements would be in place before the implementation of the regulation in most cases.

Back to top

Timeframe for completion

The timeframe for completion of the PIR depends on the reasons for its trigger, as outlined in Table 1.

Table 1: Completion timeframes for post-implementation reviews

Reason for PIR	PIR required to be completed within:
Substantial or widespread impact on the Australian economy	5 years3 of implementation
Impact Analysis not prepared for a final decision on a regulatory change	2 years of implementation
Impact Analysis not assessed by the OIA prior to a final decision	2 years of implementation
Impact Analysis assessed by the OIA as insufficient	2 years of implementation
Prime Minister’s exemption from the need to prepare Impact Analysis	2 years of implementation

Overall, the PIR completion deadlines include the time required by ACE to assess the document for adequacy. You are responsible for allowing enough time to incorporate any changes suggested by ACE during the PIR process. ACE will endeavour to respond on first drafts of PIRs within ten business days. When setting milestones for the PIR in your forward work program, you should consider how the PIR might inform future decisions.

ACE maintains a public register of outstanding PIRs and determines whether or not agencies are complying with PIR requirements. Regular liaison with agencies assists ACE in accurately representing the status of PIRs, including their timeframes for completion.

When a policy is considered to have been implemented

For some measures, particularly those that are extensive in scope, there may be some ambiguity about when the measure is considered to have been implemented. The date on which legislation passes the parliament, or a minister announces a policy change, might not be a useful proxy for the date of implementation.

In consultation with agencies, OIA and ACE determines the ‘implementation’ date when a policy can be considered to have been implemented. Based on the implementation date, ACE then determines the required timing for the PIR in order to maintain the integrity of the policy impact assessment arrangements. Implementation dates for PIRs are reported on the Department of the Prime Minister and Cabinet’s website at Published Impact Analyses.

Back to top

What a PIR needs to include

This section outlines seven questions that the content in a PIR needs to address. There are some similarities between the seven Impact Analysis questions and the content that should be included in a PIR, but a PIR needs to focus on how well the policy is achieving its original objectives.

A PIR should also assess whether the policy remains the most appropriate approach to the problem, and how effective and efficient it has been in meeting its objectives. For the PIR to fulfil its purpose, you need to gather relevant data and information about the functioning of the policy. You also need to engage in effective consultation with affected stakeholders. You should consult ACE to determine the extent of the analysis that will be needed.

The PIR is not intended to be used as a means to critique or praise government policy. You should try to ensure that the evidence is gathered rigorously and presented in a balanced fashion in the PIR.

Here are some tips for writing a good PIR:

• Give the PIR a logical structure and ensure that sections are clearly marked. This will make it easy to navigate.
• Keep it as free as possible from technical jargon.
• Include all relevant information.
• Be concise, ensuring that all the information in your PIR is relevant.
• Be transparent in your analysis by clearly differentiating between facts and anecdotal evidence. Check the accuracy of every claim. Where the data is inconclusive, say so.
• Be measured and prudent in your arguments and avoid making assertions that cannot be substantiated.
• Examine and discuss whether or not the policy continues to be appropriate for its purpose.
• Discuss the impacts (including the regulatory costs) on all stakeholders—businesses, community organisations and individuals.
• Describe the effect of government intervention in a way that helps readers understand whether or not the intended outcomes have been achieved efficiently.
• Provide a degree of detail in your analysis that is appropriate to the impacts of the policy. The larger the impacts, the deeper the analysis should be.
• Present the analysis in a balanced fashion, accurately reporting the impacts (both positive and negative) and identifying any areas of uncertainty in your data.
• Report the views of all stakeholders in a balanced way. This includes acknowledging any views that are not supported by the available evidence.
• Draw logical conclusions based on the evidence and the strength of the data and information presented.
• Ensure that the document meets the Government’s commitment to web accessibility.⁴

Policies, or the broader economic environment in which they operate, can change over time. Therefore, PIRs generally deal with one of three circumstances:

• the policy has lapsed, or is about to lapse
• the policy is continuing in its current form
• the policy is to continue, but in a different form.

The circumstances influence the nature of the PIR. For example, a ‘lessons learned’ PIR that is mainly backwards looking is suitable for a policy that has lapsed. In other cases, it will be apparent by the time the PIR is being prepared that there is an ongoing need for the policy, but that some aspect of it needs to be changed. This is likely to result in a new proposal for policy change, and a new Impact Analysis will usually be required. In such cases, the findings from the PIR will inform the decision about that Impact Analysis, and the PIR can be incorporated as part of the Impact Analysis. The resulting Impact Analysis can then be used to satisfy both the PIR and the Impact Analysis requirements.

The precise nature of each PIR will also depend on the individual problem and policies put in place to address that problem. The PIR may need to include other relevant information and discussion in addition to the above before ACE can assess the review as adequate.

Using consultants for PIRs

Agencies may wish to employ consultants to help prepare PIRs, particularly where the analysis of highly complex issues requires expertise not available within the agency. While this can improve the quality of the analysis in the PIR, it can also reduce the depth of policy expertise developed within the agency. The main objective of the PIR process is to assess the appropriateness, effectiveness and efficiency of the policy. Where this role is outsourced, there is a risk that the agency will not develop the depth of knowledge needed to give decision makers the best possible advice on any future changes to the policy.

The PIR remains the responsibility of the agency even if consultants are used. However, ACE can hold joint consultations with both parties to help ensure that the final PIR is adequate.

You are not required to structure the PIR in any particular way (for example, by using the criteria as headings). All that matters is that the information included in the review is sufficient to address each of the following questions.

Question 1: What problem was the policy meant to solve?

Simply and clearly explain the problem that the policy was intended to solve. Offer evidence about relevant market or regulatory failures before the introduction of the policy and describe which businesses, community organisations or individuals were being most affected. Establish the magnitude of the problem. Provide background information to the regulatory intervention, including a description of any relevant previous regulatory arrangements and any evidence that they were not already addressing the problem.

Questions on these topics to answer in the PIR could include the following.

Market failure

• Was there evidence that one or more firms were abusing their position of market power?
• Was there a lack of reliable and easily accessible information about products, services or risks?
• Were there any externalities?
• What evidence suggested that the market would not resolve the problem over time?

Regulatory failure

• What previous policy failed to address the problem?
• What caused the previous policy to fail?
• Is policy still the most appropriate solution, given a prior failure to address the problem?

Magnitude of the problem

• Was there a relatively high likelihood of a serious issue affecting a significant group in the economy or community?
• Was there a need to manage the risk of disaster? For example, would the costs of repair be so great that we would not be able to afford them?

This is not an exhaustive list of questions.

Question 2: Why was government action needed?

Establish the case for direct intervention by government at the time of the policy decision. This includes any ex ante evidence showing that policy would be likely to be effective in addressing the problem.

Explain the objectives, outcomes, goals or targets of the policy intervention.

Discuss any official announcements or media releases made by the relevant minister or the Prime Minister at the time of the policy decision.

Where relevant, discuss the set of objectives stated in the Act associated with the change.

Question 3: What policy options were considered?

Include information on other viable policy options that were discussed in consultation or considered by decision makers as part of the policy decision, together with any reasons that can be published as to why the other options were not preferred. This includes any non-regulatory approaches that might have been used to achieve the objectives of the policy.

Provide background information on the various options (for example, the policy may have been an election commitment, or an option may have already been ruled out by previous political commitments).

Discuss any options that were lobbied for or against by stakeholders.

Question 4: What were the impacts of the policy?

Discuss the implementation of the policy and any impacts on businesses, individuals and community organisations, specifying all significant competition or other impacts on them and highlighting any differences in how various stakeholders are affected.

When including information about the regulatory costs relating to the policy, those costs should be estimated in a manner consistent with the Regulatory Burden Measurement framework. See the Regulatory Burden Measurement framework guidance note for more information.

When reporting on the impacts of the policy, the baseline for comparisons should be what is most likely to have happened if the change had not been made. (This is why it is critical that performance indicators are identified during the implementation of a policy, allowing data on its efficiency and effectiveness to be collected from Day 1.)

If the policy was introduced in an area where policies previously did not exist, the ‘no regulation’ scenario is the obvious baseline. If the regulatory change replaced one policy with another, the situation under the earlier policy is probably the best baseline. Even if it is difficult to identify the correct theoretical baseline, or if the most appropriate comparisons are very difficult, you will still need to set some benchmark against which the impacts of the policy can be consistently measured. OIA can provide technical assistance and help you to explore other options.

As there are many factors that may affect the achievement of intended outcomes, and a good PIR will acknowledge the influence of these factors and clearly illustrate how the policy contributed to the measured outcomes. The use of tools such as ‘theory of change’ – see the Commonwealth Evaluation Policy and Toolkit - may support efforts to demonstrate ‘causal inference’.

The level of analysis in the PIR should be commensurate and proportionate with the impacts of the measure: the greater the impacts, the more in-depth your analysis should be. More detailed analysis will usually need to be supported by stronger evidence.

Cover a period from the introduction of the policy until the present or the time when the policy lapsed (where the policy has ceased to operate).

Analyse both the costs and the benefits of the policy. This analysis should:

• focus on outcomes since implementation;
• assess the costs and benefits based on data gathered from consultation or from independent sources, such as the Australian Bureau of Statistics;
• assess the impacts on businesses, community organisations or individuals, including distributional issues such as the impact on small business;
• assess the net impact of the policy on the community as a whole, taking into account all costs and benefits;
• quantify (for example, through consultation, surveys, or both) the effect of the policy on compliance costs for businesses, community organisations or individuals;
• recognise the cumulative burden of the policy on individuals, businesses and community organisations;
• note any significant changes to the level of competition since the policy was introduced (for example, new suppliers entering the market, or a number of small businesses leaving the sector).

The analysis should be a balanced presentation of the costs and benefits.

Even if a full quantitative cost–benefit analysis is not possible, your PIR should still quantify those costs and benefits that can be measured and analyse the material impacts of the policy on the main stakeholders. You need to also quantify the regulatory costs of the policy to businesses, community organisations or individuals, consistent with the Regulatory Burden Measurement framework.

See the Cost–benefit analysis guidance note for more information on conducting a cost–benefit analysis.

Government will almost always be a stakeholder in any policy. Therefore, you should also consider any enforcement issues highlighted by the regulator during implementation or identified by other stakeholders during consultations. Describe any unintended consequences of the policy. Say why they occurred, what impacts they had, and whether they enhanced or reduced the policy’s effectiveness. In some cases, it may be necessary or appropriate to suggest ways to change the regulation, or to put forward other non-regulatory methods to address problems that have arisen.

There may be instances where relevant information is no longer available because of the delay between the decision to introduce a policy and the preparation of the PIR, or because the original decision maker might no longer be the current decision maker. Staff turnover may also mean that corporate memory about a decision is limited. In these circumstances, and where all avenues to obtain this information have been exhausted, make that transparent in the PIR.

Question 5: Which stakeholders have been consulted?

Your PIR process should incorporate consultation with relevant stakeholders on the impacts of the policy, including competition impacts and the regulatory costs. This consultation should be in line with that detailed in the Australian Government Guide to Policy Impact Analysis and the Best practice consultation guidance note. As with the analysis, the level of consultation should be commensurate with the significance of the measure under review.

It may be more efficient for you to gather the necessary data for PIR analysis as part of an ongoing consultation starting during the implementation phase.

The PIR needs to describe how consultation was conducted (including when it was undertaken, the timeframes and the methods). You should articulate the views of those consulted, highlighting any substantial differences between the views or experiences of different stakeholders.

Question 6: Has the regulation delivered a net benefit?

Provide an assessment of whether the policy has delivered a net benefit. You should also make an assessment of the ongoing appropriateness, effectiveness and efficiency of the policy in addressing the problem and achieving the policy’s stated objectives. This statement needs to be supported by a balanced assessment of the evidence presented elsewhere in the PIR.

In some cases, isolating the impact of a single policy change will not be possible, given other influences on outcomes. In such cases, it is reasonable, and possibly desirable, to conclude and state that it is not possible to draw definitive conclusions.

If the policy change has restricted competition, you need to assess whether or not it has generated a net benefit to the community as a whole. You should also consider whether or not the restrictions on competition have been essential to achieving the Government’s objectives. The PIR should also include these findings on competition impacts as part of its conclusion. It is essential that your conclusions about the impacts and effectiveness of the regulation align with the evidence presented in your PIR.

Question 7: How was the policy implemented and evaluated?

Your PIR should record the implementation plan and note any lessons learned from the implementation of the policy. You should also summarise any review processes used to monitor the performance of the policy, other than the PIR.

How extensive the analysis should be

The appropriate level of analysis varies from review to review. In general, the greater the impacts of the policy, the more extensive your analysis should be.

The level of quantification in the analysis will depend on data availability. Effective consultation with stakeholders soon after implementation should help you to obtain relevant data.

As a guide, you should consider any relevant productivity impacts; benefits to businesses, consumers, the environment, governments and other affected groups; regulatory costs for businesses, community organisations and individuals; and any other costs to consumers, the community or the environment.

Discuss the distribution of the costs and benefits, including jurisdictional differences where they are relevant.

Back to top

OIA and ACE assistance

OIA reports on compliance with the Government’s Impact Analysis requirements, including PIRs, on the Department of the Prime Minister and Cabinet’s website at Published Impact Analyses. For further information on compliance reporting, contact OIA on (02) 6271 6270 or at helpdesk-OIA@pmc.gov.au.

For broader advice on the management of ‘ex-post’ evaluations in the Australian Public Service, please refer to the Commonwealth Evaluation Policy and Toolkit, and contact the ACE on Evaluation@Treasury.gov.au.

Back to top

Appendix 1: Applying the PIR requirements

Example 1: Policy is still in force and the agency is not developing a proposal to remove or change the policy

An agency does not prepare Impact Analysis to inform a government final decision on a policy proposal. As a result, OIA advises the agency that it must complete a PIR for the proposal within two years of the decision being implemented.

For example, the Government has made a regulation to give effect to the policy proposal and, as the proposal has been implemented, the regulation is in force. The agency is not developing a proposal to remove or change the policy, and the Government has not made a decision that the policy no longer reflects Government policy.

PIR requirements

The only way the agency can meet the PIR requirements is to complete the PIR within two years of the decision being implemented.

Example 2: A previous policy that requires a PIR is now proposed for removal

An agency does not prepare Impact Analysis to inform a government final decision on a policy proposal. OIA determines the agency must complete a PIR within two years of the decision being implemented. The Government has made a regulation to give effect to the proposal and, as the proposal has been implemented, the policy is in force. The agency is now developing a proposal to remove the regulation.

Impact Analysis and PIR requirements

The agency could meet the PIR requirements in one of two ways:

• The agency could prepare Impact Analysis to inform the decision to remove the policy and then a PIR to review the initial policy decision. The PIR needs to be completed within two years of the initial policy being implemented.
• The agency could prepare a combined Impact Analysis/PIR to inform the decision to remove the policy and to review the initial policy decision. If the agency chooses to undertake a combined Impact Analysis/PIR, the combined Impact Analysis/PIR also needs to consider the impacts of the policy while it was in force.

The relevant minister cannot seek the Prime Minister’s agreement to set aside the requirement to undertake the PIR on the basis that the approach is no longer government policy because the policy is in operation and the Government has not taken action to change the policy (for example, by making a Cabinet decision, Ministerial decision/determination, or Federal Executive Council decision and by making substantial progress to implement the decision, such as, through preparation and passage of legislation).

Example 3: Regulation has been removed and it no longer reflects government policy

An agency does not prepare Impact Analysis to inform a government final decision on a policy proposal. OIA advises the agency that it must complete a PIR within two years of the decision being implemented. The government made regulation to give effect to a policy proposal. However, after a short time the policy was removed as it was no longer Government policy. Impact Analysis was prepared to inform the Government’s decision to remove the policy.

As a result, the policy was implemented for a relatively short period of time and is no longer in force. The policy no longer reflects government policy.

PIR requirements

The agency can meet the PIR requirements in one of two ways:

• The agency can complete the PIR within two years of the decision being implemented.
• The relevant Minister can seek the Prime Minister’s agreement to set aside the requirement to undertake the PIR on the basis that the policy is no longer government policy and is no longer in effect. If the Prime Minister does not agree to set aside the requirement, the agency must complete the PIR within two years of the decision being implemented.

Example 4: Policy has been in place for a temporary period only

An agency develops a policy proposal to address a time-sensitive policy issue. The agency does not prepare Impact Analysis to inform a government final decision on the policy proposal. OIA advises the agency that it must complete a PIR within two years of the decision being implemented.

The government makes a regulation to give effect to the policy proposal, but it is an interim measure and expires after 12 months.

As a result, the policy was implemented for only 12 months and is no longer in effect. The policy may still be a Government policy and may be used to address the same time sensitive issue if it arises again.

Impact Analysis and PIR requirements

As the policy expired automatically after 12 months Impact Analysis is not required to remove the policy. The agency can meet the PIR requirements in one of two ways:

• The agency may prepare a PIR to review the initial policy decision to introduce the policy. The PIR must be completed within two years of the policy being implemented.
• The relevant Minister can seek the PM’s agreement to set aside the requirement to undertake the PIR on the basis that the policy is no longer in force. If the Prime Minister does not agree to set aside the requirement, the agency must complete the PIR within two years of the decision being implemented.

Example 5: Policy has not been implemented and the proposal no longer reflects government policy

An agency does not prepare Impact Analysis to inform a government final decision on a policy proposal. OIA advises the agency that it must complete a PIR within two years of the decision being implemented.

The government has not made a policy to give effect to the decision and it has not yet been implemented. The government has indicated that it will not be making the policy.

PIR requirements

The agency would only need to do the PIR after the policy has been implemented. In the scenario above, the agency can write to ACE to advise that the proposal was never implemented, and no longer reflects government policy. ACE can agree that a PIR no longer needs to be completed for the proposal and will publicly report this advice.

1. The PIR should be prepared by the agency responsible for bringing the original policy proposal to the decision maker. If machinery-of-government changes result in functions moving, the responsibility for preparing a PIR moves to the new agency together with the function.Return to footnote 1 ↩
2. The Australian Government Guide to Policy Impact Analysis defines a minor change as one that does not substantially alter existing arrangements and a machinery change as a consequential change in policy required as a result of a substantive regulatory decision.Return to footnote 2 ↩
3. Unless subject to one of the triggers requiring the PIR to be completed within two years.Return to footnote 3 ↩
4. The Australian Government has endorsed the Web Content Accessibility Guidelines (WCAG) version 2.0 for all government websites.Return to footnote 4 ↩

Back to top