Reports and Publications

• Evaluation Reports
• Evaluation report No. 10/2011
  • Contents
  • Letter to the Minister
  • Executive Summary
  • Chief of Defence Force Response
  • Section 1 Introduction
  • Section 2 Policy and guidance
  • Section 3 Organisation and process
  • Section 4 Future developments
  • Annex A Examples of lessons learned

 

Management of Lessons Learned by the NZDF

Contents | Previous | Next

Executive Summary

Introduction

1. This review examined how effectively lessons learned in the course of NZDF activities are being collated, analysed, resolved and shared.
2. Field work for this review was completed in February 2011.

Background

3. A lessons learned process essentially involves the collection, analysis, decision making, actioning and sharing of information relating to an activity undertaken. Almost all activities can have this process applied. The collection of data may be formal or informal, and the extent to which electronic systems are used will vary depending on the requirements of the process involved.
4. In December 2005, CDF Directive 31/2005 Introduction of a Global Lessons Learned System to the NZDF established the Electronic Activity Reporting and Lessons Learned System (EARLLS) as the NZDF global lessons learned system covering the full range of NZDF activities. ‘Activities’ were defined as operations, exercises and other activities (war games, trials, UN missions, Boards/Courts of Inquiry, courses, visits and conferences).
5. There are within the NZDF many activities, particularly at the functional level, that include a quality review and validation process and facilitate issue resolution, but do not utilise EARLLS. The most notable of these relate to logistics support, individual training, health and safety, security, and the NZDF project management framework. Boards/Courts of Inquiry activities also involve processes from which lessons are learned.
6. Much of our review has covered the utility of the global EARLLS and the operationally focussed lessons learned processes it facilitates. We also looked at whether it is feasible to incorporate other processes into EARLLS, or have formal links with EARLLS developed, to maximise the sharing of information.
7. Within this context of lessons learned, our expectations were that:
   1. there is clearly defined and effective strategic policy and guidance on identification, collection, analysis and management of lessons learned;
   2. there are effective processes and systems for gathering, analysing and sharing individual knowledge and organisational experience for future use;
   3. data gathered is being analysed and validated to form lessons learned; and
   4. lessons learned lead to changes that result in improved military or business practice.

Overview

Policy and guidance

8. Effective NZDF and Service policy on post activity reporting for lessons learned purposes is in place. This policy predominantly directs the use of EARLLS for operational activities, which covers overseas missions, major exercises, and other readiness or collective training activities.
9. For visits, conferences and courses, there is general acceptance of post activity reporting but only in the case of the Army is there a formal instruction to input details of these activities into EARLLS. We have reported previously on the variability of post activity reporting for these types of activities and recommended that the NZDF ensures reports are completed.
10. The CDF Directive is unclear, in that while it refers to the full range of NZDF activities, it also specifically identifies those that are included. This has caused some uncertainty over the intent of the policy on lessons learned and may have restricted the extent to which EARLLS has been utilised elsewhere.
11. Service specific policy and guidance is in place for lessons learned in various functional areas.
12. There is no policy or guidance on the management of lessons learned in HQ NZDF through its strategic and corporate activities. AC Capability is looking at the potential utility of EARLLS within the NZDF project management framework.

Organisation and processes

13. The NZDF has effective processes and systems in place, through EARLLS, for gathering, analysing and sharing individual knowledge and organisational experience for future use in relation to overseas missions, major exercises, and other readiness or collective training activities.
14. Extracting quantitative data from the system for reporting purposes is relatively easy but its quality is dependent on the accurate classification of activity reports and accompanying observations entered into the system. Approval or validation procedures are in place to ensure the quality of data entered into the system.
15. While the various areas that monitor post activity reports analyse trends in observations to identify and validate issues that arise, there is little performance measurement undertaken to determine and report the ongoing effectiveness of EARLLS and the lesson learned process it facilitates.
16. Other main lessons learned processes appear to be effective for the functional areas and activities they support. However, there is little sharing of information from these across the NZDF.

Future developments

17. Although HQ JFNZ has promoted the wider use of EARLLS, there remains a view that the system is predominantly for HQ JFNZ and Service operational activities. Some see difficulty in resourcing and using what they regard as a complicated and cumbersome system that has limited direction or support available.
18. Capability Branch recently looked into the adoption of EARLLS for the capability projects it manages. The review considered the responsibilities of Assistant Chief Capability, as the sponsor of EARLLS, for broadening the use of EARLLS by other HQ NZDF Branches and the NZDF generally.
19. Capability Branch identified that to achieve the use of EARLLS at the corporate/strategic level the NZDF needs policy that clearly outlines corporate intent and drives a lessons learned culture and objectives. Simply making the system available and expecting it to operate effectively across the organisation was not sufficient. There is a need for tailored processes that fit the requirements of a particular area and the communities of interest that are going to make use of it. This requires commitment, education and resourcing.
20. The successful introduction of EARLLS to facilitate lessons learned within the NZDF project management framework, and in particular the process of developing policy and procedures for this purpose, could provide an example for other areas within the organisation. As such, this could provide a starting point for the step by step expansion of EARLLS and its broader use as a NZDF lessons learned application. It could also provide the basis of a lessons learned centre of excellence within HQ NZDF for providing enhanced support to future EARLLS users. Its potential use to provide an overview of all Courts/Boards of Inquiry recommendations is a particular case in point.
21. Overcoming some of the current restrictions in EARLLS and enhancing its usability generally requires technical improvements. These include a search engine facility to undertake word based searches to seek relevant reports and observations. Currently, searching for information requires standard metadata topics (classifications). Fully interfacing EARLLS with the NZDF’s proposed new document management system will simplify the exchange of information between the two applications.

Conclusion

22. The NZDF has a variety of processes and systems through which lessons are learned. Of these only EARLLS provides a dedicated lessons learned management system.
23. Under the auspices of HQ JFNZ, EARLLS supports an effective lessons learned process used predominantly for missions, exercises and readiness or collective training activities. Other main lessons learned processes that do not utilise EARLLS appear to be effective for the functional areas and activities they support but there is little sharing of information from these across the NZDF.
24. The Army is further developing its Centre for Army Lessons to promote awareness, and achieve better coordination, of its various lessons learned processes. The platform based nature of the Navy and Air Force means that there is not the same need in those Services for a central coordination entity.
25. While benefits are achievable from establishing a single global lessons learned framework, the NZDF should seek a balanced organisational approach to lessons learned. Where appropriate, formal and informal processes that are coordinated and cross-linked should allow sharing of information across the organisation. The challenge, therefore, is in deciding what information should be shared, and where responsibilities for the development, management and support of lessons learned should be best located.
26. Capability Branch is assessing the potential utility and functionality of EARLLS within the NZDF project management framework.
27. Capability Branch has also considered whether EARLLS should be further promoted as a global lessons learned system. Clearly some other areas and activities could benefit from utilising the system and sharing information. As well as the development of policy that clearly outlines the corporate intent and drives the lessons learned culture and objectives, additional resourcing would be required to support the effective management of an expanded lessons learned process.
28. There is a need for a single senior sponsor or champion for lessons learned to coordinate future policy and development, who could also promote the effective and useful expansion of EARLLS across the NZDF.

Recommendations

29. It is recommended that the NZDF:
    1. appoints a sponsor or champion to coordinate future policy and development on lessons learned and promote the wider use of EARLLS across the NZDF; and
    2. considers the use of EARLLS for registering and monitoring the implementation of all Boards/Courts of Inquiry recommendations.
30. It is recommended that HQ JFNZ:
    1. develops performance measures against which the ongoing effectiveness of EARLLS can be gauged.

Top | Previous | Next