Auditing a System Implementation: Potential Questions, Risks, and Audit Areas of Focus

With constant improvements and technological advances, organizations regularly add new applications, consolidate functions, and move systems to the cloud. My organization, Northwest Natural, went through a major implementation not too long ago, and we had to decide how internal audit would be involved. At the time, we looked for resources to help guide us on how and when to conduct an audit, but ultimately we had to make decisions that worked for our company on our specific project. 

Many of you will be in the same situation, so I want to share some of our experiences, key considerations, and reflections to enable other auditors to leverage the work we did as well as some of our learnings along the way. 

Considerations When Planning Your Audit

At the beginning of the implementation, there were several key decisions we had to make to set the stage for our involvement. 

First, we had to understand the scale of the implementation. The scale and duration of the project would drive the internal audit resource requirements and could have an impact on our broader Annual Audit Plan. In our case, our company was deploying a large ERP system implementation that included moving from on-premise to the cloud, expanding use to include new modules and functionality, and revising longstanding business processes. 

Next, we needed to decide on the role internal audit would play on this ERP implementation. Our role could be to act as consultants at key milestones, perform a pre-implementation review, perform a post-implementation review, and/or be engaged throughout the entire project. Remember, there is no single “right” answer to these questions, and each situation should be evaluated to find the best options. If you are unsure which route to take, refer to the figure below where we outlined the possible approaches our internal audit team evaluated.

After considering all the alternatives, we met with management to discuss which route would benefit the company best. Together we decided that internal audit should be involved throughout the implementation to provide timely insights and feedback at key milestones. With this approach, if we found a missing or poorly designed control, the business team could correct it early without delaying the project and/or if we identified a gap in project management we could proactively elevate the issue in time for the company to invoke and benefit from a change. 

Our approach was balanced. We were involved consistently, but not to the point that we would require dedicated internal audit resources to be on the implementation team, and not so deeply that we blurred the line with independence. We found this to be the most optimized model for this project at our company. 

Example Areas for Internal Audit to Focus

After agreeing to an audit approach up front, we engaged on the project’s planning phase.  During planning, we selected key milestones for this implementation that we felt were important for us to be involved in: 

1. Project Initiation and Governance
2. Business Process Design
3. Data Conversion and Migration
4. User Acceptance Testing
5. End User Training and Change Management
6. Report Design
7. Application Security Design
8. Operational Readiness and Go-Live

While the milestones may differ across projects, these are fairly common. For each selected milestone, we identified questions to consider, risks to address, and areas where we should focus our attention. These examples are not meant to be comprehensive, but are instead intended to provide ideas to get you started. 

1. Project Initiation and Governance 

At the beginning of the project, the implementation team should establish a governance framework, which outlines the structure for overseeing project execution and the structure within which project decisions will be made. The implementation team also drafts a project plan, which defines project objectives, specifies tasks with ownership, describes how goals will be achieved, and identifies what resources will be needed, associated budgets, and timelines for completion.  

Example Internal Audit Considerations

• What structure is being established by management to manage and oversee the project? 
• Will the project have a steering committee?
• What decisions will need to go to the steering committee for approval? 
• What other committees or teams will be established to manage the project?
• Will scope changes go through a committee?   
• What deliverables & milestones are addressed in the project plan? 
• What will the schedule look like? 
• How is the project being resourced to ensure appropriate content knowledge and expertise?
• How will management routinely assess project status and monitor project-related risks?
• What project management method will be used – agile or waterfall?
• How do you plan to optimize internal audit’s role throughout the project considering the planned method (agile or waterfall)?

Example Risks

• Governance structure fails to enable timely decision-making, issue identification, and issue resolution at the right levels in the organization.
• Ineffective project and resource planning leads to delays in execution and missed dependencies.
• Lack of ownership and accountability of critical program components.
• Inadequate status review process (e.g. non-timely, not frequent enough, inaccurate).

Possible Audit Focus Areas     

• Effective Project Management (PM) structure with accountability, roles and responsibilities, and escalation paths well defined.
• Overall program governance structure with critical roles and expectations for the business, IT, and vendors (including system implementor).
• PM and program governance structure, which effectively and efficiently identifies and resolves risks.
• Comprehensive project plan that defines key deliverables across all project phases including the initiation phase (e.g. business case, charter, project management approach, quality management plan, etc.).
• Project plan that has been reviewed by program stakeholders for completeness to ensure it appropriately represents all parties and the true critical path.

2. Business Process Design

Business process design models future-state processes including controls based on business requirements. As the design is completed, the teams will start configuring or developing the application depending on the type of application. While the processes are being implemented, the teams should keep controls in mind. Often the controls, both operational and financial, need to be considered during the configuration/development. 

Example Internal Audit Considerations

• What business processes are in scope?
• How did management identify, confirm, and validate the business requirements for the actual design-build? 
• Do the business process designs, requirements, and initial controls seem appropriate? 
• Do the controls follow internal expectations, standards or frameworks (e.g., expected Sarbanes Oxley (SOX) IT general controls)?
• How are documented business requirements being tracked throughout the project? 

Example Risks

• Requirements emerge late, resulting in rework, delays, and missed expectations.
• Business requirements and system design are not aligned.
• Business users are unable to envision/understand/leverage system capabilities.
• Non-integrated data architecture leads to process inefficiencies.
• Controls are missing, poorly designed and/or fail to meet minimum control standards.

Possible Internal Audit Focus Area

• Design details outline the process’s future state, organization, application, and data.
• Those responsible for design have appropriate content knowledge and skills. 
• Business partners take ownership for detailed business requirements review and sign-off.
• Critical gaps were identified and addressed immediately throughout.
• Manual workarounds are thoroughly evaluated and managed throughout.
• Controls are well documented.

3. Data Conversion and Migration

Data conversion is data transformation from one format to another (i.e., extract, transform, and load data). Data migration is moving data from a source system to a destination system, which will almost always include data conversion.

Example Internal Audit Considerations

• Is existing data being loaded into the new system or will it require the data to be imported/added manually?
• Does existing data need to be updated and modified before being uploaded to the new system?
• Is there a plan to confirm the data load is complete and accurate?
• Does the data include sensitive or confidential information and how is it being controlled/managed?

Example Risks

• Lacking business engagement results in incorrect mapping of data.
• Poor data quality results in delays, rework, or system effectiveness.
• Data ownership and process for authorizing and approving master data are not defined.
• Data quality is not monitored, impacting system effectiveness.
• Sensitive data, such as personally identifiable information, is exposed.

Possible Internal Audit Focus Area

• The data conversion strategy is formally reviewed by the business owners of the data, the system integrator, and IT.
• Data conversion design has been documented and includes detailed data mapping rules and stakeholders’ sign-off on attributes.
• Data cleansing plan that outlines responsibility for validation.
• Defined data conversion validation procedures for each data object.

4. Testing

The testing strategy outlines the complete approach to validating the business process and system before implementation. There are usually four recognized levels of testing that need to be completed before a system is ready to be used. These include unit, integration, system, and acceptance testing. Regression testing is also performed to ensure any code fixes in response to test exceptions (or otherwise) are functioning as expected. In our case, we focused our audit objectives on user acceptance testing, but internal audit can be involved in other aspects of testing depending on the specific audit scope and objectives that you set out.

Example Internal Audit Considerations

• Is there the ability to review test scripts (planned testing procedures) and provide feedback before actual test execution?
• Does the scope of the test plans match the project’s scope/business requirements?
• Is the project team maintaining adequate evidence of test results in an effort to support reperformance? 
• How are test exceptions being tracked and dispositioned?
• Does the test approach encompass cross-functional business process touchpoints and handovers?

Example Risks

• Inability to perform end-to-end testing
• Inability to perform testing in a manner that includes reporting
• Inadequate testing environment (i.e.: test data that does not mirror production data)
• Showstopper test exceptions go unresolved prior to Go-Live
• Inadequate testing scope and resources, resulting in unvalidated requirements

Possible Internal Audit Focus Area

• Test plans with comprehensive coverage of end-to-end processes, a full range of business scenarios, and real-life variations.
• Effective test metrics framework and tracking process allow corrective actions and re-planning.
• Testing entry and exit criteria are clearly defined, satisfied, and approved.
• Effective defect management and prioritization process.
• Regression testing is performed consistently to assess the impact of fixes for failed tests on cases already passed.

5. End User Training and Change Management

Organizational change management guides the implementation of business process change, encompassing the company’s culture, the underlying technologies or infrastructure the company uses to operate, and its internal processes. Typically, three major phases are preparation, implementation, and follow-through. End-user training supports successful implementation.

Example Internal Audit Considerations

• Is there the ability to analyze the AS IS vs. TO BE business process design in an effort to identify changes?
• Was a comprehensive change impact assessment performed?
• Which areas are most impacted?
• What training plans and job aids are in place upon Go-Live for the impacted stakeholders?
• Were corporate policies assessed in order to determine alignment and/or if policy changes are required?
• Are desktop procedures being revised in advance of the implementation?
• Are accommodations available for remote and asynchronous workers?
• What is the post-Go-Live support model and how is it being resourced?
• Is there a plan for post-implementation user assessments? (e.g. post-Go-Live surveys or otherwise)

Example Risks

• Users and business process owners are unprepared for the implementation.
• Lack of focus on transition change management leads to ineffective and inefficient processes and disruptions after Go-Live, regardless of the quality of the implemented system.
• Inability of the business to sustain the newly implemented system and processes after the project team exits the post-implementation stabilization period

Possible Internal Audit Focus Area

• The organizational impact of systems and process changes has been determined, assessed, and planned.
• Analysis of stakeholders has been performed, and change enablement strategies are developed to ensure changes are effectively implemented.
• Change the enablement plan with buy-in from stakeholders to support.
• Training and desktop procedures and process documentation have been developed for the user community.
• Comprehensive post-Go-Live live support model has been defined and communicated.

6. Report Design

Report design models future-state reports based on well-defined business and infrastructural (IT) requirements. The system must be configured with the right data elements to produce the reports. In many cases, the data may need to be transformed or enhanced for useful reporting. 

Example Internal Audit Considerations

• Has the business created an inventory of key reports?
• Were detailed reporting requirements documented and shared with relevant team members? 
• Have business process owners reviewed and validated the key reports inventory and detailed reporting requirements?
• Were reports adequately encompassed in user acceptance testing?

Example Risks

• Reporting requirements are not captured, and reports are not validated during testing, disrupting the business.
• Data integrity issues with reporting
• Delayed delivery of reports impacting ability to perform business processes
• System performance issues as a result of sub-optimal reporting solutions and related infrastructure

Possible Internal Audit Focus

• Stakeholders have verified a complete inventory of reports.
• Enterprise has reviewed and validated overall architecture, including tools or technologies for reporting to ensure it meets future needs.
• The new system delivers critical analytics capabilities the business requires.
• User acceptance testing encompassed reports and included data integrity validation
• Consideration of formal governance program to sustain the business intelligence environment.

7. Application Security Design

Security design outlines the features and related processes being implemented to minimize human (manual) and software vulnerabilities.  

Example Internal Audit Considerations

• What security features have been enabled within the software?
• Does the design align with corporate security policies?
• Were IT security teams involved in the design and configuration?
• What is the process for making security changes upon Go-Live?

Example Risks

• Project planning fails to incorporate security and internal control project activities resulting in costly post-Go-Live remediation
• Expensive manual workarounds are required to compensate for the failure of the new system to deliver security and internal controls that meet audit and compliance requirements.

Possible Internal Audit Focus Area

• Security design aligned with best practices
• Security and controls are integrated into architecture, design, configuration, and testing, with interdependencies considered and managed
• Security and internal control design provide a high degree of automation
• User access lists (assignment of security roles to users) reviewed/approved before going live
• Security design aligns with expected segregation duties of requirements
• Security design addresses access to key databases, and the database user list is reviewed/approved before going live

8. Operational Readiness and Go-Live

IT operational readiness is a structured process for ensuring the IT organization acquires the tools, skills, and documentation to operate and maintain a newly completed system implementation project. Once completed and validated, the steering committee should approve the final Go-Live decision. Communication should be sent to the stakeholders with explicit expectations and dates for transition.

Example Internal Audit Considerations

• Is there defined go/no go criteria which includes expected aspects?
• Which individuals need to validate the final product for approval?
• Is there a communication plan for stakeholders to announce the Go-Live and transition?
• Is HyperCare or other priority support available post-Go-Live?
• Is there defined project exit criteria which considers requirements for measuring stabilization level prior to handover from project team to sustaining IT support organization?

Example Risks

• Failure to plan for adequate project team support during post-Go-Live stabilization period
• IT is not prepared to assume ownership and maintenance responsibilities from the project team. 
• Failure of IT to adequately support the new system results in a lack of confidence and failed adoption of the new system.
• Failure to monitor service provider(s) performance as a result of unclear/inadequate expectations and/or lacking oversight processes.

Possible Internal Audit Focus Area

• Strategy for post-Go-Live IT support is developed early during the project lifecycle and not as an afterthought.
• Clear project team exit criteria, including:
  • Well-defined roles/ responsibilities for post-stabilization business and IT owners
  • Minimum requirements (documentation, training, etc.) for ownership transition from the project team to future IT and biz owners
• Service level arrangements (SLAs) are in place with service providers (where applicable) before Go-Live.
• Plan to incorporate the system under existing operational change control processes post-Go-Live (i.e., code cannot be modified without following productional change control procedures, etc.)
• Approach for conducting a post-implementation review (i.e., analysis of ticket volume following Go-Live, etc.)

Sample Project Audit Objectives

After formulating the considerations, risks, and focus areas, we built audit objectives to ensure we captured all the information needed. The following table includes a sample of those objectives for the various project phases. As with the previous section, the information below is not meant to be comprehensive, but to provide ideas to get you started across a few sample focus areas. 

Business Process (BP) Design

Potential Risk Considerations: 

• Business requirements emerge late, resulting in rework
• Misalignment between business requirements vs. system design
• Failure of the design to achieve minimally expected internal control standards
• Overcomplicated or non-optimized business process design resulting from:
  • Overreliance on manual vs. automated controls and/or manual process workarounds
  • Failure to consider best practices when designing business processes
  • Unwillingness to change business processes to simplify and/or fit the system
  • Project team who does not know or understand full system capability/possibilities
• Business process design that fails to address downstream customer and/or partner needs
• Non-integrated data architecture leads to process inefficiencies, errors, and potential security exposures

Project Governance and Execution

Potential Risk Considerations:

• Failure to enable timely decision-making, issue identification, and issue resolution at the right levels in the organization
• Inability to deliver the project on time and on budget
• Poor project management, resource planning, technology planning, and oversight
• Failure to implement a technical solution that addresses business needs/requirements
• Post-implementation challenges, including business disruption, resulting from lacking project oversight, change management/user preparedness, testing, data quality, etc.
• Implementation of a solution that is unsustainable over the long-term solution

Operational Readiness Reviews

Potential Risk Considerations: 

• Inadequate testing plan, testing scope and/or test scripts
• Inadequate and/or unknowledgeable testing resources
• Incorrect mapping of data and/or poor data quality
• Process and ownership for validating converted/migrated data not well defined
• Data quality is not adequately validated pre-implementation
• Failure of the new system to deliver security that meets audit, segregation of duties, and compliance requirements
• Inadequate training/defined procedures preparedness upon Go-Live resulting in user unpreparedness, business disruption, and/or inefficiencies
• Post-implementation issues, rework, business disruption, and/or inefficiencies
• Training/procedural content that fails to provide a long-term, sustainable post-Go-Live solution

Reflections

As you start planning internal audit’s role in any system implementation, my advice is to have open communication about the role your internal audit team will play, understand management’s expectations, and approach the project with an agile mindset. Keeping these goals in mind will drastically improve the outcome of your project.

Even with the best planning and strong teams, system implementations, especially large-scale systems, are notoriously stressful. Before we started, we talked to other companies as well to learn from their experience. Most audit teams we spoke to were involved in pre-implementation reviews in a consulting capacity. Others decided to only audit the project post-implementation and did not engage in the governance aspects at the beginning and end of the project.    

Some of our reflections following the conclusion of this system implementation include:

• We needed to be flexible and adopt an agile mentality. Large implementations are multi-year engagements, and our involvement was not on a strict schedule. We found it best to assign our most experienced auditors, who could be flexible and practical when working toward a moving target. We also found it important that our broader Audit Plan was built with flexibility in mind, and took care that key stakeholders such as the Audit Committee understood potential implications on other operational audits due to the evolving nature of the broader system implementation.
• We found it important to determine internal audit’s and external audit’s roles where relevant to minimize redundancy. In our case, we found that our external auditors could leverage some of internal audit’s work as well which was a huge positive!
• Even though we performed this work in a consulting-type capacity, we formalized internal audit’s objectives in the form of a scope memo that we shared with key stakeholders including our system integrator and our external auditors. The formal scope memo contributed meaningfully to build our organization’s credibility and formalize alignment on our specific audit plans.
• Although internal audit functions may not choose to issue formal audit reports when performing work in a consulting-type capacity, we opted to publish formal audit reports at key milestones. We found this to be useful because it enabled external auditors to leverage our work, documented our recommendations real-time so that they didn’t get lost in the shuffle, and provided us with a record of those recommendations that we could reference if needed as the project evolved. A streamlined memo format for our audit reports helped to keep the reporting process timely.

Internal audit plays an important role in system implementations. These are high-risk projects with long timelines that are prone to complications. We learned that the most critical factors to our success were the early alignment effort with management on our role and the flexibility we adopted during the life of the implementation. After discussing the project and our approach with our management teams, our chosen model hit the sweet spot. We had management support and upfront planning, and we were able to adjust as the project evolved. 

I hope that by sharing our experience, your project goes smoothly as well.