Organizations that are subject to Sarbanes-Oxley (SOX) compliance use various reports from the source systems to perform the controls defined by management. Essential to the control design is the completeness and accuracy of the report(s) being used to execute the control. Ensuring completeness entails gaining comfort that a full population of the relevant transactions are captured and presented in the report. Ensuring each report is accurate involves validating that the integrity of data compiled by the report is maintained throughout the report creation process.
The validation process is a bit of a circular exercise in that often you are querying data directly from the database or using other reports to verify its completeness and accuracy (C&A). How the C&A is tested for each report is typically reviewed by both internal and external auditors as it forms the foundation for the reliance on the report in the execution of the control.
The PCAOB in Auditing Standard 15 provides this guidance:
"10. When using information produced by the company as audit evidence, the auditor should evaluate whether the information is sufficient and appropriate for purposes of the audit by performing
procedures to: Test the accuracy and completeness of the information, or test the controls over the accuracy and completeness of that information;"
Two approaches an organization can apply:
1. Monitor the accuracy and completeness through changes to the report configuration
2. Monitor the accuracy and completeness by testing the change management process.
Both approaches require that an initial understanding of the report configuration be gained and tested. Tracing transactions from data entry to the report results will test for completeness and accuracy. Reperforming any calculations used to create the report also tests for accuracy. Once a baseline understanding of the report is obtained, one of the two approaches mentioned above are needed to maintain ongoing assurance over completeness and accuracy.
The first approach – to test the accuracy and completeness of the information – is a precise approach. This would require management to identify the reports that are 'in-scope' and monitor any potential changes to such reports. If any component of the report were to change, the testing of the C&A of the report would need to be re-evaluated. The benefit to this approach is that all fields, configurations & etc. relevant to the creation of the report are monitored for changes whether intended or not. This allows an organization to precisely determine whether changes to a report impact C&A.
The second approach – to test the controls over the accuracy and completeness of the information – is a much broader approach that would rely on the testing of the change management process. It is also the most commonly used by external auditors. To the extent that the change management process was reliable, the assumption is any in scope report would have been subject to the change management process and had its C&A re-validated. Organizations should ensure themselves that key reports are in- fact going through the change management process. One challenge with the approach is that there can be unintended false-positives from re-compiling of objects causing a lot of unnecessary audit work. Additionally, monitoring of the object changes (one component that needs to be monitored in our Oracle E-Business Suite example below), often done through the DBA_OBJECTS table has a lot of other types of objects contained in the same table, causing the need to justify why certain objects are not relevant to the scope of the audit.
Many organizations that are subject to SOX compliance have struggled with how to put together a complete program to meet these requirements. As such, the external audit firms have developed a testing approach that is much broader than would be otherwise warranted. This includes testing the change management process as a whole to get comfort over this issue rather than testing the specific reports that have been identified by management. Most external audit firms have taken the second approach.
In this white paper, we will define a more precise approach that management should take to increase its confidence level related to the completeness and accuracy of the data in the reports. The approach should also decrease the level of effort that external auditors need to perform in their audit procedures.
We believe management should take the first approach. That is, focus on the specific reports that are being relied upon and minimize the scope of testing to just the in-scope reports. We are not suggesting that testing the overall change management process isn't necessary, but are suggesting that such testing should NOT be necessary to provide reasonable assurance over the C&A of in-scope reports.
Management needs to develop a process that includes:
1. Identify a complete population of the in-scope reports
2. Identify which ways a report can be changed
3. Pro-actively monitor the changes to the components that can change the report
4. Re-perform completeness and accuracy testing for those reports that have been changed
Much of our consulting work is done in the Oracle E-Business Suite arena. As such, we are going to dive further into this topic with specific examples of how this would be executed for organizations using Oracle E-Business Suite (EBS):
Step 1: Identify a complete population of the in-scope reports
Depending on the level of documentation maintained by an organization, identifying in-scope reports should be the easiest step for an organization. In-scope reports are likely already documented in either the documented control descriptions, walkthrough documentation, or work papers created when testing a control. In the above example, the report name is "Active Responsibilities and Users". We suggest referencing the Control ID and the Control Owner. This will help facilitate the notification to the person that will be responsible for the re-performance of the C&A testing.
Step 2: Identify the core components that can effect changes to completeness and accuracy of in-scope reports
Identifying how the report can be changed is the secret sauce that is unique to each system from which the report is derived. Each 'system' from which a report is used for testing should be an in-scope system for SOX purposes. Thus, we recommend that all the reporting for in-scope reports be done from the core system, not bolt-on applications. We highly recommend NOT using Oracle reporting products like OBIEE or Discoverer or third-party products like GL Wand for in-scope reports to avoid having to worry about full ITGC testing that should be performed on those systems if you are using reporting from them. Instead, take the logic that has been developed within those reports and build them into Oracle as Concurrent Programs.
For core reports in Oracle EBS, Concurrent Programs, there are three areas where the report could be changed. First, the object at the OS level. Second, the registering of the object in the applications as an Executable. Third, the registering of the Executable against a Concurrent Program. If any of these three components are changed, the C&A testing should be re-performed. Refer to Step 3 for descriptions of the three areas.
Additionally, some reports may also be impacted by a configuration. For example, if the report is an AR Aging, the AR Aging Buckets configuration would need to be monitored as it would impact the aging of an invoice.
In the above spreadsheet example, the report name is "Active Responsibilities and Users". For that report, we need to identify any configurations that could influence the report, the Executable that calls the object in the database, and the Concurrent Program that calls the Executable when running the report.
See Appendix A for screen shots of the Concurrent Programs and Executable forms.
Step 3: Pro-actively monitor the changes to the components that can change the report
Once you have identified the core components in the report that need to be monitored, management needs to put in place a process to pro-actively monitor for changes. Management should NOT assume that all changes to these components will be processed through the change management procedures, but rather should proactively query the database for changes to the components.
Concurrent Programs and Executables Changes
The Concurrent Programs and the Executables data are stored in the database so these can be easily monitored. It would be best to have a trigger- or log-based software package in place to develop the audit history of such changes. We prefer solutions from Oracle – CCG or from CaoSys – CS*Audit. Both are trigger-based solutions that can have meta-data mapped and can have real-time notifications built. The reasons we prefer trigger-based solutions over log-based solutions are beyond the scope of this white paper. Please contact me at firstname.lastname@example.org for more information.
Absent the implementation of a trigger- or log-based auditing solution, the next best approach would be to query the tables within the database on a periodic basis (weekly would be preferable) to monitor whether anything has changed for the in-scope reports. The frequency of the review would partially depend on how often the controls using the reports are performed. Ideally, you'd want to identify if the report was changed before it was used again so that the C&A could be relied upon.
The changes to the objects which are stored in the Operating System are trickier. It is difficult to get a complete population of changes similar to what triggers do. Certain third-party applications are available to help, but they are based on snapshots. Snapshots compare the objects as of two points in time rather than providing a complete audit trail of all changes. The audit industry has accepted the use of the DBA_OBJECTS table as proxy for what happens at the OS level. The DBA_OBJECTS table contains CREATED and LAST_DDL_TIME fields with associated dates when additions and changes are made to the objects.
The challenge with the DBA_OBJECTS table is that the LAST_DDL_TIME is updated when an object is re- compiled. Therefore, a re-compile ends up being a false positive. As management identifies updates to the LAST_DDL_TIME that are false positives, they should investigate to confirm, then document that assumption as audit evidence.
The approach that many audit firms have taken related to in-scope reports has been the second approach described above – to test the controls over the accuracy and completeness of the information. This is what I would call a general ITGC / change management approach. Taking this approach requires ALL objects within that table be subject to testing. Reportedly, one of the Big 4 firms required their client to tie back every one of the objects that had been created or updated to a change ticket. A recent query I did of the DBA_OBJECTS table resulted in 430,000 rows. There were thousands of records that had the LAST_DDL_TIME within the audit period. The same client had less than 40 in-scope reports. A risk-based approach to the audit would have focused on the 40 or so in-scope reports rather than requiring the client to evaluate the changes to thousands of records related to objects with little risk related to the SOX audit. Most of those updates were likely due to the recompiling of objects.
I am not suggesting that management, nor the IT auditors no longer test the object changes as it relates to the change management audit. That exercise is still important.
However, taking this approach should help to resolve the over-scoping of object changes related to in- scope report object changes and potential control violation(s) that would need to be considered if there were a failure related to an in-scope report or management hadn't documented a change related to an in-scope report.
Step 4: Re-perform completeness and accuracy testing for those reports that have been changed
The final step in the process is to re-perform the C&A testing for those reports that have been identified as having been changed. That exercise is outside the scope of this white paper.
One final takeaway for management
Take an inventory of the systems from which in-scope reports are pulling data. To the extent possible, reduce reliance on non-core systems such as external "end-user type" reporting tools such as spreadsheet-based solutions or data warehouse reporting type tools. In Oracle's terms, avoid reports that are developed via Discoverer, OBIEE, or third-party tools like GL Wand. Such tools do not lend themselves to easily monitor for changes to the report's components and are difficult to develop access controls and a change management process over such changes. To the extent possible, take the logic built into such reports and build them within the core reporting capabilities. In Oracle's terms build them into the core application as a Concurrent Program.
Management should be able to significantly reduce the scope of internal and external auditors by putting in specific procedures to identify when changes are made to in-scope reports. Doing so will require some discussions with the external audit firms to change their approach to testing. However, these additional procedures should make it well worth the effort because of the reduction of audit scope.
Appendix A – Screen Shots of Executable and Concurrent Program