Tim Mccollum, David Salierno "Choosing the Right Tools"
By TIM MCCOLLUM and DAVID SALIERNO
Today's auditors face a dizzying array of software choices. Advice from the experts, along with the results of Internal Auditor's ninth annual software survey, may help narrow the field and facilitate decision-making.
CORPORATE GOVERNANCE AND REGULATORY concerns are among a number of forces putting pressure on organizations to establish stronger internal controls over financial records and systems. Consequently, internal audit departments are turning to audit software tools to help them meet the tighter deadlines and stricter requirements imposed on their companies by laws like the U.S. Sarbanes-Oxley Act of 2002.
Sarbanes-Oxley, enterprise risk management (ERM), increased reliance on enterprise software systems, and other factors are making chief audit executives (CAE) and internal audit managers more interested in using computer audit tools, according to the 2003 Internal Auditor software survey. Compared to last year, more survey respondents reported that their audit departments were using software for data analysis, fraud detection, control self-assessment (CSA), continuous monitoring, and workpaper automation. (See The IIA's Web site, www.theiia.org, for last year's survey results.)
Now that audit software tools are becoming more necessary, audit professionals must determine what types of software best fit their requirements, while avoiding the common pitfall of buying products that may not fit their audit environment. This can be a daunting task given the ever-growing assortment of commercially available audit packages, which must be weighed against using general-purpose software such as database or spreadsheet programs and developing in-house, tailor-made solutions from scratch.
To ease the decision-making process, we've gathered advice from several experts and surveyed practitioners on their product choices and selection criteria. The survey results and expert advice may help auditors make better-informed software decisions and increase the odds of finding the right tools for their department.
NEEDS SHOULD DRIVE SELECTION
Experienced information technology (IT) auditors say an organization's specific needs should guide audit software selection, just as they would for any other software tool. However, it's tempting for audit managers to think buying software will solve their problems. According to the experts, internal auditors often fail to use audit software correctly because they choose the wrong tools for the job.
"A lot of software companies are saying that you need their software to ensure Sarbanes-Oxley compliance," says John Tongren, managing partner of audit consulting firm Tongren & Associates in Norton Shores, Mich. "The assumption is that the software is doing things correctly. Does the internal auditor know enough about the organization's requirements to make sure that the software is doing what it really should be doing?"
The most difficult task for some audit shops is defining their specific requirements, according to Dave Coderre, audit principal with the Royal Canadian Mounted Police (RCMP) in Ottawa, Ontario. "Auditors may not know what is possible," he says. "And they may only be looking at automating what they currently do manually, and that's a limited view."
ERM is a leading reason that organizations are using audit software, particularly CSA and risk management/analysis tools, according to the Internal Auditor survey. For many internal audit organizations, audit tools are a low-cost and automated way to meet their increased governance and risk management responsibilities.
In the past three years, auditors at Denver-based business software maker J.D. Edwards have been acquiring new tools to help the company with a variety of audits. J.D. Edwards has only a nine-person audit shop to serve a company with offices in more than 20 countries, but like many technology companies, it relies on software to enhance the capabilities of auditors and do more with minimal personnel, says Senior IT Audit Manager Mark Bigler.
"Instead of adding bodies, we've tried to get smarter and taken more of a risk-based audit approach using automated tools," Bigler says. "We're using tools that help us identify potential problems on the front end, as opposed to detecting them on the back end." Bigler says the company is currently shopping for risk management and internal controls-related software, along with forensics software, and is developing IT s own tools to perform software license audits at client sites.
According to audit consultant Tongren, before audit departments can determine their audit software needs, they must first define:
- THE AUDIT MISSION, OBJECTIVES, AND PRIORITIES. Corporate governance and risk management issues have given internal auditors new responsibilities and requirements. Auditors must work with senior management and the corporate board to determine how audit software will help them meet these responsibilities and which responsibilities are most important. Auditors also need to know how much freedom management will give them to conduct different types of audits that software tools may facilitate.
HBOS PLC, the result of a merger between Bank of Scotland and Halifax PLC, chose to upgrade its audit management software last year to align with the bank's new group internal audit minimum standards methodology. The new methodology reinforces a risk-based approach to audits and is used in conjunction with a risk assessment model that identifies risk at the group, divisional, and enterprisewide levels, according to Bob Holroyd, HBOS senior audit manager for quality assurance and continuous improvement.
"We changed our approach to using the tool, as a result of the methodology," Holroyd says. "Previously, people tried to work within the constraints of the software and were, in effect, driven by it. But the system is now seen as a tool that helps auditors undertake their audit work within our methodology, which allows far more flexibility and creates consistency because they are using the same basic framework."
- THE TYPES AND SCOPE OF AUDITS. What is the purpose of the audits? What kind of questions will auditors be asking? What are their boundaries?
- THE TECHNOLOGY ENVIRONMENT. What kinds of hardware, software, and network systems does the organization have in place? Audit tools will need to fit into the existing environment. Some enterprise systems such as relational databases may have built-in tools that auditors can use to obtain and analyze the audit data they need, which could reduce their need for an audit package. Also, auditors may already be using tools such as spreadsheets and databases for audits that serve the same needs.
- THE RISKS. Like any software, audit software has its risks and can sometimes cause more problems than it solves, Tongren says. Automated tools may make assumptions that run counter to a business' operations. These tools may also interfere with normal operations. Tongren notes that, unless they're used judiciously, such products may be intrusive by uncovering every little defect and encouraging clients to hide problems.
In addition, organizations should look at whether the benefits of the audit tools will be worth the investment. For off-the-shelf packages, any cost savings the software allows auditors to uncover can easily exceed the cost of licensing it, says Rich Lanza, an internal audit manager at a Fortune 200 retailer and founder of AuditSoftware.Net, a Web site that advises auditors about using audit tools. Expensive applications that require considerable customization and user training may not pay off as quickly, if they pay off at all, he adds.
WEIGHING THE OPTIONS
Once an organization has determined its audit software needs and requirements, it can begin to look at its options. Audit managers have a wide array of software choices these days, ranging from data analysis and extraction packages to more specialized tools for CSA, information-security vulnerabilities, Sarbanes-Oxley compliance, and other audit concerns. However, these aren't the only solutions internal auditors have at their disposal. "You really should be using the right tool for the job and not trying to focus on a single tool and forcing it to do everything," Coderre says.
Many internal audit shops effectively use general-purpose software such as databases and spreadsheets for audits. This makes sense for several basic audit tasks, as long as the amount of data doesn't exceed the limits set by the software, Lanza says. For example, many spreadsheets are limited to 65,536 rows of data, whereas audit-specific data-analysis tools can handle considerably more.
"You can do a lot of analyticals such as regression analysis, ratio analysis, trend analysis, and performance measuring using spreadsheets," Lanza explains. Moreover, he says general-purpose tools may be a better option for some audit tasks because they cost less and auditors are already accustomed to working with them. "Why buck the trend and learn new software, when you might already have a familiar in-house tool available?" he says.
If auditors need more robust capabilities, they may be able to harness tools built into business-intelligence software or enterprise business and database systems such as Oracle, PeopleSoft, and SAP. These tools are typically under the control of the IT department, so auditors may have to ask IT staff to extract and provide data, Lanza says.
In addition to using the tools the organization already has in place, internal audit departments may be better served by developing their own audit software — either stand-alone tools or by adding capabilities to existing tools. Respondents to Internal Auditor's survey reported using such homegrown solutions in areas such as fraud detection, risk analysis, CSA, and automated workpapers. This may be the only option for organizations in highly specialized environments that aren't covered by the audit routines of off-the-shelf products.
Software company J.D. Edwards developed its own tools to allow internal auditors to measure customers' software license compliance during annual reviews. The company is able to do this because it has its own software engineers on staff, although Bigler says engineers don't always have time to work on audit tools because they are focused on developing the company's business application products. He says it makes sense to develop tools in-house when there is a specific need that can't be filled by commercially available software and when engineers can build a tool based on technology used in their existing enterprise or database software products.
Not all organizations have their own software developers, however, and non-audit-specific tools such as database and spreadsheet programs may fail to meet all of the department's requirements. For many organizations, using off-the-shelf audit products may be the best approach.
Data analysis, CSA, risk management, and other audit software have matured in recent years and incorporate features geared specifically to auditors' needs, IT audit experts say. Many products now work more seamlessly with enterprise systems, making it easier to extract and analyze data. In addition, the increased focus on governance and risk management are likely to spur demand for audit-specific tools.
At the same time, there is a growing number of auditors with experience using off-the-shelf audit products who have become in-house advocates, experts, and trainers for using the software. Also, organizations that have used audit products in the past are likely to upgrade to newer versions because auditors already know how to use them.
"It was sensible for us to use a product that was already well-established in our department," Holroyd says of HBOS's recent upgrade of its audit management software, which it has used for four years. "Many auditors were already familiar with it, and training requirements were limited."
EVALUATING SOFTWARE
Regardless of the approach, organizations should evaluate audit software based on their defined needs and requirements. In general, organizations must weigh their choices against a mix of criteria such as capabilities and features, ease-of-use, service needs, vendor record and expertise, and cost.
"You have to mix it all together and establish your own priorities about what needs to be done and at what cost," consultant Tongren says. "This comes out in questions related to risk management, which means that the process becomes more organizationally dependent."
Product features are a large concern, because they determine whether a particular tool will be able to perform the types of analyses that auditors need. In fact, respondents to the Internal Auditor survey cited features as their No. 1 criterion for audit-tool selection. One crucial feature is the software's ability to process and analyze the volume of data that auditors need. Another is its ability to extract data from the organization's databases and data warehouses.
According to Lanza, the majority of leading off-the-shelf audit packages share most of the same features and perform most of the same analyses. Here, ease-of-use — the second-highest concern among respondents to the survey — may be an important distinguishing factor to help auditors perform data tests and produce reports.
There are greater features differences among more specialized tools, Lanza says. This is particularly true of newer categories of tools like Sarbanes-Oxley products.
J.D. Edwards' Bigler is currently reviewing responses from a request for proposals for Sarbanes-Oxley compliance tools. In addition to requirements such as ease-of-use and database and operating system compatibility, Bigler's key criterion is that the software be Web-enabled, so that the company's various offices can document controls and provide financial reporting, operations, and compliance data online in real time and auditors can review the company's internal controls.
"There are so many people who need to contribute information," Bigler explains. "It's a lot easier if we can allow them to update their information over the Web, as opposed to trying to get input from other offices via e-mail or some other less effective/efficient method."
Tongren says vendor record, reputation, and expertise are key software-selection criteria. Internal auditors need to know what type of service they will receive from vendors, particularly for complex enterprise systems. They also need to know that the vendor is a viable company that will be around to pro-vide service, updates, and upgrades in the future.
In addition, Tongren says internal audit departments need to work with vendors who understand the organization's particular requirements. The vendor's audit expertise is also important, because it shapes the analytical assumptions that are built into the software. This is particularly necessary for small organizations that are more reliant on the software and for products such as risk-management systems. "If you have very specific requirements, and they aren't understood by the vendor, you start out with a high potential for problems such as false assurance," Tongren says.
One way to evaluate vendors and software packages is to ask companies that use the software about their experiences — particularly organizations in similar industries or with similar requirements. Audit managers should ask vendors for referrals, but also obtain a more independent perspective from members of product user groups. Tongren says many of the established audit software products have large and active user organizations.
The software's cost is another factor that will determine both the type of software purchased and the extensiveness of the product-evaluation process. A software package costing hundreds of thousands of dollars may require a significant purchasing process involving auditors, IT , purchasing, and senior management. The process for less expensive off-the-shelf products may be simpler.
"If I want to buy a computer forensics tool for ,000, I'm not going to go through a full-blown acquisition life cycle," Bigler explains. "But if it's a system that's going to cost ,000 or ,000 to help us with Sarbanes-Oxley compliance, then we'll go through the hoops on that."
PUTTING IT TOGETHER
Purchasing audit tools doesn't necessarily mean they will be used effectively — or used at all. Lanza says many organizations buy audit tools and then don't put them to use because they don't know what questions to ask, something they should have addressed during their needs assessment.
Internal auditors often need to learn how to put audit tools to work, which is where IT audit specialists like RCMP's Coderre can help. RCMP auditors had been using audit software for less than a year when Coderre joined the organization to help improve the efficiency and effectiveness of the internal audit function. Coderre acts as an in-house consultant, showing auditors what the tools are capable of doing and helping them plan audit strategies and formulate questions that identify risks and produce more accurate and useful audit reports. His most important task is helping auditors better understand the data they are analyzing.
"I don't care what tool you are using, you have to start with the data," Coderre says. "Over the past 10 or 15 years, access to data has become easier, but understanding the data still remains difficult and perhaps is becoming even more difficult. People tend to think that if they have an effective analysis tool, it's going to shortcut the process of understanding data, but it's not."
The more complex the tool, the more auditors will need training and guidance to use it effectively, Coderre says. In many cases, new tools and new audit demands require auditors to learn additional skills and change the way they perform audits. Some auditors adapt to new tools and techniques more quickly, but Coderre says others may lack the necessary computer skills.
Learning to use audit tools is a start, but only if audit departments are committed to using them on most of their audits, Lanza says. "Everybody has audit software," he notes, whether it is off-the-shelf packages, specialized tools, or simple spreadsheets. "Everybody has some way to get at their data. The reality is, they aren't really sitting down and doing it."
SIDEBAR 1: The 2003 Software Survey Results
SIDEBAR 2: Sarbanes-Oxley Software
TIM MCCOLLUM is editor of ITAudit, The IIA's electronic information technology publication.
DAVID SALIERNO is senior editor of Internal Auditor.
To comment on this article, e-mail the authors at dsalierno@theiia.org.
Side Bar 1: The 2003 Software Survey Results
Survey data compiled by The IIA's Global Auditing Information Network (GAIN)
AUDIT PRACTITIONERS WERE EAGER to discuss their use of software tools for Internal Auditor's ninth annual software survey. In fact, we heard from a record number of participants this year, with responses from 688 out of the 4,709 IIA members who were invited to participate. Although most of these individuals hailed from the United States and Canada, 10 percent were from outside North America — another record for the survey. Among countries represented in the sample were Australia, Indonesia, Qatar, Switzerland, the United Kingdom, Iceland, Kuwait, Namibia, and South Africa.
Respondents included audit directors, chief audit executives, and IT audit directors from a wide range of industries. Banking/financial services comprised the largest industry segment, followed by government, manufacturing, insurance, wholesale/retail, and services, respectively. Company size also varied widely, although most of the participants work at firms that employ fewer than 5,000 people. Approximately 20 percent of the responding auditors come from organizations that employ 10,000 workers or more.
Survey participants told us about the products they're using in several categories of software, including data extraction and analysis, fraud detection, network security assessment, automated workpapers, control self-assessment, and continuous monitoring. The auditors indicated product preferences, performance ratings, and the purposes for which they are using their software tools.
Overall, software usage is up compared to last year, and the applications for audit tools are expanding. Detailed results are presented in the following graphs.*
*Performance ratings such as overall satisfaction, effectiveness, and training required should be viewed with caution, particularly for products used only by a small percentage of respondents.
Side Bar 2: Sarbanes-Oxley Software
INTERNAL AUDITOR'S 2003 SURVEY IDENTIFIED tools used to help ensure compliance with the U.S Sarbanes-Oxley Act of 2002 as an emerging category of audit software. Among those polled, 27 percent of auditors indicated that they were either using this type of tool or were in the process of evaluating software of this nature. The segment of respondents that showed interest in this category of products is likely restricted to public companies subject to the new legislation.
Most participants are looking to Big Four firms for their Sarbanes-Oxley software needs, although other off-the-shelf audit products, as well as general-purpose and internally developed solutions, are also being employed. When asked to specify the main reason for choosing their particular product, 38 percent said "features" — the highest-ranked selection criterion. Fourteen percent made their decision primarily based on recommendations from others, whereas 10 percent based their choice on the product's ease of use. Cost was the deciding factor for only 6 percent of participants.
Internal auditors were not alone in the software decision-making process. Among those who either purchased or are considering purchase of a Sarbanes-Oxley tool, most said that company officers outside of internal auditing were — or will be — involved in the selection of the software. Participating company officers included the controller, the chief financial officer, and the chief information or technology officer. More than one-fifth mentioned the involvement of IT staff in the selection process, and nearly 25 percent cited participation by a member of the accounting department.
Approximately two-thirds of those using or considering the use of Sarbanes-Oxley tools expressed the need for both product configuration and training from a software provider. Given that these products have been introduced only recently, and the newness of the legislation itself, participants' need for implementation guidance is not surprising.
Although Sarbanes-Oxley tools seem to be on the minds of many respondents to the survey, this category of software is still in its infancy. More product offerings will likely emerge in the coming months, and those companies with tools already on the market will seek to hone their existing products based on the early experiences of their user base. The importance of the Sarbanes-Oxley legislation, and consequences for non-compliance, almost guarantees that these tools will be around for some time, and that interest in using them will continue to grow.
