The content on this page has been converted from PDF to HTML format using an artificial intelligence (AI) tool as part of our ongoing efforts to improve accessibility and usability of our publications. Note:

No human verification has been conducted of the converted content.
While we strive for accuracy errors or omissions may exist.
This content is provided for informational purposes only and should not be relied upon as a definitive or authoritative source.
For the official and verified version of the publication, refer to the original PDF document.

If you identify any inaccuracies or have concerns about the content, please contact us at [email protected].

Thematic Review Audit Sampling

Purpose and scope

Audit sampling is a fundamental tool for auditors, allowing the auditor to draw conclusions about a population based on the sample selected, given it is rarely practical to test 100% of a population. Different sampling approaches can result in significant differences in the size and composition of the sample, and how representative it is of the overall population. This impacts the extent to which auditors can draw valid conclusions about the population as a whole and obtain sufficient appropriate audit evidence.

The FRC has undertaken a thematic review of the audit sampling methodologies and guidance for audit sampling for the seven firms' classified as Tier 1¹ in the 2022/23 inspection cycle. This was undertaken to understand the extent to which the firms use sampling, including the impact of advances in technology, and understand and assess the different sampling methodologies used by these firms, including the role each firm attributes to sampling in building up the body of audit evidence. The principles-based nature of the ISA (UK) requirements allows for significant variation across firms.

The purpose of this thematic is to:

Identify common practice, concerns, and good practice across these firms to drive improvement and support our monitoring of the firms' systems of quality management.
Share findings to educate the wider audit market, as sampling has been an area of repeated Audit Quality Review (AQR) findings for smaller firms, and
Support Audit Committees in understanding and evaluating the approach taken by audit teams.

To undertake this thematic we:

Reviewed and benchmarked the firms' methodologies, guidance, and tools for selecting samples, extrapolating findings, and following up anomalies and errors. We reviewed the resources available to engagement teams from December 2021 onwards.
Assessed how much flexibility firms allow engagement teams in determining their sampling approach.
Held discussions with the UK audit firms included in this thematic review.
Reviewed specific audit working papers from eight AQR inspections in the 22/23 cycle.
Reviewed historic findings that related to sampling for the 2021/22 AQR inspection cycle.

The three key areas within the scope of this thematic review were:

Audit firms' methodologies relating to sampling as described in ISA (UK) 500 Audit Evidence and ISA (UK) 530 Audit Sampling as a means of selecting samples on which to perform tests of detail.
Sampling methods deployed in testing information produced by the entity (IPE) and attribute testing.
Sampling methods deployed in tests of controls.

Key observations and recommendations

Audit sampling, for tests of detail and controls, is still prevalent, despite the advent of tools such as Audit Data Analytics (ADA).

Most firms' methodologies are based on similar statistical models with firms building on these with their own guidance and preferences. This has led to substantial variation in the firms' final methodologies.

This variation does not indicate one approach is better, but stakeholders, such as audit committees, need to be aware of these variances to understand how the firms obtain audit evidence.

Recommendations

Audit committees should understand how auditors obtain audit evidence to support their choice of auditor when tendering and to aid understanding of how their auditor undertakes the audit.

When applying these methodologies in practice, professional judgement is key, with significant professional judgements made throughout the use of audit sampling.

Judgement is needed to use firms' sample size calculators, including to assess inherent risk and determine the contribution of evidence from other procedures. The extent of firms' guidance to support these judgements is variable.

Audit firms should ensure they provide engagement teams with sufficient guidance and training to support their use of professional judgement in audit sampling.

Previous AQR findings, and our sample review of on-going audit inspections, indicate insufficient evidencing of the key professional judgements made when determining sample sizes. Evidencing these key judgements is vital.

<blockquote="1"> All audit firms should update their methodologies and guidance to drive better documentation of key professional judgements in this area.

Background and key terminology

ISA (UK) 500 Audit Evidence establishes Audit Sampling as one of the primary means by which an auditor may select items for testing from a population in order to obtain audit evidence, alongside ‘Selecting all items (100% examination)’ and ‘Selecting specific items’². Population in the context of audit evidence is defined as “the entire set of data from which a sample is selected and about which the auditor wishes to draw conclusions”³. ISA (UK) 530 Audit Sampling then expands on this, with specific requirements and application material. The requirements in ISA (UK) 530 are applicable when selecting samples for performing tests of details and tests of controls⁴.

Audit sampling is a frequently used technique in the evidence gathering phase of an audit as it is often impractical to test 100% of a population (for example, where a population comprises many smaller items). Audit Sampling allows auditors to use statistical theory to obtain sufficient appropriate evidence that a population is not materially misstated as a whole, without having to examine each item within a population. It is also commonly used when undertaking tests of controls, where it would be impractical to test the operating effectiveness of all occurrences of a control.

Objective of audit sampling

“The objective of the auditor, when using audit sampling, is to provide a reasonable basis for the auditor to draw conclusions about the population from which the sample is selected."

ISA (UK) 530 Audit Sampling, Paragraph 4

ISA (UK) 530 sets requirements in relation to the following key areas:

Sample Design, Size and Selection of Items for Testing
Performing Audit Procedures
Nature and Cause of Deviations and Misstatements
Projecting Misstatements
Evaluating Results of Audit Sampling

Key definition: audit sampling

"The application of audit procedures to less than 100% of items within a population of audit relevance such that all sampling units have a chance of selection in order to provide the auditor with a reasonable basis on which to draw conclusions about the entire population.”

ISA (UK) 530 Audit Sampling, Paragraph 5

ISA (UK) 530, alongside defining Audit Sampling, includes several other important definitions and concepts:

Statistical Sampling⁵ An approach to sampling that has the following characteristics:
1. Random selection of the sample items, and
2. The use of probability theory to evaluate sample results, including measurement of sampling risk. A sampling approach that does not have characteristics (i) and (ii) is considered non-statistical sampling.
Sampling Unit⁶ The individual items constituting a population.

Sample selection methods are split broadly into two groups, in line with the Statistical Sampling definition, as illustrated below:

A diagram titled "Audit Sampling" splits into two main categories: "Statistical" and "Non-statistical".

Under "Statistical" methods, the following specific approaches are listed: - Random Sampling - Monetary Unit Sampling (MUS) - Systematic Sampling

Under "Non-statistical" methods, the following specific approach is listed: - Haphazard Sampling

In addition, the following definitions of inherent risk, information produced by the entity (IPE) and attribute testing, and Monetary Unit Sampling (MUS) are important for understanding the subsequent sections of this thematic review.

Key definition: inherent risk factors

"Characteristics of events or conditions that affect susceptibility to misstatement, whether due to fraud or error, of an assertion about a class of transactions, account balance or disclosure, before consideration of controls. Such factors may be qualitative or quantitative, and include complexity, subjectivity, change, uncertainty, or susceptibility to misstatement due to management bias or other fraud risk factors insofar as they affect inherent risk.”

ISA (UK) 315 (Revised July 2020) Identifying and Assessing the Risks of Material Misstatement

Key concept: information produced by the entity testing

IPE testing, in a similar manner to controls testing, uses fixed sample sizes, with engagement teams using these samples to ensure that reports provided to them by audited entities are reliable. For example, it could be used to test completeness by ensuring that supplier invoices are included in the payables report.

Key concept: attribute testing

Attribute testing is used to gather sufficient evidence to either accept or reject a characteristic or attribute of interest. It does not provide evidence over the monetary amount within a population. For example, it could be used to test if a sample of transactions have had the correct VAT % added to them.

Key definition: monetary unit sampling (MUS)

"Monetary Unit Sampling is a type of value-weighted selection in which sample size, selection and evaluation results in a conclusion in monetary amounts.”

Value weighted selection is a means of sampling whereby the identified sampling unit is the individual monetary units which make up the population. Having selected specific monetary units from within the population, for example, the accounts receivable balance, the auditor may then examine the particular items, for example, individual balances, that contain those monetary units.

ISA (UK) 530 Audit Sampling, Appendices 1 and 3

Audit methodology and application observations

In this section, we outline the detailed findings of our review of sampling methodology and its application by:

Summarising each type of sampling and its importance for a high-quality audit.
Highlighting common areas of practice across the firms reviewed.
Describing good practice identified from audit firms' methodology and guidance.
Outlining good practice identified through the application of the audit methodology on the audits reviewed, and
Providing examples of poorer audit methodology, or application, as potential pitfalls for others to avoid on future audits, based on the work performed for this thematic.

1. Summary of approaches

ISA (UK) 530 Audit Sampling contains the requirements auditors need to meet when using audit sampling as a means of selecting items for tests of detail and tests of control. Though it includes specific definitions, it does not prescribe a specific approach or underlying statistical model that must be deployed to meet the ISA (UK) objectives and firms are required to develop a methodology which ensures engagement teams meet the requirements of the ISAs (UK) when using audit sampling.

In reviewing the firms' methodologies and guidance, we noted no significant deficiencies in meeting the objectives of ISA (UK) 530 Audit Sampling, and methodologies provided a range of statistical and non-statistical tools for engagement teams to deploy.

Most firms' sampling methodologies are based on the American Institute of Certified Public Accountants (AICPA) Audit Sampling Guide which introduces statistical and non-statistical sampling approaches and includes case studies, MUS sample size tables and methods for projecting errors across populations. The AICPA approaches are not prescribed in ISA (UK) 530, or in any other standard, but it has become the most common foundational model for the audit firms within this thematic.

Our thematic review does not include a detailed evaluation of the AICPA's approaches; our specific focus is on how the firms in scope have interpreted the requirements and principles in the ISAs (UK) and applied them to develop their audit sampling methodology.

Three firms make only small additions to the AICPA approaches, usually in their approaches to calculating sample sizes, and their methodologies are very similar, or identical to those included within the AICPA sampling guide. Four other firms build significantly on the AICPA model with substantial additional guidance, case studies to assist engagement teams and stated preferences for certain approaches, while still allowing engagement teams to judge when other techniques might be most applicable.

Five firms did not express an explicit preference for any approach over another when selecting samples for tests of detail and leave the method of sample selection to the engagement teams' judgement. One firm's methodology stated a preference for MUS. This firm stated that it preferred MUS as it can be easier to apply in a consistent manner. One firm had a stated preference for the use of non-statistical sampling though it noted that the outcomes are broadly consistent with established statistical principles.

Some firms provide substantially more guidance to engagement teams, with detailed case studies to help facilitate effective audit sampling, while others provided more limited or focused guidance. This was particularly true when looking at methodologies relating to the use of audit sampling of IPE and attribute testing where the extent of additional guidance provided was highly variable.

Most firms make use of internally developed tools that facilitate the deployment of their sampling methodologies, and which are usually mandated, including:

Sample Size Calculators – These range from reasonably simple spreadsheet-based tools to more complex bespoke solutions. Generally, engagement teams are required to input the population size and materiality, indicate if any key items or transactions are tested elsewhere and select the determined level of inherent risk. They are usually also required to input if they have obtained any evidence over the balance or transactions from other procedures, for example if they have performed tests of controls. Some tools will select a random sample for the audit team while others provide just a sample size and teams must select items themselves.
Monetary Unit Sampling (MUS) Tools – These tools are used at some firms to aid in the semi-automated use of MUS. These tools require similar inputs as more general sample size calculators but will typically select a sample automatically for the engagement team to examine.

All firms provided a methodology for projecting errors in a sample over the population as a whole in accordance with ISA (UK) 530⁷. For MUS tools, most firms provided an automatic calculation of the projected error in the total population or made use of a specific error extrapolation tool. For other types of sampling, all firms required the use of either the Ratio Method (where the projected error is calculated as the error rate in the sample multiplied by the population) or the Difference Method (where the projected error is calculated by taking the average difference between the recorded value and the actual amount determined by the auditor and multiplied by the number of items in a population).

Though all methodologies have a statistical model as their basis, one of the key determining factors in effective audit sampling is professional judgement and the application of this judgement to key decisions made throughout the process, specifically around the:

Level of Inherent Risk – The level of risk attributed to a balance or series of transactions has a significant effect on the number of items selected when sampling as this is a key input into sample size calculators. Balances or transactions at the lower end of the spectrum of inherent risk⁸ will require fewer samples to be tested for an engagement team to be able to conclude.
Level of evidence obtained from other procedures – The amount of evidence obtained from other procedures has a significant impact on the sample size. Where engagement teams state that they have obtained assurance from other procedures, such as Substantive Analytical Procedures (SAPs), most firms' methodologies allow the engagement team to select smaller sample sizes.

Key definition: professional judgement

“The application of relevant training, knowledge and experience, within the context provided by auditing, accounting and ethical standards, in making informed decisions about the courses of action that are appropriate in the circumstances of the audit engagement."

ISA (UK) 200 (Revised June 2016) Overall Objectives of the Independent Auditor and the Conduct of an Audit in Accordance with International Standards on Auditing, Paragraph 13 (k)

Application of methodology

As part of our work on this thematic, we reviewed historic AQR findings where audit sampling was a factor, and reviewed approaches taken to sampling on eight ongoing inspections. While this is only a small sample, it provided us with several observations regarding the practical application of audit sampling methodologies:

Despite professional judgement being one of the biggest factors driving the quantum of samples used, the evidencing of the key judgements was poor. This was particularly where engagement teams were relying on assurance gained from other procedures with very little explanation given (usually none at all) as to why the engagement team believed it had assurance from other work. This often substantially reduced the number of items selected, with instances of engagement teams selecting too few items to be able to conclude, due to having overestimated the amount of assurance obtained from other procedures.
Confusion as to the function of testing information produced by the entity and the use of attribute testing are common. Some engagement teams did not understand that IPE testing assesses the reliability of the information to be used as audit evidence, rather than being a test over the monetary value of a population.

As part of our review, we compared the relative sample sizes used by engagement teams when using audit sampling when performing tests of detail, IPE, and attribute testing, and when testing controls. Within the Controls Testing and Sampling section we have presented a table comparing the sample sizes used for controls occurring at differing frequencies. When considering sample sizes used for tests of detail, the number of factors that influence sample sizes makes direct comparison impractical. As such, we have included a discursive comparison of sample sizes in relation to tests of detail in the Sampling in Tests of Detail section of this thematic.

2. Sampling in tests of detail

Sampling undertaken when performing tests of detail forms only part of most audit firms' approaches to obtaining sufficient appropriate audit evidence. SAPs⁹, ADA¹⁰ and tests of controls are usually used alongside sampling to obtain sufficient appropriate audit evidence. The diagram below shows, on a generic basis, how tests of detail fit into the wider picture of the audit evidence obtained. The more evidence is obtained from other procedures, such as controls testing, the less needs to be obtained from a sample-focused test of details.

A diagram illustrating how Tests of Detail fit into the overall audit evidence process:

The process begins with "Risk Assessment," which categorises risk into: - Significant Risk - High Risk - Medium Risk - Low Risk

All these risk levels lead to "Sufficient Appropriate Audit Evidence."

"Sufficient Appropriate Audit Evidence" is formed by combining outputs from four areas: - Controls - Substantive Analytical Procedures - Data Analytics - Tests of Detail (Sample Based)

The diagram implies that evidence from Controls, Substantive Analytical Procedures, and Data Analytics contributes to the "Sufficient Appropriate Audit Evidence" and thus impacts the extent of "Tests of Detail (Sample Based)" required.

Moreover, the overall amount of audit evidence required is driven by the risk assessment of the balance being audited, with audit teams typically placing balances or transactions at three or four points along the spectrum¹¹ of inherent risk from Significant Risk¹² to Low Risk.

Audit firms often express this spectrum as a range of Confidence Levels (CL). Each of the risk levels above are assigned a required CL that must be obtained through all sources of evidence for an engagement team to conclude that it has sufficient and appropriate audit evidence.

Key definition: confidence level

In the context of audit sampling, the confidence level (CL) is the % probability that the auditor is required to have that a balance is not materially misstated. For example, a test performed to a 95% CL is interpreted by the auditor to mean that there is a 95% probability that the balance being tested is not materially misstated.

The CL that an engagement team is required to meet is a matter of professional judgement and is not defined in the ISAs (UK). Given the principles-based nature of the ISAs (UK) and the application of judgement, there is variation across the audit firms as to the required CL for different risk levels. Generally, the firms' methodologies required CLs are in the range of:

Significant Risk (90-95% CL)
High Risk (80-90% CL)
Medium Risk (70% CL)
Low Risk (33-50% CL)

These are a generic representation of the levels used across the seven firms in scope. Although no specific CL is required by the ISAs (UK), audit firms must be satisfied that a given CL is sufficient for obtaining evidence to support their conclusions over the specific risk.

For example, if a balance is assessed as high risk, the firm may judge that, for an engagement team to be able to conclude that it has sufficient, appropriate audit evidence, it would need enough evidence to conclude with 80% probability that the balance being tested is not materially misstated. For the same balance, if it was assessed as a significant risk, it would need enough evidence to conclude with 95% probability that the balance being tested is not materially misstated. This approach allows audit firms to adjust the amount of work undertaken in relation to a balance or series of transactions based on risk. However, they must still ensure that they collect sufficient, appropriate audit evidence to provide reasonable assurance over the financial statements as a whole¹³.

Many firms attach a numerical measure to the procedures, other than the test of detail element so that engagement teams are able to understand the extent of sampling required to reach a final conclusion on a balance. Generalised indicative ranges, based on the seven firms' methodologies, of the CLs obtained from other procedures are explained below:

Type of Procedure	CL % Obtained From Other Procedures (Ranges across firms in scope)	Observations
Controls Testing over relevant assertions		In most methodologies this is a binary choice to take controls reliance or not, though some firms allow for engagement teams to take enhanced reliance where they have tested additional controls above the minimum required.
Substantive Analytical Procedures	40 to 60%	The CL obtainable is usually dependent on the tolerable difference between the actual amount and auditors expectation. SAPs performed with a lower tolerable difference will usually generate higher amounts of evidence, for example to achievable a 60% CL, the difference between actual and the auditors expectation would have to be very small.
Data Analytics	20 to 60%	The CL obtainable is dependent on the sophistication of the analytic being used and in instances where the analytic involves setting an expectation, how close that expectation is to the actual.

In practice, the calculation is usually undertaken within the audit firms' sample size calculator, where an engagement team is able to select the amount of evidence obtained from other procedures from drop-down boxes. For example, an engagement team might determine that it has achieved the required evidence from controls work. Selecting this in the sample size calculator, will present the team with a lower required sample size than if it had undertaken no controls work.

Determining how much assurance is obtained from other procedures is challenging as CLs are calculated statistically by reference to populations and cannot easily be assigned to other types of procedures with a non-statistical basis. At most firms, these challenging judgements have been partly made in advance of the audit by central technical teams through setting ranges for CLs obtainable from procedures within the firms' audit methodology. This means that with engagement teams are only able to select from a limited number of options for the assurance obtained from other procedures. Some firms do not assign a numerical value and leave the determination of amount of evidence obtained to auditor judgement.

Following the above generalised model, an example of how the auditor might determine that they have obtained sufficient, appropriate audit evidence over a balance could be:

An example scenario showing how an auditor might determine sufficient, appropriate audit evidence.

The "Risk Assessment" is for "High Risk (90% CL)".

This high risk assessment leads to "Sufficient Appropriate Audit Evidence", which is achieved by combining: - Controls (contributing CL = 60%) - Substantive Analytical Procedures - Data Analytics - Tests of Detail (contributing CL = 30%)

The diagram indicates that the sum of CLs from Controls and Tests of Detail (60% + 30% = 90%) matches the required CL for "High Risk".

In this example, the auditor has obtained most of their evidence from controls testing and therefore places reliance on those controls when determining how much test of detail work is required. In most firms' methodologies, this will substantially reduce the number of items examined in the substantive sampling testing, highlighting the importance of making appropriate professional judgements regarding the amount of evidence obtained from other procedures. In instances where engagement teams overestimate the amount of evidence obtained from other procedures, they are likely to select too few items to allow them to conclude, a finding we have raised in several AQR inspections.

Some firms' methodologies provide more substantive guidance to engagement teams on how to determine the amount of evidence obtained from other procedures. One firm, whose determination of the level of evidence is discursive rather than numerical, included a number of example scenarios to support engagement teams in their determinations. Other firms provided less guidance material to engagement teams to aid in judgements about the amount of evidence obtained from other procedures.

AQR file findings

When considering professional judgements concerning the level of evidence obtained from other procedures, our AQR findings indicate a lack of evidencing of key judgements.

Several reviews had engagement teams that stated they had obtained larger amounts of evidence from other procedures than had been achieved. This led to a sample size that was too small to conclude upon.

Conversely, some teams have been uncomfortable with reducing sample sizes where other good evidence had been obtained.

Given the importance of this key professional judgement on the sample size calculator, audit firms should ensure that they provide audit teams with sufficient guidance to support professional judgement in this area. Firms with less guidance and support should consider expanding it.

In our review of AQR inspections, we found several instances where the amount of evidence from other procedures was judged inappropriately. This led to, in most instances, the selection of fewer items than was necessary to conclude on the population. In two instances, more items than were necessary were selected. Evidencing the judgements made was generally poor, with engagement teams infrequently documenting the rationale behind their judgements over the amount of evidence obtained from other procedures.

Sample Size Comparison Across Firms

As described above, the number of variables that impact sample sizes makes direct general comparisons between firms impractical. It was, however, possible to approximately compare the sample sizes by making some assumptions regarding the inputs into each of the audit firms sample size calculators. For each firm we input the following scenario:


Total Population Value (Gross Amount)	£31,500,000
Performance Materiality (PM)	£1,200,000
Population as Multiple of PM	x27
Conditions	Significant Risk, No Controls Reliance, No assurance obtained from other procedures

The range of sample sizes calculated with the firms' tools was 71 - 79 indicating that, in this specific scenario, the sample sizes across firms are consistent.

3. Key items selection and selecting specific items

Key concept: selecting specific items

Selecting specific items is a means of selecting items to test where an auditor does not apply sampling techniques. Engagement teams select items based on their understanding of the entity, the assessed risk of material misstatement and the characteristic of the population being tested.

This section discusses specific item selection as a means of obtaining audit evidence, in addition to testing key items from a population.

All the firms' methodologies allow engagement teams to select and test key items from the population before then selecting a sample of the residual population. All the sample size calculators reviewed allowed for the removal of key items tested elsewhere.

Most firms provide guidance to engagement teams on selecting key items, with a focus on high value items and those which indicate an increased risk of fraud, though there is variation in the extent of this guidance. Two firms provide limited guidance which focuses almost exclusively on the size of the items, with less consideration given to other risk factors. Two firms have substantially more detailed guidance than other firms on the range of factors that might indicate that something is a key item, with a particular focus on understanding the risks associated with items within the population.

AQR review comments

In several reviews, we saw insufficient documentation of the reasons for selecting items either as key items when audit sampling, or as specific items. When we did see justification, it was generally focused on size, such as “selecting everything over 50% of PM", with no consideration of why that was an appropriate threshold.

We saw good practice in one review, where they selected specific items for testing based on risk, understood the population well and documented their judgements and conclusions effectively.

As part of our AQR inspections, it was noted that the reason for items being identified as key items was rarely recorded. When it was, it was generally usually simply to state "selecting everything over 50% of performance materiality” with little or no justification as to why 50% was a meaningful percentage. As noted above, given the significance of these judgements, we would encourage all firms to communicate with staff the importance of appropriately recording judgements, and consider if their methodologies would benefit from additional guidance material.

Judgement is also applied when selecting specific items for testing as described in ISA (UK) 500¹⁴. With this approach, sampling techniques are not applied, and engagement teams select items based on their understanding of the entity, the assessed risk of material misstatement and the characteristic of the population being tested.

4. Haphazard sampling

Haphazard sampling was historically most useful when transaction listings were not available from audited entities in an electronic format that would allow for random sampling. Today, transaction listings and trial balances can typically be exported into a format suitable for analysis and use in sampling tools. This makes random sampling substantially simpler to perform, although there may still be instances where haphazard sampling is the most appropriate method, for example in stock-count floor-to-sheet testing.

A sample selected haphazardly rather than randomly, has a greater risk of bias. As such, extrapolated errors are less likely to be representative of the error rate in the population as a whole.

Key definition: haphazard selection

"Haphazard selection, in which the auditor selects the sample without following a structured technique. Although no structured technique is used, the auditor would nonetheless avoid any conscious bias or predictability (for example, avoiding difficult to locate items, or always choosing or avoiding the first or last entries on a page) and thus attempt to ensure that all items in the population have a chance of selection. Haphazard selection is not appropriate when using statistical sampling.”

Key definition: random sampling

“Random selection (applied through random number generators, for example, random number tables)."

ISA (UK) 530, Audit Sampling, Appendix 4

AQR review comments

In several reviews, we saw confusion in the method sample selection applied. The sample calculator stated “Random” as the means of sample selection but “Haphazard” was actually used by the engagement team. This led in some instances to potentially inaccurate projection of errors and to improper consideration of bias in the sample.

In multiple reviews we saw no documentation or consideration of why “Haphazard” sampling would be the most appropriate method when “Random” was clearly a plausible option and would have reduced bias.

Whilst haphazard sampling is permissible in the context of the ISAs (UK), and may, in certain situations, be the most appropriate, firm methodologies should actively encourage the use of random sampling over haphazard sampling where it is feasible to do so.

Following our review, all firms involved agreed that they will consider amending their guidance to ensure random sampling is clearly labelled as the preferred method over haphazard.

5. Sampling methodologies for information produced by the entity (IPE) and attribute testing

Sampling methodologies for information produced by the entity (IPE) and attribute testing are typically similar, with some firms describing both concepts individually, and other firms using the same approach for both methods.

Key concept: information produced by the entity (IPE) testing

IPE testing, in a similar manner to controls testing, uses fixed sample sizes, with engagement teams using these samples to ensure that reports provided to them by audited entities are reliable. For example, it could be used to test completeness by ensuring that supplier invoices are included in the payables report.

Key concept: attribute testing

Attribute testing is used to gather sufficient evidence to either accept or reject a characteristic or attribute of interest. It does not provide evidence over the monetary amount within a population. For example, it could be used to test if a sample of transactions have had the correct VAT % added to them.

The objective of IPE testing is to determine if the IPE, such as receivables ageing reports or loan schedules, is reliable in accordance with ISA (UK) 500¹⁵, before audit procedures are performed upon them to obtain audit evidence. This includes information that the entity provides that has been system generated, or manually produced.

The objective of some types of attribute testing is to obtain audit evidence over non-monetary assertions, for example if a set of transactions have been classified in the correct financial statement line.

There was significant variance in the extent of guidance provided to engagement teams on IPE testing and/or attribute testing. Two firms had very limited material within their methodologies, with sample size tables presented with little explanation of the scope and aims of IPE testing and/or attribute testing. Two firms had extensive guidance with focused case studies designed to help engagement teams understand the form of evidence obtained from IPE testing and/or attribute testing. One firm has removed attribute sampling from its methodology as its internal review process highlighted too many concerns regarding the quality of application, though IPE testing is still used to assess the reliability of information that will be used as audit evidence.

Some firms' methodologies allow engagement teams to test IPE by either testing the controls relevant to the report or by performing tests of details on the report itself. Other firms only allow engagement teams to make use of test of details approaches, though often with fixed sample sizes. Even at those firms where testing controls is an available approach, tests of detail has been the approach most commonly seen in AQR inspections. These approaches can be summarised as follows:

A diagram outlining approaches to IPE Testing to determine the reliability of information, for example a fixed asset register extracted from an accounting system.

There are two main approaches:

Approach One: Test of Controls - Test controls relevant to the extraction of the information from the entities system. - Approach to calculating sample size is firm dependent: 1. Sample sizes used for test of controls; or 2. Other fixed sample size. - Deviations are addressed in line with controls testing methodology and the number of deviations planned for in testing. This may involve concluding the information is NOT reliable if deviations indicate controls cannot be relied upon.

Approach Two: Test of Details - Test the detail of the report, agreeing a sample back to the system. - Approach to calculating sample size is firm dependent: 1. Sample size calculator used for test of details; or 2. Specific IPE sample calculator; or 3. Fixed sample size. - Errors are addressed in line with tests of detail methodology. This may involve concluding the information is NOT reliable where errors are found and are not determined to be isolated.

Most firms included guidance within their methodology on how to undertake dual-purpose testing. Dual-purpose testing is where an engagement team selects a sample and performs both IPE or attribute testing and undertakes additional procedures to obtain assurance over the monetary value of the population.

Given IPE and attribute testing sample sizes are generally lower than those required to conclude on a population's monetary value, a larger sample is selected. IPE or attribute testing is then only applied to the relevant proportion of that, which is a reasonable approach for dual-purpose testing such as this. Where the population is a smaller multiple of performance materiality, IPE, attribute, and test of detail sample sizes are usually closer to each other. For example, a sample size calculator may indicate that 40 samples are required in order to obtain sufficient appropriate audit evidence over a balance, with the IPE table stating that only 15 samples are required to assess the completeness of the report that balance is drawn from. As such, the engagement team would select 40 items and complete substantive procedures relating to the monetary value on all 40, and IPE testing on just 15 of the items.

Though most firms explain this concept clearly in methodologies, our AQR findings indicate that engagement teams appear to struggle with practical application and are sometimes unclear as to the dual objective of their tests. This led to, for example, teams only selecting the smaller number of samples and testing the monetary value on that sample, without including the additional samples needed to be able to conclude on the monetary value of the population.

Given our AQR file findings and discussions with the firms involved in the review, most audit firms would benefit from incorporating additional case studies and examples in methodologies to help engagement teams understand and deploy IPE testing effectively.

Firms without extensive additional guidance and case studies within their IPE and/or attribute testing methodologies to consider how their inclusion could support more effective deployment of IPE testing, particularly more complex techniques such as dual-purpose testing.

6. Controls testing and sampling

A test of controls is “an audit procedure designed to evaluate the operating effectiveness of controls in preventing, or detecting and correcting, material misstatements at the assertion level”¹⁶. Obtaining assurance over the operating effectiveness of controls allows an engagement team to reduce its sample sizes when undertaking substantive sampling by stating that it has controls reliance (as described in the Sampling as Audit Evidence section of this thematic).

All firms' methodologies included controls testing as a tool available to engagement teams, though two firms explained that they use controls testing less routinely as the audited entities within their portfolio typically have less mature control environments.

Key requirement: controls testing

"In designing and performing tests of controls, the auditor shall obtain more persuasive audit evidence the greater the reliance the auditor places on the effectiveness of a control.”

ISA (UK) 330 (Revised July 2017) The Auditor's Responses to Assessed Risks, Paragraph 9

All audit firms provide guidance to staff on selecting a sample of control occurrences to test. The firms provide sample size guidance for controls testing that is related to the frequency of the relevant internal control. The guidance describes how these sample sizes will allow the auditor to obtain a planned level of assurance and how the sample size is also dependent on the risk of material misstatement addressed by the control, the risk that the control will fail and/or the number of deviations in the test that are acceptable. Two audit firms have a separate sample size, set by the central team, specifically to be used for testing a control operating multiple times a day where a deviation is expected. Other firms do not have a centrally set sample size for that situation but would expect engagement teams to consult a sampling expert if they were anticipating control deviations.

Basis of sample sizes

Three of the firms have used predominantly the AICPA Audit Sampling Guide and three other firms have used the AICPA Audit Sampling Guide as well as aspects from another statistical model, such as the Poisson probability distribution model, to inform their controls testing sample sizes. One firm, however, does not base its sample sizes on a statistical model.

Comparison of sample sizes – frequently occurring controls

Each firm uses different terminology to describe the level of risk it attaches to a control or the level of assurance it wants to obtain from testing the controls, which makes direct comparison between audit firms' methodologies challenging. Most firms plan controls testing on the basis that there will be zero deviations, or failures of the control.

When testing a control operating multiple times a day (with zero planned deviations), sample sizes can range across firms from 10 to 60, although the larger samples are not used frequently as they are designed for specific scenarios, such as if the engagement team is only planning to test one control per assertion and similarly the smaller samples are not used frequently as they are designed for specific scenarios, such as when both the assurance level required from the control is low and the control risk is low.

Key definition: tolerable rate of deviation

“A rate of deviation from prescribed internal control procedures set by the auditor in respect of which the auditor seeks to obtain an appropriate level of assurance that the rate of deviation set by the auditor is not exceeded by the actual rate of deviation in the population."

ISA (UK) 530 Audit Sampling, Paragraph 5

The AICPA Audit Sampling Guide's Statistical Sample Sizes for Tests of Controls table¹⁷ allowed us to make an approximation of the tolerable rate of deviation being used in different firms, based on the sample sizes outlined in their methodologies. For the purposes of our analysis, the sample sizes used to make these estimates were the minimum and maximum sample sizes required by firms for testing a control which operates multiple times per day with zero planned failures of the control, at a 90% confidence level. The analysis demonstrated that different firms accepted broadly similar maximum rates of deviation from a prescribed internal control, despite not necessarily using the same basis to inform sample sizes.

Two firms allow the audit team to design their tests allowing for one deviation when testing controls operating multiple times daily. This does require a relatively big increase in sample sizes from 25 items to 40 items. Given the increase in sample size it is important to assess whether a deviation is expected. Audit firms should consider whether guidance on making this assessment is sufficient and appropriate.

Audit firms should also remind engagement teams that, in some cases, the prescribed sample sizes are minimum levels and that teams should consider whether these should be increased.

Comparison of sample sizes – less frequently occurring controls

For controls that are operated less frequently the table below summarises the firms' recommended minimum sample sizes and shows that the firms are broadly in line with each other as well as the AICPA Audit Sampling Guide's suggested number of items to test for small populations.

Source of sample size	Weekly	Monthly	Quarterly
AICPA suggested¹⁸	5-9	2-4	2
Range of all firms	4-11	2-4	1-3

Use of judgement

Two audit firms allow engagement teams to use professional judgement when assessing what the appropriate sample size is to test a control, this allows teams to increase sample sizes above the minimum if there are concerns about the operation of a control. Another audit firm allows the use of professional judgement in the scenario where engagement teams are only planning to test one control relating to an assertion. These professional judgements cannot be extended to scenarios where the engagement team considers the level of risk to be low and therefore would seek to reduce sample sizes below the minimum required.

Three of the firms provide a list of factors which, if present, engagement teams should take into consideration when judging if a sample size should be increased above the minimum stated in the tables. However, not all firm methodologies guide teams on the number of additional samples required and engagement teams must instead use their professional judgement. The factors include observation of deficiencies in the control environment and the importance of the control to the accuracy of financial reporting.

As with audit sampling in substantive testing, the application of appropriate professional judgement is the key to ensuring the effective use of audit sampling methodology in test of controls. Firms should ensure that engagement teams understand the importance of appropriate professional judgements and are able to evidence their judgements appropriately.

7. Sampling and International Standard on Quality Management (ISQM) (UK) 1

All the firms' methodologies were driven by a global methodology, usually developed centrally outside the UK, and then adopted by the member firms worldwide. Almost all the firms reviewed had additional UK-specific material to:

address either ISA (UK) requirements (where they are higher than the international version), and or;
respond to specific inspection findings at a firm level.

While we expected this, three firms relied very heavily on their global methodology teams to address our questions. While support from the global central functions is appropriate, we were surprised by the extent to which some firms relied on them to explain how underlying statistical models were used to develop methodology applied in the UK.

This is particularly important given the requirements in ISQM (UK) 1, which states that even when firms belong to networks and make use of resources, the firm "remains responsible for its system of quality management, including professional judgements made in the design, implementation and operation of the system of quality management"¹⁹. As such, audit firms need to ensure they have a proper and full understanding of the sampling techniques developed globally, and are able to understand and apply those methodologies in the UK.

In addition, some firms struggled initially to explain how their methodologies were developed from more general statistical models, often due to the time that had elapsed from the model's original development. Even though these models had been developed since their initial deployment, sometimes decades ago, audit firms need to be able to clearly explain how they developed and deployed the tools used in audit sampling. Audit firms need to ensure that their understanding of how their methodology relates to key statistical concepts is current.

8. Future of sampling in tests of detail

All the firms within the scope of this thematic have invested, often substantially, in tools and technologies, including those that make use of emerging technology such as Artificial Intelligence. Many of those tools focus on improving the auditor's ability to conduct risk assessments, though many are now being deployed in the evidence collection phases of audits. When discussing emerging technology usage as a means of collecting audit evidence, there was a diverse range of opinions across the firms, though all agreed that technology has a significant role to play in how engagement teams collect audit evidence and that was likely to impact on the extent to which substantive sampling is used as a source of evidence.

For some firms tests of detail remain the dominant form of audit evidence for most audit engagements. These firms all noted they are trying to reduce reliance on substantive sampling through increasing the controls work they undertake for clients with suitable systems of internal control. These firms noted that their approach was often driven by the nature of their clients, with some audited entities unable to provide data of high enough quality to facilitate the use of ADA tools. These firms also described issues with recruiting those with appropriate skills as a potential limiting factor in the deployment of tools and technologies.

Other firms have begun to reduce, or consider reducing, their reliance on sampling, with a move towards increased use of ADA during evidence collection, most commonly in revenue testing. One firm noted that it envisioned a future where sampling was a 'last resort' source of evidence where controls and ADA tools were not suitable for use. IPE testing to assess the reliability of information provided by the entity was noted as an area where sampling would still be used even as firms move towards greater reliance on ADA and reduce their reliance on substantive sampling. Controls testing would also continue to be used extensively.

However, even these firms recognised that sampling will still be frequently used given the number of audited entities where the quality of data is not sufficiently high to be used in ADA.

Audit sampling will also still have a place in IPE, attribute and controls testing even at firms where it is used less for tests of detail, as it may remain the most efficient means of obtaining assurance over the operating effectiveness of controls and in the application of assertion testing.

Appendix

Limitations of this thematic

We outline below the limitations of our scope as part of performing this thematic:

This thematic was primarily based on our review of the firms' methodology and ancillary guidance, and on representations made by the firms in response to specific questions. While instances of good practice have been highlighted, we do not provide any assurance over the sufficiency of an individual firms' methodology, policies and procedures.
For the AQR inspections, we did not have direct access to the full audit files and this thematic did not assess the quality of the audit.
The good practice examples referenced in this thematic may not equate to good practice reported in the Audit Quality Inspection and Supervision Reports for individual firms, or good practice included in AQR inspection reports on individual audits. Similarly, examples of poorer audit work may not equate to issues which may be included in those reports.

Other relevant FRC publications

The following FRC publications should be read in conjunction with this thematic. These outline key auditing principles and messages that are also relevant to aid in achieving quality audit evidence through the use of audit sampling:

Professional Judgement Guidance
Using Technology to Enhance Audit Quality
The Use of Technology in the Audit of Financial Statements
What Makes a Good Audit?
What Makes a Good Environment for Auditor Scepticism and Challenge

Liability

No party accepts any liability for any loss, damage or costs howsoever arising, whether directly or indirectly, whether in contract, tort or otherwise from any action or decision taken (or not taken) as a result of any person relying on or otherwise using this publication or arising from any omission from it.

© The Financial Reporting Council Limited 2023 The Financial Reporting Council Limited is a company limited by guarantee. Registered in England number 02486368. Registered Office: 8th Floor, 125 London Wall, London, EC2Y 5AS

Financial Reporting Council 8th Floor 125 London Wall London EC2Y 5AS

+44 (0)20 7492 2300 Visit our website at www.frc.org.uk

PWC, Deloitte, KPMG, EY, Mazars, BDO and GT ↩
ISA (UK) 500, Audit Evidence, Paragraph A63 ↩
ISA (UK) 530, Audit Sampling, Paragraph 5 (b) ↩
ISA (UK) 530, Audit Sampling, Paragraph 1 ↩
ISA (UK) 530, Audit Sampling, Paragraph 5 (g) ↩
ISA (UK) 530, Audit Sampling, Paragraph 5 (f) ↩
ISA(UK) 530 Audit Sampling, Paragraph 14 ↩
ISA (UK) 315, Identifying and Assessing the Risks of Material Misstatement (Revised July 2020), Paragraphs A208 to A214 ↩
ISA (UK) 520, Analytical Procedures, Paragraph 4 ↩
AQR Thematic Review, The Use of Technology in the Audit of Financial Statements ↩
ISA (UK) 315, Identifying and Assessing the Risks of Material Misstatement (Revised July 2020), Paragraphs A208 to A214 ↩
This is a risk that is close to the upper end of the spectrum of inherent risk due to the degree to which inherent risk factors affect the combination of the likelihood of a misstatement occurring and the magnitude of the potential misstatement should that misstatement occur - ISA (UK) 315, Identifying and Assessing the Risks of Material Misstatement (Revised July 2020), Paragraph 12(1) ↩
ISA (UK) 200 (Revised June 2016) Overall Objectives of the Independent Auditor and the Conduct of an Audit in Accordance with International Standards on Auditing, paragraph 11 ↩
ISA (UK) 500, Audit Evidence, Paragraph A63 ↩
ISA (UK) 500, Audit Evidence Paragraph 7 ↩
ISA (UK) 330 (Revised July 2017) The Auditor's Responses to Assessed Risks, Paragraph 4 (b) ↩
AICPA Audit Sampling Guide, Table A-2 ↩
AICPA Audit Sampling Guide, Table 3-5 ↩
ISQM (UK) 1, Paragraph 48 and 49</blockquote="1"> ↩

File

Name Thematic Review Audit Sampling

Publication date 23 November 2023

Type Thematic review

Format PDF, 1.5 MB

Download original

Thematic Review Audit Sampling

File

Is this page useful?