The content on this page has been converted from PDF to HTML format using an artificial intelligence (AI) tool as part of our ongoing efforts to improve accessibility and usability of our publications. Note:

No human verification has been conducted of the converted content.
While we strive for accuracy errors or omissions may exist.
This content is provided for informational purposes only and should not be relied upon as a definitive or authoritative source.
For the official and verified version of the publication, refer to the original PDF document.

If you identify any inaccuracies or have concerns about the content, please contact us at [email protected].

Technical Actuarial Guidance: Models (October 2024)

The FRC's purpose is to serve the public interest by setting high standards of corporate governance, reporting and audit and by holding to account those responsible for delivering them. The FRC sets the UK Corporate Governance and Stewardship Codes and UK standards for accounting and actuarial work; monitors and takes action to promote the quality of corporate reporting; and operates independent enforcement arrangements for accountants and actuaries. As the competent authority for audit in the UK the FRC sets auditing and ethical standards and monitors and enforces audit quality.

The FRC does not accept any liability to any party for any loss, damage or costs howsoever arising, whether directly or indirectly, whether in contract, tort or otherwise from any action or decision taken (or not taken) as a result of any person relying on or otherwise using this document or arising from any omission from it.

© The Financial Reporting Council Limited 2024 The Financial Reporting Council Limited is a company limited by guarantee. Registered in England number 2486368. Registered Office: 8th Floor, 125 London Wall, London EC27 5AS.

1. Introduction

Purpose

1.1The FRC issues guidance for a number of purposes, for example to support compliance with requirements, or for interpretive, explanatory, contextual or educational purposes to support the use of judgement in applying principles-based standards. The overall purpose of the FRC's guidance is to improve the quality of technical actuarial work. The guidance is persuasive not prescriptive, and compliance is encouraged.

1.2Following our consultation on the revision of TAS 100 and associated stakeholder engagement, practitioners have expressed desire for additional guidance to be issued in relation to the Models Principle in TAS 100. Areas of clarification cited include where the technical actuarial work involves third-party proprietary models or modelling work is undertaken by other functions, how model governance (including validation and change control) activities are taken into account across key functions in an organisation and how TAS 100 applies where models use Artificial Intelligence (Al) or Machine Learning (ML) techniques.

1.3The purpose of this guidance is to help practitioners in complying with Principle 5 Models of 'TAS 100 (General Actuarial Standards)' which states:

Practitioners must ensure models used in their technical actuarial work are fit for purpose and subject to sufficient controls and testing, so that the intended user can rely on the resulting actuarial information.

1.4It comprises five provisions:

P5.1 Practitioners must ensure they understand the models used in their technical actuarial work, including intended uses and limitations.
P5.2 Practitioners must ensure that the models they use for technical actuarial work have in place an appropriate level of model governance.
P5.3 Practitioners must identify the extent of any material biases within the models that are used.
P5.4 Where material limitations exist in models or methodologies used, the practitioner must consider the implications of those material limitations.
P5.5 Where key stakeholders such as management, sponsors, trustees and regulators require the model to incorporate effects of material actions, practitioners must consider the implications of these actions.

1.5Our regulatory expectations on model use and bias are further set out in three application statements within TAS 100 (see Appendix 3).

Intended audience

1.6The guidance is aimed at practitioners who develop, use, validate and/or own models for technical actuarial work. Intended users of actuarial work may also find it useful.

2. Models in scope of TAS 100

2.1A successful application of Principle 5 requires practitioners to determine which models fall within the scope of TAS 100. This section guides practitioners in determining this. Appendix 2 provides illustrative examples to further aid understanding.

Model definition

2.2The TAS 100 glossary defines a model as follows:

A simplified representation of some aspect of the world. The model produces a set of outputs from inputs in the form of data, assumptions and parameters. Inputs and outputs may be qualitative or quantitative. The model is defined by a specification that describes the matters that should be represented, the inputs, and the relationships between the inputs, and the resulting outputs. The model is implemented through a set of mathematical formulae and algorithms (e.g., a computer program).

2.3Other model definitions are in use, notably the following in the PRA's 'SS1/23 - Model risk management principles for banks':

'A model is a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into output. The definition of a model includes input data that are quantitative and/or qualitative in nature or expert judgement-based, and output that are quantitative or qualitative.'

2.4These definitions have in common three elements: 1) inputs; 2) outputs; and 3) a quantitative approach to processing the inputs into outputs.

Scope - models used in technical actuarial work

2.5Within any given entity (as defined in TAS 100) there may be a very large number of models, but the scope of Principle 5 is models used in technical actuarial work¹. Principle 5 therefore applies to all models (as defined in paragraph 2.2) used to carry out technical actuarial work, regardless of whether the models themselves use principles or techniques of actuarial science and judgement is exercised in the model.

2.6Practitioners may apply proportionality (see Section 7 and Appendix 1) when considering the efforts to be expended in their compliance with Principle 5 in respect of each model.

Scope - indirect use of models

2.7Practitioner A carrying out technical actuarial work may use information provided by a second practitioner (B) as input to practitioner A's model. Practitioner B may have used a model (model B) to generate the information which is provided to practitioner A. We refer to this as 'indirect' use of practitioner B's model by practitioner A.

2.8Practitioner A's TAS 100 responsibility starts once they use the information from practitioner B's model in their technical actuarial work:

If the information received from practitioner B is used as a data input in practitioner A's model, then TAS 100 Principle 3 applies i.e. practitioner A needs to ensure that the data is accurate, complete and appropriate.
If the information received from practitioner B is used as an assumption in practitioner A's model, then TAS 100 Principle 4 applies, and practitioner A needs to ensure that the assumption is appropriate.

2.9Practitioner A does not bear responsibility for the compliance with Principle 5 of practitioner B's model. Compliance responsibility in this regard rests with practitioner B, if their work to generate the information for practitioner A meets the definition of technical actuarial work.

Scope - third party model or code

2.10It is common for practitioners to use a model developed by a third party for technical actuarial work. Also, practitioners may be involved in the development of models for use by a third party. The third party may be within the same entity or an external entity.

2.11From the perspective of a practitioner A providing the third-party model, if their work meets the technical actuarial work definition, then practitioner A must comply with TAS 100.

2.12From the perspective of a practitioner B using the third-party model, when the practitioner uses the third-party model for their technical actuarial work to provide output to their intended users, then practitioner B must comply with TAS 100 in respect of their technical actuarial work for their intended users, and the third-party model comes within the scope of Principle 5. Practitioner B must therefore ensure the model used is fit for purpose and subject to sufficient controls and testing. They may exercise judgement on the extent to which reliance may be placed on practitioner A's TAS 100 compliance and evidence, where available, when complying with Principle 5.

2.13There may be situations where practitioners utilise open-source code. This is code that is designed by a third party to be publicly accessible. Where practitioners use such code for their technical actuarial work, the code falls within the scope of Principle 5 and the responsibility for compliance rests with the practitioner. Practitioners may wish to consider section 7 of this guidance in considering proportionality in complying with Principle 5 for open-source code.

Scope - artificial intelligence and machine learning

2.14We consider models which use Al and ML techniques to fall within the scope of TAS 100 Principle 5 if they meet the model definition and are used in technical actuarial work as set out above.

3. Model understanding

3.1TAS 100 P5.1 requires practitioners to ensure they understand the models used in their technical actuarial work, including intended uses and limitations. We cover model limitations in detail in Section 5, so cover them only briefly in this section.

3.2An understanding of a model's use is essential to avoid its use in ways that are inconsistent with the original intent or capabilities without consideration of the risks of doing so.

3.3Practitioners may wish to understand the use(s) of the model intended by the model design. This includes understanding:

The purpose (e.g., regulatory capital, pricing, valuation, assumption setting, setting funding and investment strategy for scheme journey-planning).
The use by country, legal entity, product/liability/asset types, and customers/clients.
The environment in which a model performs reliably, for example, economic and market circumstances (including whether the model has sufficient regard to extreme events or outliers per TAS 100 A5.1).
The level of knowledge and understanding of potential users of the model.

3.4Practitioners using models in their technical actuarial work may wish to have a good understanding of the operation of the model, including its user controls (to mitigate misuse risk), the methodology underpinning the model, the model's intended use and the model's limitations.

3.5Practitioners may also wish to understand their responsibilities and those of others in relation to the models. In addition, practitioners may wish to understand the material judgements within the model, the model outputs, and the model's governance framework.

3.6Beyond this, the level and type of model understanding required may depend on the nature of the technical actuarial work. For example:

A practitioner whose technical actuarial work is model development (responsible for designing, developing, testing and documenting a model) may need to understand the detailed specification and the source code.
A practitioner whose technical actuarial work is model validation (responsible for validating that a model performs as expected) may similarly need to understand the detailed specification but also the design and effectiveness of the model control framework.

3.7An understanding of models can be gained directly or indirectly. The direct approach is where the practitioner can review the code underlying the model and the resulting parameters and use this to gain sufficient understanding. The indirect approach is where the practitioner applies other tools or techniques to gain this level of understanding of the original model, as the level of complexity is too high to allow the direct approach. Where AI/ML techniques are used, gaining an understanding of the model could be more challenging, especially where the inner workings cannot be explained in a way that can be easily understood or accessed.

3.8The mechanism of achieving this understanding may need to vary if established techniques for directly understanding the model are not available. For example, practitioners may wish to perform analysis to help explain how a model has generated a particular result to aid understanding of a model. Such analysis is itself typically subject to limitations, and practitioners may wish to consider the impact of these limitations on their findings (this is discussed further in Appendix 2 example 7). If, after such analysis to explain the model, a practitioner is still unable to achieve an understanding of the model due to the AI/ML techniques used, then they may be unable to comply with TAS 100 P5.1 and may-wish to reconsider whether the model is still fit for purpose as set out in TAS 100 P5.

3.9Practitioners may exercise judgement in deciding the necessary level of understanding to satisfy themselves that the model they are using in their technical actuarial work is fit for purpose. A greater understanding may be needed the more significant a model is in terms of its potential impact on the decisions made by the intended users, and/or the higher the degree of model complexity or judgement within the model.

3.10Where a practitioner uses models indirectly, by virtue of receiving output/information from those models for use in their own models, then the practitioner is the intended user of that model output. In these circumstances it may be important for the practitioner to understand the model governance of those models, the material judgements in the models and their limitations, but the practitioner may judge that an understanding beyond this is not necessary.

4. Model risk and model governance

4.1The following sections describe the sources of model risk, and how model governance can mitigate this risk, in order to assist practitioners in determining an appropriate level of model governance for their models.

Model risk

4.2The TAS 100 glossary defines model risk as:

The risk that models are either incorrectly implemented (with errors) or make use of assumptions that cannot be justified rigorously, or assumptions that do not hold true in a particular context.

4.3Some examples of common areas of model risk are set out below:

Data: inaccurate, incomplete and/or inappropriate.
Assumptions: incorrect and/or inappropriate.
Methodology: unsuitable model design choices (e.g., features captured/omitted, use of proxies, choice of statistical method, non-compliance).
Specification: inaccurate specification, or specification not reflective of the intended methodology.
Coding: source code incorrect and not reflective of the intended specification, and/or coding design sub-optimal and detrimental to performance.
Misuse: model inputs entered incorrectly and/or model outputs extracted incorrectly.
Misapplication: model used in circumstances not intended (e.g., for products, entities/structures, geographies, markets or time horizons not intended, or in economic circumstances not envisaged, or for pension increase caps and collars not designed).
People: insufficient skills or knowledge or resources to implement and/or operate the model.

4.4Additionally, the IT infrastructure may give rise to model risk if it is not stable and/or at risk of becoming obsolete (and unsupported by the IT function or external vendor) and/or not aligned to the IT strategy.

4.5Model risk may increase with the volume of data inputs, the number of and uncertainty in underlying assumptions, the complexity of the model design, and the materiality of the model. Model risk may also increase when assessed in aggregate as interactions/dependencies between models or reliance on common inputs or methodologies or code may adversely impact several models at the same time.

Model governance

4.6TAS 100 P5.2 requires practitioners to ensure that the models they use for technical actuarial work have in place an appropriate level of model governance. Model governance is defined in the TAS 100 glossary as:

A set of activities, policies and procedures for identifying, managing and mitigating model risks. Actions to mitigate model risks include clear model ownership and responsibilities, documentation, model validation, a change control process including for example, appropriate checks to ensure the stability of model outputs.

4.7The term model governance is sometimes used to refer only to model roles, responsibilities, and approval and oversight mechanisms (e.g., committees/forums). For TAS 100, a broader definition applies, including additionally model risk identification and management, and model risk mitigants such as model documentation, model change controls, and model validation.

4.8TAS 100 P5.2 requires practitioners to ensure there is an appropriate level of model governance in place for their models. This does not mean that practitioners are required to be responsible for managing or overseeing all model governance activities, policies, and procedures. To ensure there is an appropriate level of model governance in place, it could be sufficient for practitioners to be sighted of, and satisfied with, the model governance for their models, including the model governance for models provided by third parties.

4.9In this section we describe key elements of model governance. However, what is 'appropriate' may have regard to proportionality and we are expecting different levels of model governance across different models. Additional levels of governance may be needed where models use Al/ML techniques and this is discussed further in example 8. The concept of proportionality, as applied to all TAS 100 provisions and regulatory expectations, is covered in 'TAS 100 Guidance – Proportionality'. Proportionality, as applied to Principle 5 in particular, is covered in Section 7 and Appendix 1 of this guidance.

Model risk identification and management

4.10Risk identification refers to the process for detecting and assessing sources of potential model risk. Paragraph 4.3 of this guidance gives some examples of common areas of model risk, but it is important for practitioners to consider all potential sources. Having a standardised model risk taxonomy (ideally across entity as defined in TAS 100) may assist in the consistent identification and classification of model risks across all models.

4.11An assessment of the identified risks involves analysis and measurement. It is preferable for model risks to be assessed consistently across models (again, ideally across entity and at group level where applicable) and this may be facilitated by setting out possible measurement approaches alongside a model risk taxonomy. For example:

How recently the model was reviewed without reliance on previous assessment (also known as baselining the model).
The number and magnitude of out-of-model adjustments² (OOMAs), to address model limitations, by category (e.g., data, assumptions, methodology, misuse, misapplication).
The results of backtesting the model by using historical data and comparing the output to past results.
The results of sensitivity analysis, scenario testing and testing of stability to reveal the material assumptions and the reliable range for the model. Testing of stability is discussed further in example 9 of Appendix 2.
The number of run failures, model outages and the trend in runtimes.
The number of model risk incidents recorded (e.g., specification/coding errors identified post implementation, incorrect outputs provided to intended user, policy/standards/regulatory compliance breaches).

4.12Risk identification processes reveal the exposure to model risk. Model risk management is then about managing the identified model risk exposures. To aid this, it is helpful to define and agree how much model risk exposure is acceptable for the entities or models in the context of the intended user and intended use of the model, perhaps as part of a wider risk management framework.

4.13A common way of setting boundaries is to establish the model risk appetite and tolerances, and associated triggers. Appetite is frequently expressed as a high-level qualitative statement of the attitude to risk (e.g., 'no appetite' or 'limited appetite' or 'some appetite'). These are then implemented through risk tolerances and triggers, using quantitative measures where possible, aligned to the model risk taxonomy and measurement approaches outlined in paragraphs 4.10 and 4.11 above.

4.14The successful identification of model risks and the monitoring of their exposures against risk tolerances and appetite are key elements of model risk management. Practitioners may wish to consider these elements for their models when assessing an 'appropriate level of model governance', allowing for proportionality which is further discussed in Section 7 of this guidance.

Model documentation

4.15TAS 100 sets out the requirements for documentation of technical actuarial work in general, requiring documentation on: judgements; data; assumptions; model use, limitations and how it is fit for purpose; model governance and controls/testing; and material modelled actions.

4.16Good documentation is a mitigant for model risk and may include the following:

Policies and procedures, which establish minimum requirements and actions in areas such as model development and testing, data quality, risk identification, monitoring and reporting, model change and model validation.
Model documentation e.g., model inventory and/or documentation inventory, model methodology and technical specification, user guide (including the purpose and use cases of the model), model success criteria and results.
A control framework which sets out the risks in the design and operation of the model(s) and the controls in place to mitigate (e.g., risk of user input error, do/review control for all inputs into the model(s) to mitigate risk).
Management information on the ongoing adequacy of the model(s) (e.g., performance against model success criteria, risk measurements and tolerances/triggers, findings from model validation and remediation status).
Where a model updates itself automatically for new training data, it may be necessary to consider the frequency with which documentation should be updated. This may include further information on how the model is developing over time, any drift in model accuracy or model bias and how any model updates may affect previous analysis carried out to explain the model.

4.17Practitioners will wish to apply judgement when considering the necessary documentation:

For a material model used to generate information for published communications, a comprehensive suite of documentation may be appropriate.
For a simple model or an immaterial model informing local, internal decisions, then reduced documentation may be more suitable. For example, commentary within the model itself summarising the model purpose and use, roles, description, limitations, version history and operating instructions, together with an articulation of the model controls (either within the model itself or set out outside the model).
Where documentation includes model limitations and range of circumstances in which it can be used as part of the operating instructions, practitioners may wish to consider the needs of the model users and whether to check understanding of this documentation with users.

Model change controls

4.18TAS 100 defines a change control process as:

A process that: '(i) only allows authorised changes to the model; (ii) documents any changes made, testing carried out, and any material impact on the model or its outputs; and (iii) allows any changes to be reversed.

4.19Practitioners may exercise judgement when considering the adequacy of the model change process, including the appropriate level of authorisation for changes, considering both the materiality of the model and the materiality of the changes.

4.20For material models, it may be appropriate to have a trigger framework (setting out triggers for review/change and materiality thresholds for change approvals) and regular monitoring and reporting of those triggers. A material change to a material model may require a formal validation of the change together with a formal governance process.

4.21Where the materiality threshold does not bite, either individually or cumulatively for a change, then it may be acceptable for the model owner (who is accountable for the model) to simply authorise the change subject to being satisfied that relevant controls have been applied (e.g., documentation and review of changes).

4.22TAS 100 defines materiality in the context of influencing significant or relevant decisions. For model changes, it is helpful to have specific materiality criteria linked to the nature of the model, for example, based on funding level, investment value at risk, capital requirements, solvency, profit metrics, asset and/or liability and/or net asset valuations, or price as relevant. Such criteria are helpful in both determining triggers for change and assessing the materiality of the change in model outputs.

Validation

4.23The TAS 100 defines model validation as:

The processes and actions verifying that a model is performing as expected and is fit for purpose.

4.24The aim of validation is to provide an unbiased opinion on the adequacy of a model. An unbiased view of a model is best achieved through independence between the developers and validators. This can take many forms: from use of another organisation to perform the validation to having the validation performed by another team/individual within the organisation in a separate division/unit/reporting line. The greater the separation the greater the independence.

4.25In larger entities there may typically be 'three lines of defence' with a second line risk function overseeing first line and a third line internal audit function overseeing the first and second lines. In these circumstances, validation responsibilities often reside with the second line. In smaller entities there may not be the opportunity for a three-lines model and validation may be carried out within the same team.

4.26The following processes are commonly used to verify that the model is performing as expected:

Assessing the design and operational effectiveness of the model controls, including the application of the model governance framework.
Assessing the quality of data inputs. This may need additional consideration if AI/ML techniques were used in generating that data.
Reviewing the completeness and quality of the model documentation.
Assessing the model methodology for compliance against relevant regulations.
Evaluating the appropriateness of the methodology by comparison with alternative methodologies.
Evaluating the appropriateness of the assumptions and methodology by backtesting the model and benchmarking.
Assessing the stability and limitations of the model through sensitivity testing and scenario analysis, including extreme event scenarios where appropriate.
Evaluating the performance of the model, which may include testing performance of models using AI/ML techniques on a testing data set that is separate from the data used to train the model, and testing for model convergence where relevant.
Assessing the accuracy of the model by reproducing output in part or in full using an alternative model.

4.27Validation is an important process for confirming that a model is performing as expected, and for identifying model limitations and interdependencies with other models. In particular, sensitivity testing and scenario analysis assist in understanding a model's reliable range with regards to economic, market and demographic circumstances.

4.28Validation checks can also be performed on model outputs to check for bias between different sub-groups within the population considered. This could be particularly relevant for models incorporating AI/ML techniques, where more complex techniques and larger data sets used in predictive modelling could mean that the relationships between inputs and outputs and the decision-making methods are obscured. This in turn means that identifying bias could be more difficult (this is discussed further in Appendix 2 example 6).

4.29The choice of the sub-groups to investigate will be a matter of judgement for the practitioner and could be determined by the legal, regulatory and ethical requirements of the intended user of the model output.

5. Model limitations and bias

Model limitations

5.1TAS 100 P5.1 and P5.4 require practitioners to understand model limitations and consider the implications of material limitations.

5.2A model limitation may be regarded as a known issue with a model, as compared with a model risk which is a potential issue i.e. a chance of something happening that may have an adverse impact.

5.3Models used in technical actuarial work cannot typically be designed to fully capture the impact of all elements that may affect an actual outcome. Limitations therefore arise from approximations inherent in the model design. Limitations are also a consequence of assumptions underlying a model that may limit the scope of application to a specific set of circumstances.

5.4Model design approximations frequently arise as a result of trade-offs between model accuracy, complexity, efficiency and cost. Some examples of common limitations are as follows:

Classes of assets or liabilities, or features of classes, may not be captured in a model.
Data model points may be used rather than a full data set, for example for capacity reasons and runtimes.
Asset and liability proxy models may be deployed rather than full valuation models.
Assumptions may be calibrated to proxy data and/or by expert judgement and/or to data that does not adequately capture future uncertainty.
Some risks may not be captured in the modelling.
Models may understate the likelihood of extreme events if historical training data does not contain sufficient outliers in the experience.
Limitations in stochastic modelling from the choice of statistical approach or scenario generator or the number of simulations.
Explainability and communication may be more challenging if the practitioner chooses to use models which are inherently less understandable.

5.5Limitations may be identified as a consequence of design decisions in the development phase of a model. Validation may further highlight limitations using validation techniques such as backtesting, sensitivity and scenario testing, and comparisons with alternative models or methodologies.

5.6It may also be the case that limitations emerge as a model is used over time, if the environment or portfolios change, or as errors are identified, or if a model is used for a different purpose than its original intent. It is important that all limitations are recorded, whatever the source.

5.7Practitioners may wish to quantify a limitation through re-running or estimation, and the impact may then be compared with any triggers and materiality thresholds established as part of any model change process.

5.8However, it is not always straightforward to assess the materiality of a limitation without considerable effort especially, for example, where the limitation relates to unmodelled features of assets or liabilities. In these cases, a practitioner may exercise judgement as to whether a qualitative assessment can be undertaken based on an understanding of the risk factors underpinning the limitation.

5.9For limitations assessed as material, when considering the implications as required by TAS 100 P5.4, practitioners may wish to consider an appropriate course of action to mitigate the limitation. For example:

Scaling data inputs for incomplete data.
Re-fitting proxy models where the fit is inadequate.
Re-calibrating assumptions where the calibration data and/or economic or market or demographic circumstances have changed.
Applying a different methodology, for example, choice of statistical approach.
Building new features into the model.
Applying an out-of-model adjustment³ (OOMA) to address limitations.
Restricting the use of the model in circumstances where the limitations bite.
Performing analysis to help explain how the model has generated a particular result to aid understanding.
Abandoning the model.

5.10The choice of action may have regard to the nature of the limitation. It may also have regard to an aggregate view of the limitations and the associated actions. For example, applying an OOMA may be reasonable to address a limitation in a model when considered in isolation, but potentially not if the OOMAs for that model are already significant, in number and/or magnitude, in which case model change(s) or restricting the model's use may need to be considered.

5.11Practitioners may wish to consider whether the assessment of materiality and the determination of an appropriate course of action for a limitation is a material judgement. In which case, in compliance with TAS 100 P2.3, the judgement should be reviewed to ensure that it remains appropriate over time.

5.12More generally, practitioners may wish to consider reviewing the materiality assessments periodically to ensure they remain accurate, particularly where the exposure to the limitations changes. For example, a limitation in respect of an unmodelled feature may be assessed as being of low materiality, but the projected or actual exposure may grow (e.g., as a result of sales or a change in investment strategy or economic movements) and modelling the feature may need to be re-considered.

Model bias

5.13TAS 100 P5.3 requires practitioners to 'identify the extent of any material biases within the models that are used', where bias is defined in TAS 100 as a 'disproportionate weight in favour of or against something'. In accordance with the application statements (TAS 100 A5.2 and A5.3), practitioners are expected to:

Consider whether the model leads to consistent over or under estimation.
Consider whether the model contains systematic error leading to results that are not representative of the intended design.
Improve the model to reduce the impact of material bias.

5.14Bias is a type of model limitation and may therefore be considered as part of the wider consideration of limitations as outlined above, including adjusting for material bias.

5.15Some common sources of model bias are as follows:

Data – data bias may arise if the methods of sampling data have inherent bias and/or if the data itself reflects historical bias (including bias due to over-representation of certain groups) and/or if attributes that may help provide an unbiased prediction of matters being modelled are omitted from the data.
Algorithm - the design of the model's algorithm may cause model bias, for example, if an algorithm is written in a way that erroneously gives more weight to some data than other, or to exclude variables that might otherwise improve unbiased estimation, or to systematically round imputed values such that it introduces rounding error into the model output.
Assumptions – the data used to calibrate model assumptions may be biased per above, or judgement may be applied to the calibration and this judgement may be biased, for example, as a result of cognitive bias.
Outputs - bias may arise as a consequence of the way in which model output is designed, perhaps as a consequence of the designer downplaying or overemphasising results seen as less or more desirable respectively.

5.16Practitioners may typically be familiar with how to evaluate the bias of methods and models through comparisons of estimated values versus actual values and statistical metrics (e.g., Chi-squared test). Practitioners may also wish to consider the possibility of bias across each element of their models, for example, by comparing model outputs using different data sets or methods or assumptions, or examining different sub-groups of the population. This may include consideration of the ethical and reputational impact of generating biased output.

5.17As set out in TAS 100 A5.3, where material biases are identified, practitioners should seek to address them. Data bias may be mitigated by eliminating problematic data or removing/adding specific components of data. Potentially biased judgements made in the algorithmic design or assumption calibration may be mitigated through seeking a broader range of views. An adjustment may also be made to model outputs to remove bias.

5.18Models using Al/ML techniques could be complex and could have self-learning capabilities (they recognise patterns in the data, learn from them and become better at connecting inputs to outputs over time, which allows the model to change while in use). As with any models, such models are at risk of bias, and the use of potentially complex and opaque models makes the risk of algorithm and data bias especially significant.

5.19The data for models using Al/ML techniques may be more complex in terms of volume, variety and velocity⁴. To scrutinise the data appropriately, practitioners may wish to consider what historical biases may be contained within the data. The practitioner may wish to determine whether the data is relevant to the aim of assessing the predicted values accurately and document a causal explanation of how the data may be expected to impact predicted values.

5.20In addition to checking the data for biases and, where appropriate, considering whether the model is set up in a way that will perpetuate biases, there may be a need to check for bias in model outputs. For example:

Practitioners may choose to investigate output from a testing data set split by sub-groups. This would check for bias where the model may be over predicting one sub-group balanced by under predicting another sub-group.
Practitioners may also consider investigating disparate model outputs by sub-groups which could lead to disparate outcomes for individuals or groups of individuals. Where appropriate this would allow the practitioner to highlight disparate outcomes and provide information to the intended user that supports the intended user's decisions.

5.21This is discussed further in Appendix 2 example 6.

6. Modelled actions

6.1TAS 100 P5.5 requires that 'where key stakeholders such as management, sponsors, trustees and regulators require the model to incorporate effects of material actions, practitioners must consider the implications of these actions.'

6.2Examples of such actions include:

Discretionary pension increases.
With-profits management actions.
Future charges/fees.
Terms for member/customer options.
Cost management.
Discretionary pay.
Future employment/payroll levels.
Dividends.
Capital-raising.
Re-structuring and risk mitigation activities.
Investment strategy (e.g., modelling a scheme journey plan allowing for rebalancing of the investment portfolio following a period of under or outperformance).

6.3To incorporate the effects of actions, practitioners may wish to understand:

The nature of the actions being modelled and the timeframe over which they are assumed to be implemented.
The context and why the actions are being incorporated; in particular, whether stakeholders intend to implement the actions and under what circumstances would these actions be implemented (e.g., normal or worst/best case).
How and where the model outputs are being used.

6.4Practitioners may also wish to be satisfied that consideration has been given or is being given to:

How realistic/viable the actions are (e.g., if there is an action to manage costs through outsourcing, whether there are third parties able to provide the administration and at what cost).
Any legal implications arising from the actions (e.g., if modelling actions to change customer terms in specified circumstances).
Whether the actions are consistent with any applicable regulations.
Whether the actions are in line with customer/member contract terms and conditions or Scheme rules.
Whether the actions are aligned with the entity's internal policies and standards.
Any new risks arising or any existing risks becoming more/less material as a consequence of the actions.
The procedures for implementing the actions.

6.5Where the regulatory rules are prescriptive in the area of assumptions about future management actions in regulatory valuations, it may be appropriate for practitioners to familiarise themselves with those rules.

6.6TAS 100 P5.5 requires practitioners to consider the implications of material actions. The considerations above should be commensurate with the materiality, and the implications of material modelled actions are expected to be described in communications to intended users as set out in A7.6 e) of TAS 100.

7. Proportionality as applied to Principle 5

7.1As set out in TAS 100 paragraph 1.5, practitioners are encouraged to have regard to the guidance on proportionality to inform how they will comply with TAS 100. This includes TAS 100 Principle 5.

7.2The 'TAS 100 Guidance – Proportionality' states that 'the requirements of the TAS 100 should be met in a way that is proportionate to the nature, scale and complexity of the decision or assignment to which the technical actuarial work relates and the benefit that the intended user would be expected to obtain from the work'.

7.3Practitioners may wish to consider a proportionate approach to complying with Principle 5 based on a risk assessment of the model. Such an approach may build on the risk identification processes described in Section 4 of this guidance, considering the model risks, the materiality of the model and the extent/nature of its use.

7.4It is good practice for entities to establish an entity-wide approach to categorising models by risk rating and to periodically assess models according to this categorisation. This may then be used to inform the application of TAS 100 Principle 5. For example, a reasonable outcome from this approach may be a risk-based sliding scale of model governance. We provide an illustration of this as an example in Appendix 1 overleaf.

8. Appendix 1: Risk-based model governance illustration

	Very low risk model	Very high risk model
Model risk framework	Entity framework – identification and reporting (taxonomy, measurement basis, appetite, tolerances, triggers)	Entity framework – identification and reporting (taxonomy, measurement basis, appetite, tolerances, triggers)
Policies	Entity policies – model development/testing, data quality, model change, validation	Entity policies – model development/testing, data quality, model change, validation
Inventory	Model inventory – model reference, owner, purpose, risk assessment	Model inventory – model reference, owner, purpose, risk assessment
Documentation	* Model purpose/use/roles/ version history/limitations Description of methodology Instructions	* Model purpose/use/roles/ version history/limitations Methodology documentation Technical/functional specification * User guide
Defined model success criteria	No	Yes, aligned with nature of model (e.g., could be in relation to model fit and/or model risk measurements)
Controls	Checklist of actions with doer/ reviewer	* Documented control framework (risks/controls) * Evidence of completion of controls
Change process	Model owner authorisation subject to the changes being recorded and tested	* Trigger framework Material changes validated Formal approval of changes and validation
Validation	Review by model owner	* Maximum independence (e.g., through 2^nd line) Full suite of validation tests Results reported to senior management
Monitoring	Model risk incidents only	Yes - model success criteria, model risk measurements, model change triggers

9. Appendix 2: Scenarios

Scope – indirect models

Example 1

9.1A practitioner is setting expense assumptions to recommend to the senior management of an insurance company. The activity involves projecting costs, allowing for policyholder demographics and economic assumptions, and exercising judgement in the demographic and economic assumptions as well as in other areas (e.g., the allowance for new business).

9.2The activity of expense assumption-setting is technical actuarial work, and the projection model is used directly in the work. The practitioner must therefore comply with TAS 100, and the projection model is within the scope of Principle 5.

9.3A key input to the projection model is the output from a cost allocation model, which allocates all of the entity's costs by type (e.g., initial, renewal, fixed, variable, recurring/non-recurring) and line of business. The practitioner is the intended user of the output.

9.4The cost allocation model is used indirectly by the practitioner in their technical actuarial work through use of the model output. The practitioner is not therefore responsible for Principle 5 compliance of the cost allocation model (unless the practitioner is also responsible for the cost allocation activity, and it constitutes technical actuarial work). The practitioner does, however, have to comply with Principle 3 Data of TAS 100 in respect of the allocated cost inputs.

Example 2

9.5A practitioner (A) is performing a funding valuation for a pension scheme. This involves using a valuation model to project member benefits allowing for demographic and other assumptions.

9.6The activity of performing a funding valuation is technical actuarial work, and the valuation model is used directly in the work. Practitioner A must therefore comply with TAS 100, and the valuation model is within the scope of Principle 5.

9.7A key input to the valuation model is the mortality assumption. This is provided by another practitioner (B) within the same entity who has used a model to analyse the scheme mortality experience and fitted it to (a variation of) a CMI model.

9.8Practitioner A is therefore the intended user of the mortality assumption produced by practitioner B. Practitioner A's TAS 100 Principle 5 compliance responsibility is in respect of the funding valuation model only, not the mortality model for which practitioner B has TAS 100 Principle 5 compliance responsibility. Practitioner A does, however, have to comply with Principle 4 Assumptions in respect of the mortality assumption inputs.

Scope – third-party models

Example 3

9.9A practitioner develops an Economic Scenario Generator (ESG) for a client for use in modelling guarantees. The contractual arrangement is such that the client is permitted to place reliance on the model, but the practitioner has not been engaged to provide advice or assurance in relation to the client's subsequent use of the model.

9.10The practitioner's work meets the technical actuarial work definition and the actuarial information provided to the client is the model (and associated documentation). As such, the practitioner is obligated to comply with TAS 100 in respect of the model, including Principle 5, and should be prepared to provide evidence of compliance to the client on request. The subsequent use of the model and its output by the client is not, however, within the scope of the practitioner's technical actuarial work.

9.11The client runs the ESG model to generate scenarios for a pricing model to cost the guarantees for a potential new product in order to make a recommendation to the Pricing Committee.

9.12The client's pricing work meets the technical actuarial work definition. The client therefore has a separate TAS 100 obligation to the intended users, the Pricing Committee. The client is using the ESG model in this work and so the ESG model comes within the scope of Principle 5. The client may exercise judgement on the extent to which reliance can be placed on the TAS 100 compliance by the practitioner providing the ESG model.

Example 4

9.13A firm providing actuarial consulting services to defined benefit pension schemes develops a tool for the use of trustees, corporate sponsors and investment managers of UK schemes. The output from the tool is intended to facilitate discussions between the trustees, sponsors and investment managers and support decisions on asset allocations and contribution levels.

9.14The tool allows users (typically with no actuarial expertise) to project the scheme funding position under various scenarios, by varying key inputs such as asset and liability data, asset allocations, contribution levels, asset returns, inflation and liability discount rates. The tool is intended to be used autonomously without ongoing support from the consulting firm and the setting of the inputs is at the discretion of the users of the tool and not the consulting firm.

9.15The development of the tool constitutes technical actuarial work and the trustees, sponsors and investment managers who will purchase it are the intended users as it is their decisions that the tool is aiming to assist. The practitioner responsible for the development of the tool at the consulting firm is therefore obligated to comply with TAS 100, including Principle 5 in respect of the tool.

9.16Given the direct use of the model by a wide group of users, the practitioner may wish to consider, in particular, the documentation and communication of the intended uses of the model and any material limitations (TAS 100 P6.1 and A7.6).

9.17The practitioner may also wish to pay close consideration to the communication of the scope and purpose of their technical actuarial work (TAS 100 P7.4), notably that the setting of inputs by the intended users is outside of their scope but that the output from the tool, given the inputs, is within scope and intended users can place reliance on the tool and its outputs for any valid set of input assumptions.

Model risk and model governance

Example 5

9.18A practitioner at an insurance company has worked with a third-party consultancy firm and internal stakeholders to develop a stress testing model. The model will be used by the company to analyse the impact on solvency and profit of a range of economic, expense and demographic stresses.

9.19The practitioner and the company's Chief Finance Officer (CFO) have agreed that the practitioner will be the model owner. The practitioner's team will operate the model. The output from the model will be used in actuarial information provided by the practitioner for business planning, hedging, risk management, and reporting. The development project is sponsored by the CFO, who has executive responsibility for risks arising from models owned within the Finance division.

9.20The practitioner considers the model will be used to carry out technical actuarial work by virtue of the techniques used in the calculations, the judgement exercised and the actuarial information which will be communicated. As such, the practitioner will be required to comply with Principle 5 of TAS 100 when using the model in the work.

9.21The practitioner is considering how to ensure that the model has in place an appropriate level of model governance as required by TAS 100 P5.2. To assist in determining what is appropriate, the practitioner undertakes a model risk identification exercise on the model. The practitioner concludes that the model risk exposure is high. This is due to the large number of inputs required, the complexity of its design, its wide usage in important areas and the need for validation by use including statutory entity considerations within the insurance group of companies. If the model is inaccurate or misused, it will result in unexpected solvency/profit movements and/or reputational damage. The practitioner decides that strong model governance is necessary.

9.22The practitioner assesses the quality of the model documentation to be very good but considers there to be gaps in other areas of model governance such as a documented change and authorisations process and independent validation of the model.

9.23The practitioner considers that this is a matter for the CFO's attention, both as sponsor of the model development and the executive responsible for risks arising from models owned within the Finance division. The practitioner therefore makes a recommendation to the CFO, setting out the results of the model risk assessment, the current gaps in model governance and the actions needed to close the gaps in order to facilitate compliance with TAS 100 P5.2 when the model is used in future technical actuarial work.

Model bias

Example 6

9.24A practitioner is carrying out technical actuarial work to support the pricing framework for motor insurance where the intended user of the work is a governance forum. The work involves a modelling exercise to estimate claims using a new source of data. The role of the governance forum is to decide whether the pricing framework, including models and assumptions is appropriate for the insurer's live environment. The Senior Manager under the FCA's SM&CR⁵ who is responsible for pricing for the insurer sits on the governance forum.

9.25The practitioner wants to identify and mitigate against unacceptable levels of data and model bias. They are aware that the intended user will need to consider requirements of the Equality Act 2010 to not discriminate against protected characteristics as well as obligations from the FCA Consumer Duty to ensure good outcomes for retail customers. The practitioner plans the approach to reviewing the model for potential bias and what metrics will be used for consideration, taking into account the insurer-wide approach to bias. When developing the model, the practitioner considers what outputs may be needed to test for potential bias.

9.26The practitioner uses subject matter expertise and judgement to determine whether the data is relevant to the aim of assessing risk accurately. The practitioner decides that in this case it would be appropriate to document a causal explanation of how the data may be expected to impact insurance claims to justify the data items being included in the model. The practitioner also scrutinises what historical biases may be contained within the data. For example, over-representation of certain groups in an input data set could lead to bias.

9.27The practitioner excludes certain input factors where they judge that direct and/or indirect discrimination would occur. The practitioner considers that bias could influence the choices surrounding the manipulation of input data which could lead to biased model output. In spite of identifying and mitigating for data bias, the model may still lead to disparate outcomes for groups of policyholders due to interactions between factors.

9.28Having chosen a proposed model, and as part of model validation, the practitioner carries out analyses to investigate bias in the model output. This includes running the proposed model on an independent sample (testing set) which is separate from the sample used in the original training of the model (training set) and investigating the output split by sub-groups. The practitioner uses their judgement to choose the sub-groups, considering any relevant legal, regulatory and ethical requirements. Where data is not sufficient to classify policyholders between sub-groups, the practitioner considers if a suitable proxy could be used. The practitioner tabulates metrics to investigate discrepancies for the sub-groups as below.

Metric	Training set Sub-group 1	Training set Sub-group 2	Training set Total	Testing set Sub-group 1	Testing set Sub-group 2	Testing set Total
Exposure
Claims cost:
• Actual
• Predicted
• Actual vs predicted

9.29The practitioner considers any discrepancies looking at the following comparisons:

The practitioner looks at the actual versus predicted claims costs across the sub-groups. The overall accuracy is good for the testing data set, but the accuracy by sub-group for some factors is unbalanced. The practitioner determines the actions to remedy the unbalanced results for each factor. The practitioner makes a judgement on the materiality of the discrepancy and considers options which include accepting the difference, applying an out of model adjustment or re-running the model taking out some factors and/or altering the specification of the model.
The practitioner considers where the model may lead to disparate outcomes by looking at the average claims cost by sub-group. The practitioner looks at sub-groups where there may be legal, regulatory or ethical considerations in order to provide information to the governance forum. The practitioner investigates the other factors in the model that contribute the most to the different average claim cost by sub-group in order to be able to communicate issues to the governance forum.

9.30There could be a trade-off between overall accuracy and eliminating bias. The practitioner documents the considerations and judgements around bias so that the decisions concerning bias are recorded.

9.31The role of the governance forum is to decide whether the pricing framework, including models and assumptions is appropriate for the insurer's live environment. The practitioner provides a report to the governance forum that is clear, comprehensive and comprehensible to allow the governance forum to make informed decisions, including in relation to bias. The report includes a recommendation on the model using the new source of data; the practitioner details the factors in the data that they propose should be used and how they have addressed the issue of bias in terms of the accuracy of the model split by sub-groups. In addition, the report includes information on where there are potential disparate outcomes for a selection of sub-groups of policyholders to allow the governance forum to assess the appropriateness of the differences in line with the legal, regulatory and ethical requirements of the insurer.

Use of artificial intelligence and machine learning techniques

Example 7 – model understanding and communication

Task specification

9.32A practitioner is carrying out technical actuarial work to support the pricing framework for motor insurance where the intended user of the work is a governance forum. The practitioner is investigating moving from using a Generalised Linear Model (GLM) in an established software package, to a more complex technique which may not be as intrinsically understandable. This investigation involves building the more complex model and using the same data set as the existing GLM and reporting the advantages and disadvantages of using the more complex model including its accuracy.

9.33GLMs are typically considered to be intrinsically understandable – a practitioner can explain and communicate the model's working, although the ease of this may depend on the GLM in question and how many factors and interactions it includes. With the GLM, the practitioner and reviewer can check their understanding of the model using approaches that may not be available for a more complex model, such as:

The coefficients for factors can be plotted so that the direction and extent of a relationship can be checked using subject matter expertise.
The specification of the GLM can be checked against established practice with a step through of the process which follows the sections of TAS 100: data, assumptions, and models.

9.34The practitioner is considering how to understand the workings of more complex models and explain these models to the governance forum.

Model choice and features

9.35The practitioner uses the GLM as a baseline model where the specification and relationships are understood. The practitioner specifies a range of more complex models, of which some are not intrinsically understandable, and others have been designed to be more understandable.

9.36The practitioner builds and trains the range of models optimising for accuracy. The models are ranked by performance metrics on the accuracy of predictions for an independent testing data set which has been kept separate from the training data set. There are established metrics used by the insurer and the governance forum to compare the accuracy of models.

9.37To support the understanding of the models, the practitioner uses a range of techniques which help to show the relationships between input variables and the output variable. The choice of techniques for understanding the models is a judgement for the practitioner, considering the information that may be needed by their intended user. The practitioner views these techniques as models and considers the limitations of these techniques and whether these limitations are material to understanding the output from the complex models.

9.38The practitioner carries out analysis at the portfolio level, as well as for individual policyholders. They understand that examining the impact on the whole portfolio will depend on the portfolio at a point in time and the practitioner will consider how any changes to the portfolio will impact the overall relationships.

9.39The techniques considered by the practitioner for understanding the model include:

Individual conditional expectation (ICE) plots for a selection of input features.
Partial dependency plots which show the average impact of an input across the portfolio.
Shapley values to gain more understanding of the factors that have the most impact on the portfolio. The practitioner recognises that Shapley values include approximations and assume features are independent.
Fitting a Local Interpretable Model-agnostic Explanation (LIME), to help show an approximate linear relationship which can be understood by the practitioner. The practitioner understands that LIME approximates the complex model and there are limitations from the sampling basis and the simpler modelling. ⁶

9.40The practitioner uses their judgement to rank the intrinsic understandability of the models. The GLM is ranked as the baseline model which is understandable without the application of subsequent analysis. The models can then be compared for both accuracy and explainability relative to the GLM. A different ranking is given for being intrinsically understandable before use of further analysis and being explainable after the analysis has been applied. The practitioner chooses the explainability rankings guided by their ability to answer the following questions:

Does the relative importance of the various features of the model seem reasonable?
Do the interactions between features in the model seem reasonable?
Does the output of the model seem reasonable given the inputs?

Model	Accuracy ranking (1 = Most accurate, 5 = Least accurate )	Intrinsic understandability ranking	Explainability ranking
GLM	5	High	High
Complex model 1	1	Low	Medium
Complex model 2	3	Low	High
Complex model 3	4	Medium	High
Complex model 4	2	Medium	Medium

9.41The practitioner recommends that complex model 3 is used within the pricing framework. This recommendation is based on considering the existing knowledge and understanding of the governance forum as the intended user of this recommendation.

The practitioner makes a judgement that it is necessary for the model to be equally explainable as the existing GLM, in order to provide comfort for the governance forum in approving the model. They therefore exclude complex model 1 and complex model 4.
The practitioner chooses to recommend complex model 3 over complex model 2, even though it is less accurate, as it has higher inherent understandability, which means fewer additional techniques are needed to give it a high explainability ranking. This is important as the additional techniques needed to improve the explainability are themselves models with limitations.
The practitioner recommends complex model 3 over the existing GLM as they consider the potential commercial advantages from increased model accuracy outweigh the additional risk from limitations of techniques used to increase explainability.

Communication to the intended user

9.42The intended user of the technical actuarial work is the governance forum that will decide whether the pricing framework, including the models and the assumptions, is appropriate for a live environment.

9.43As part of considering TAS 100 Principle 7 Communications, the practitioner focuses on TAS 100 A7.6. The practitioner's report includes an explanation of the key judgements, the data used, the key assumptions, an explanation of the methodology, and a description of the steps taken to validate the recommended model.

9.44As the governance forum is not familiar with the proposed model, the practitioner also provides an outline of how the model works. This includes reporting on how the model responds to changes in key input variables, as shown through the application of techniques to increase explainability, and any limitations of these techniques they consider material to the decision being made.

Example 8 – model governance

9.45A practitioner at a UK general insurer is investigating using a neural network⁷ to calculate the reserves for claims liabilities to consider this as an alternative to their current chain ladder approach. The intended user for the model investigation is the Model Control Board of the insurer.

9.46The practitioner considers the appropriate level of model governance in line with TAS 100 P5.2. The practitioner concludes that the model risk exposure is high and that a strong level of model governance is needed. This is due to several factors:

The model is complex with a large number of inputs.
The decision makers are unfamiliar with neural networks.
The output from the model will be used in business critical areas.

9.47The practitioner follows the established governance process for data and models at the insurer. The practitioner considers whether existing governance processes are sufficient to address the areas of heightened risk within this project. This includes considering if additional governance steps are required in relation to data security, data privacy, model validation or use of open-source software. The practitioner also considers whether any additional internal stakeholders should be engaged or consulted as part of the governance process, including within the insurer's IT function or any firmwide ethics committee.

9.48Given their unfamiliarity with the neural network techniques being proposed, the practitioner decides to provide a training session for the Model Control Board on neural networks setting out the advantages and limitations, with comparisons to other available modelling techniques.

9.49The practitioner identifies additional steps beyond the existing governance process, were the new technique to be used:

The model developer records seeds that are used in random processes to allow a reviewer to reproduce the random processes if needed.
Testing data sets are used to investigate how the model performs on an independent data set that is separate from the data used to train the model.
The practitioner checks the version control system for managing code and model development is capable of managing the neural network development process.
The use of open-source libraries and code is documented so that judgements surrounding the reliability of the libraries and code can be assessed.
Judgements around the choice of model and model structure are recorded including the choice of hyperparameters⁸.

9.50The practitioner acknowledges that the additional steps considered are dependent on the nature of the project, in this case relating to a quantitative calculation of reserves. The practitioner recognises that different steps may be required for technical actuarial work using other AI techniques such as natural language processing or generative AI.

9.51The practitioner ensures the communication to the Model Control Board is comprehensible given the technical knowledge of the members of the Board. The practitioner includes background to the techniques and the results of the analysis to help the Model Control Board understand the contribution of inputs to the model output (see example 7 paragraphs 9.37 to 9.39 for examples).

9.52As the model risk was assessed as high, the practitioner judges that it would be appropriate to have peer reviews carried out by both a reserving and a machine learning subject matter expert to ensure the review is comprehensive. The practitioner agrees with each reviewer the scope of review they are expected to perform. The practitioner judges that the scope of review is sufficient for the risk of the work, and that reviewers have the appropriate experience and expertise.

9.53The Model Control Board considers the practitioner's work and assesses the risks and limitations of neural networks along with the benefits and approves the model.

9.54Following approval of the neural network model, the practitioner chooses to further develop the model so that it can update its parameters while in use, as it takes in new data as soon as possible. Any automated process for taking on new information would follow the insurer's data governance approach.

9.55The practitioner considers the appropriate level of model governance in this scenario in line with TAS 100 P5.2. The practitioner concludes that there is an extra element of model risk exposure and that additional elements of model governance are needed due to the lack of scrutiny at the point of data ingestion and model change validation.

9.56In addition to the governance process for the model above, the practitioner decides to introduce the following additional steps:

The practitioner explains the risks and limitations of the self-updating capabilities of the model to the Model Control Board. The practitioner arranges a training session for the Model Control Board to cover the additional concepts.
The Model Control Board is provided with frequent reporting on overall prediction trends and trends split by key categories. This includes developing ongoing automated processes to identify any drift in model accuracy or model bias.
The practitioner identifies a range of model outputs that are considered acceptable, where results which are too high or too low will be flagged for further inspection.
The practitioner identifies the triggers for stopping the model updating the parameters by itself and details how the model can be reverted to a version which does not update while in use.
A record is maintained of self-updates to the system to enable it to be restored to a previous version if deemed necessary. Self-updates to the system would be monitored on a regular basis by the practitioner, and guard rails developed for the level of change in model outputs that would require a fuller model review, or completion of the organisation's full model governance process.

Example 9 – model stability

9.57A practitioner is developing a model to produce short and medium-term estimates of national mortality rates. They are considering developing a neural network to estimate mortality rates across different cohorts of individuals based on past mortality experience. They are considering the work required to validate this model.

9.58TAS 100 P5.2 states that 'Practitioners must ensure that the models they use for technical actuarial work have in place an appropriate level of model governance.' Model governance is defined in TAS 100 to include for example, appropriate checks to ensure the stability of model outputs.'

9.59The practitioner considers what checks are appropriate to ensure the stability of model outputs for this alternative model, taking into account the materiality of the decisions that will be made based on the model output. In addition to following their existing governance process, the practitioner decides to test the following in relation to stability:

That if a different set of data is used to train the model that the practitioner expects to show similar trends in mortality rates then it would result in a model that produces broadly similar results.
How the output may differ if training data differed more substantially, to identify how the model may perform under different conditions. This may require testing with synthetic data to simulate the impact of differing mortality trends or shocks.
Comparison of the results of the modelling, with alternative modelling based on the same underlying data. In this case, they consider how projections under this alternative model compare to projections under the CMI model with parameters the practitioner judges are appropriate for the situation considered.
That varying the hyperparameters used in training the model would not result in a disproportionate change to the outputs.
The impact of adjusting the model to place a different weight on mortality rates experienced during the Covid-19 pandemic.

9.60The practitioner considers whether there is also a need to demonstrate that the model output is 'reproducible' – such that sufficient supporting information is held to enable an identical model, producing identical outputs, to be trained. This is not required by TAS 100, and the practitioner judges it is not required by any other regulations applicable to their work. As a result, for this work they do not seek to achieve or demonstrate reproducibility of model outputs.

9.61The practitioner also develops procedures for future testing for stability of the model. These include:

Setting out the conditions under which the model may be considered appropriate for use, including how frequently or under what circumstances updating data and retraining the model may be required.
Defining limits either on the level of change in underlying data, or the change in predictions from the model before a full review (without reliance on previous assessment) of the model is required.

9.62The practitioner documents the results of their testing for stability. In the communication to their intended user, they explain how the model has been tested for stability and that they are satisfied that the model is stable. The practitioner also sets out recommendations of appropriate future testing for stability of the model when it is in use. After considering TAS 100 P7.3, they decide not to communicate the detailed results of this testing, as this may obscure more material actuarial information relating to the accuracy of the model. However, the practitioner makes it clear that this testing has been carried out and is available on request.

10. Appendix 3: Application statements

A5.1In ensuring models are appropriate for their intended use, practitioners should consider whether the model has sufficient regard to extreme events or outliers.

A5.2In identifying whether models include any material bias, the practitioner should consider whether:

The model leads to consistent overestimation or underestimation.
The model contains systematic error, leading to results that are not representative of the aspect of the world that it is designed to model.

A5.3If material biases are identified, the practitioner should seek to improve the model, by adjusting it, if appropriate, to reduce the impact of this bias. Where model bias gives rise to material limitations in actuarial information, the practitioner should assess the implications.

Financial Reporting Council 8th Floor 125 London Wall London EC2Y 5AS +44 (0)20 7492 2300 www.frc.org.uk

Footnotes

Work performed for the intended user where the use of principles and/or techniques of actuarial science is central to the work and which involves the exercise of judgement, or which the intended user could reasonably regard as technical actuarial work by virtue of the manner of its communication. ↩
An out-of-model adjustment is an adjustment which is made to the output of the model ↩
An adjustment which is made to the output of the model ↩
Volume refers to the amount of data, variety refers to the number of types of data and velocity refers to the speed of data input and processing. ↩
Financial Conduct Authority's Senior Managers and Certification Regime ↩
For a description of a number of these techniques, see: What-should-an-actuary-know-about-Artificial-Intelligence.pdf ↩
A neural network is defined by the Turing Institute as an artificial intelligence system inspired by the biological brain consisting of a large set of simple, interconnected computational units ('neurons'), with data passing through them as between neurons in the brain. Neural networks can have hundreds of layers of these neurons, with each layer playing a role in solving the problem. ↩
A hyperparameter is a parameter that defines the configuration of a machine learning process and is set before training a model. ↩

File

Name Technical Actuarial Guidance: Models (October 2024)

Publication date 03 October 2024

Type Guidance

Format PDF, 351.1 KB

Download original

Technical Actuarial Guidance: Models (October 2024)

File

Is this page useful?