Organizations are increasingly harnessing the power of data through models and using analytics to create reports and enhance efficiencies. As a result, regulatory bodies (such as the U.S. Securities and Exchange Commission [SEC]) have issued guidance and increased scrutiny during examinations surrounding improper model oversight. Reliance on poorly designed models or errors in model output could result in missed opportunities or prevent management from identifying large threats on the horizon. Testing model inputs, calculations, and outputs will give management confidence that their decisions are based on reliable information.
Creating a Model Risk Management Program
The first step in creating a strong program is to design a model risk management policy that ensures all departments within the organization are applying the same definition and oversight of models. The policy should:
- Classify end-user computations versus models for inclusion in the organization’s model inventory
- Describe the step-by-step process for new model creation
- Develop a standard model risk assessment framework
- Establish ongoing oversight
- Identify who’s responsible for its oversight and execution
- State the frequency and extent of model validation based on risk
Identifying and Assessing Models
Organizations should identify what programs, analytics, and end-user computations are in use to compare against the policy’s model definition. An inventory should be created to capture all of these that meet the model definition. End-user computations should be catalogued separately. Although end-user computations aren’t as complex or as heavily relied upon, it’s important to incorporate them in audits to verify the completeness of inputs and the accuracy of calculations.
Each model in the inventory should be annually risk assessed using the organization’s framework. Factors that should be incorporated into this framework include:
- Business decision impact
- Complexity
- Financial impact
- Input volatility
- Model design
- Model use
Each model should be given a final risk score that will determine the frequency of required validations.
Proactively Monitoring Models
In conjunction with the annual risk assessment process, organizations should develop a standard Annual Touch questionnaire. The Annual Touch should be reviewed with the model owner to determine if there are any changes to the model’s design, oversight, and inputs, or if there are other additional factors to consider when identifying the model’s validation frequency. In addition to verbal responses, documented support (such as mapping documentation, evidence of model owner review, and assumption documentation) should be obtained to corroborate responses. The reviewer should also follow up on any prior validation comments to ensure they’ve been remediated, and discuss any user overrides to the model. Significant changes or overrides may result in performing a model validation sooner.
Model Validation
Historically, regulators have primarily focused on requiring independent validations of top tier models such as the automated anti-money laundering (AML) software models. Regulatory scrutiny has increased, and all models must now have a validation schedule and verify adherence with the schedule. Model validations should verify that the model is performing as expected and in accordance with its business use. It’s critical that the validation is performed by someone who is independent of the oversight of that particular model, and who has the appropriate expertise to validate the model. The extent of the validation will depend on the complexity of the model and the potential risks pertaining to the model.
Conclusion
Establishing a comprehensive model risk management program will allow your organization to fully capitalize on beneficial data, deter possible operational disruptions, and allow management to make strategic business decisions based on the most current, accurate information.
