How to keep your models conceptually sound

Conceptual soundness in financial services’ model risk management has a lot in common with the cubism movement that became popular in the art world in the early 20^th century. Artists such as cubism movement pioneer Pablo Picasso focused on utilizing geometric shapes to depict realistic objects in an abstract view.

What does cubism, a discipline filled with subjectivity, have to do with the largely impartial field of modeling? Cubism integrates visual mathematics with human interpretation; this draws a parallel to conceptual soundness for modeling. While conceptual soundness leverages the application of textbook mathematical concepts, vagueness and ambiguity exist in this phase of the modeling process as well. As such, many institutions struggle to understand the criteria they are expected to adhere to in order to maintain a conceptually sound model.

What is conceptual soundness?

Conceptual soundness incorporates two key elements: the technical component and the discretionary component. The technical component invokes the use of standard mathematical or statistical practices for model construction. The discretionary component incorporates judgment from model stakeholders to build in the nuances that make the model fit the institution’s strategy and business objectives.

As a fundamental aspect of the model risk management framework, conceptual soundness is highlighted by the Supervisory Guidance on Model Risk Management (SR 11-7 / OCC 2011-12) issued by the Federal Reserve and Office of the Comptroller of the Currency (OCC). The guidance states that conceptual soundness entails the “quality of the model design and construction,” which includes a range of elements:

• Model purpose
• Model theory, methodology and mathematical components
• Consideration of multiple development methods
• Variable selection
• Key assumptions and limitations
• Data relevance
• Results of sensitivity analysis
• Qualitative information and management judgment

Given the importance of conceptual soundness for the ongoing success of models, the OCC has also provided more detailed guidance in the Comptroller’s Handbook on Model Risk Management (OCC 2021-39), which lists the following areas to analyze with regard to conceptual soundness:

• Soundness of statistical or mathematical principles applied
• Practicality based on business operations, product behaviors, or environmental factors
• Model stability and accuracy
• Evaluation of biases in model data and results

From principle to application

Once the framework for establishing conceptual soundness is in place, key questions need to be asked, such as how and when to put the principle into play and by whom. While conceptual soundness is a part of the modeling process, modeling experts alone cannot ensure a model is conceptually sound and appropriate for its purpose; key business stakeholders play an equally important role throughout the model lifecycle. The requirement for human insight and model customization establishes a need for model developers to partner with the business on key decisions.

The model validation process prioritizes the review of conceptual soundness; SR 11-7 notes the evaluation of conceptual soundness is a key pillar for model validation. The guidance establishes the regulatory expectation that model validators review, assess and thoroughly challenge the elements related to conceptual soundness of the model.

A robust validation process should include an analysis of the relevance of data used to develop the model. The expectation is to ensure the model is built with appropriate data and that the rationale for data selection and how the data relates to the institution’s business is thoroughly documented. By incorporating these considerations into the model development phase, the model owners and users can have confidence that they are aligned to the criteria they will be evaluated against during the independent validation.

Navigating the world of artificial intelligence

There is never a one-size-fits-all approach for model risk management and specifically the pillar of conceptual soundness. The methodology for establishing conceptual soundness largely differs depending on whether a model is rule-based, statistical or more complex. With the emergence of artificial intelligence and its growing use across the financial services industry, conceptual soundness has attracted the scrutiny of regulators such as the OCC. Artificial intelligence techniques often result in a black box model, in which the underlying logic is a mystery. As such, the lack of transparency and explainability for these types of models presents an avenue for key stakeholders to question what is going on behind the scenes of an institution’s decision-making processes, according to an OCC bulletin published in 2021.

To address these concerns, financial institutions should consider the model use, complexity and degree of risk to ensure crucial decisions have a sustainable foundation. The lack of knowledge around the model’s processing component poses a considerable limitation. Because of this, enhanced challenge and review of model inputs and outputs is paramount. Model owners must maintain support for the quality and relevance of data under which the model was built, and a model’s operational performance must be robustly evaluated on an ongoing basis. Moreover, the foundation of the model should be thoroughly documented and challenged to validate that it continues to provide adequate support for business operations.

Key takeaways

Financial services regulators are placing a substantial emphasis on conceptual soundness. Guidance from the Federal Reserve, OCC and Federal Deposit Insurance Corporation addresses conceptual soundness, and these regulators specifically spotlight conceptual soundness as a key aspect in the model risk management process.

Conceptual soundness presents a crucial intersection between the business and technical contributors to model development. Collaboration between these two groups is imperative for success during the model lifecycle.

With the emergence of new technology in a constantly evolving world, financial institutions must focus on the transparency of model functionality to maintain accountability over key decisions. Similar to cubism and other art movements formed as a response to revolutionary changes, financial institutions must learn to adapt to a transformative world by marrying technical aspects with nuanced business judgment.