Personal tools
You are here: Home → Community → Blog → Analyzing simulation results

Analyzing simulation results

Experience is required to properly analyze simulation results, but also IT support.

My experience of the typical actuarial tool evaluation is:

  • 80% of the questions relate to model building: Does your tool have this copula, that interest rate model, etc.
    All the top players offer about the same. And if in one environment a component is missing, then it can often be built pretty easily. The typical business component contains only a couple of hundred lines of code. I'll show some nifty features of PillarOne.Riskanalytics in another blog; but that's not the point now.
  • 20% of the questions relate to data issues: How the data get into the tool, how is result analysis supported, how does it get out of the tool (for example for reporting), how do you migrate data from one model version to the next, etc.
    In this area there are huge differences.

Here is a little exercise: classify the actuarial tools into the following categories
  • Tools which claim to be easy and compact and offer only a restricted set of key figures. Don't dare to ask questions during a result analysis - but the actuary should ask questions during this process.
  • Tools which claim to offer support for data handling and analysis. But when you look into the details, they usually offer Excel or proprietary file formats. Tools in this category usually don't offer the result analysis I require. I would like to draw on statistics, put individual results into the statistical context, compare results from different input parameters, etc. In addition, executives don't sit in front of my favorite actuarial  tool, they request a report. And if you ask an auditor, then an Excel sheet with mostly unprotected cells does not really qualify as a professional report.
  • Tools which work with standard database backends and offer some convenience in analyzing the potentially huge amount of output data from simulation runs. Those tools usually also don't have a problem with reporting, since all reporting tools can draw data from standard databases.

 

Here is a little hint into which category RiskAnalytics belongs:

I need to get my data in - of course if you have it on a database or risk data warehouse you could load it differently.

import-export

 

I start analyzing. For some key figures stats make sense and for some not - RiskAnalytics labels the non-stochastic results and properly excludes them from stats. Example: There is no VaR and TVaR of a premium which was entered as a fixed value.

stats

 The results look strange and I would like to see more details - drill down.

drill-down

 

Now that I understand the result, I would like to compare it to my previous result. High lighting the differences is really cool, if you have a lot of data. Yes, of course, RiskAnalytics lets you compare more than just two results.

compare

 

In practice, at least 50% of the time and money during a modeling project is spend on data issues. Once a model is in operational use, it is most likely around 80%. So there is plain economic reason to ask data related questions during an evaluation process.

My plea: Spend more time analyzing the results to extract the key messages.

--Markus

Document Actions