In the pipeline series – anomaly detection models and data assurance

Recently, the team at Draper & Dash have been working with a number of trusts to improve their data quality and to provide external assurance that their processes are fit for purpose. This has been an exciting challenge and it has fun to get to work with various sets of data.

We have structured a very interesting programme and have created tools that effectively risk score certain data items – these work by analysing the various stages of the data process and looking at each one of the areas to effectively identify, track and rectify issues:

Our programme works with trusts to effectively tackle specific areas of the DATA framework to aid all areas of data improvement, confidence and assurance:

This type of model falls into the creative solutions sub triangle and the approach we have used allows the capturing of indicator outliers across a multidimensional data set.

When trying to detect anomalies normally an analyst undertakes time consuming analysis to effectively identify metrics or points that deviate massively (outliers) from the normal excepted range.

The benefits of this approach are that this model can look across thousands of indicators / metrics and identify those that deviate outside the normal range aka outliers and it can reduce the manual intervention and analysis that an analyst has to conduct to find these anomalies. Therefore, saving time to get on with analysing other key business areas.

The model we have chosen to discuss in this post is the isolation forest this is very similar to random forest used normally for non-linear classification / regression.

The main idea, which is different from other popular outlier detection methods, is that Isolation Forests explicitly identify anomalies instead of profiling normal data points. Isolation Forest, like any tree ensemble method, is based on decision trees. The tree partitions are created by first randomly selecting a feature (indicator or metric) and then selecting a random split value between the minimum and maximum value of the selected feature.

The forests then are able to detect, using similar partitioning methods to random forests, the normally distributed observations and the outlier (abnormal) observations. The added twist, is that these models can accurately detect outliers across lots of columns (features).

From all these clever statistics, an anomaly score is created, the mathematical formula is not listed here for clarity and not to make this too theoretical. What essentially the model does is creates a score between 0 to 1, these scores can be interpreted as below:

A score close to a 1 indicates an anomaly
Scores much smaller than 0.5 indicates normal observations
If scores are close to 0.5 then the entire sample does not seem to have clearly distinct anomalies

This was created using an R package and shows the normally labelled points (light blue) and the outliers (pink). However, another way to use this would to be to produce a flat file to be consumed by any BI solution.

This can be used as below:

• Comparison of metrics across a number of dimensions. This could be RTT incomplete, admitted, non-admitted. This would use this dataset and detect where outliers exist across all these indicators together
• Linkedin use this method for detecting and preventing abusive accounts
• Across two or a number of different data quality markers – this is the application of this model currently relating to the data assurance programme.