Understanding your model with LIME
With the advancement in machine learning and deep learning algorithms , models now have prediction accuracy, process efficiency, and research productivity . However, these are black box and generally don’t explain their prediction .This becomes a barrier to the adoption of machine learning models especially in world of finance where explainable AI is usually a regulatory requirement.
There are many method which help in improving the explainability ofthe model predictions such as SHAP,LIME and Permutation importance .Today we will discuss the theory and practical application of one such method — LIME.
LIME
Introduction
Lime is a post hoc technique, which means that this is applied after the event i.e. after model training. So, the idea is to first build complex models which can achieve high performance and then explain those decisions by using a simpler model. In other words, constructing complex model first and assumptions will come later.
Lime stands for Local Interpretable Model-agnostic Explanations. Here each word has its own meaning:-
Local - Instead of explaining the predictions globally by building a global surrogate model, LIME focuses on training local surrogate models to explain individual prediction. Surrogate models are models trained to approximate the predictions of the underlying black box model.
Interpretable- The Idea behind LIME is to make easy to interpretable local model such as linear regression which can explain your black box model locally.
Model Agnostic- LIME can be applied to any black box model irrespective of the technique as long as it can predict probability .
Explanations- LIME will be able to explain how the model behaves, which features it picks up and what kinds of interactions between them takes place to drive the predictions
How Lime Works?
The idea on which lime works is that it is much easier to approximate a black-box model by a simple model locally (in the neighborhood of the prediction we want to explain), as opposed to trying to approximate a model globally. In the figure below, the decision boundary of a black-box model is represented by the blue-pink background, which seems too complex to be accurately approximated by a linear model. However, when we zoom in and look closely in the small enough neighborhood of an instance, no matter how complex the model is at a global level, the decision boundary at a local neighborhood can be a much more simpler, can in fact be even linear.
The overall goal of LIME is to identify such an interpretable model over the interpretable representation that is locally faithful to the classifier. Moreover, it must achieve so without making any assumptions about the model decision function, as we want the explainer to be model agnostic.
The idea of LIME is quite intuitive. First forget about the training data and imagine you only have the black box model which you can probe as much as you want to get prediction. LIME want to understand what happens to the predictions when you give variations of your data into the machine learning model to understand the effect of different feature on your model prediction. LIME generates a new dataset consisting of perturbed samples and the corresponding predictions of the black box model. On this new dataset LIME then trains an interpretable model, which is weighted by the proximity of the sampled instances to the instance of interest(which is defined by the parameter kernel width).This new model will have good approximation of the complex model locally but doesn’t have a good global approximation. This kind of accuracy is also called local fidelity.
For better understanding of loss function and working of LIME you can go through these source (“Why Should I Trust You?”) and Interpretable ML book
The recipe for training local surrogate models:
- Select your instance of interest for which you want to have an explanation of its black box prediction.
- Perturb your dataset and get the black box predictions for these new points.
- Weight the new samples according to their proximity to the instance of interest.
- Train a weighted, interpretable model on the dataset with the variations.
- Explain the prediction by interpreting the local model.
How do you perturb data in LIME ? This depends on the type of data, which can be either text, image or tabular data. For text and images, the solution is to turn single words or super-pixels on or off. In the case of tabular data, LIME creates new samples by perturbing each feature individually, drawing from a normal distribution with mean and standard deviation taken from the feature.
LIME step by step guide in python
Lime can be applied to any type of dataset such as tabular data, image or text. For this exercise we are taking an example of a tabular dataset. I used Titanic dataset for my experiment. It is a classification problem which aims to predict whether a person survived(1) on titanic or not(0), with the use of features such as age, sex, ticket fare, cabin , embarked etc. I did some pre processing on the data and split my data into 70–30 ratio. Build a basic XGB model on it. The pre processing and model building code can be found on my Github
LIME EXPLAINATION
As opposed to lime_text.TextExplainer, tabular explainers need a training set. The reason for this is because it compute statistics on each feature (column). If the feature is numerical, it compute the mean and standard deviation, and discretize it into quartiles. If the feature is categorical, we compute the frequency of each value. These computed statistics are used for two things:
- To scale the data, so that it can meaningfully compute distances when the attributes are not on the same scale
- To sample perturbed instances — which it does by sampling from a Normal(0,1), multiplying by the standard deviation and adding back the mean.
#importing the required library for lime
import lime
import lime.lime_tabularexplainer = lime.lime_tabular.LimeTabularExplainer(X_train.values[:,:],feature_names = X_train.columns,class_names=['not_survived','survived'], categorical_features=categorical_features, categorical_names=categorical_names,discretizer='decile',kernel_width=5,discretize_continuous=True,verbose=True)
Point to Note: LIME takes in numerical data, even if the features are categorical .So you will have to convert the categorical . You can transform all of the string attributes into integers, using sklearn’s LabelEncoder and also use a dict to save the correspondence between the integer values and the original strings, so it can be presented later in the explanations.
Lime doesn’t support OHE because it perturbs the dataset so let’s say we encode the variable “sex” with one hot encoding, such that [1, 0] is male and [0, 1] is female. While, perturbing the dataset , Lime with generate [1, 1] which is a nonsensical input (which will give your model something invalid).For further details you can refer to this thread
There is work around the use of One hot encoding or any other type of transformation .you can follow the steps in this notebook
Let’s understand the arguments of LIME’s tabular Explainer-
- categorical_features: list of indices (ints) corresponding to the categorical columns. everything else will be considered continous
- categorical_names: map from int to list of names, where categorical_names[x][y] represents the name of the yth value of column
- verbose: if true, print local prediction values from linear model
There are two more important arguments:
- Discretize: if discretize_continuous is set to true then all non-categorical features will be discretized into ‘quartile’ . By default quartile is used, you can set it to ‘decile’ or ‘entropy’. Discretization is used/preferred as the numeric values may be in different ranges. Thus we standardize the data, but then the meaning of the coefficients changes . It’s hard to think about double negatives (i.e. negative weight for a negative feature = positive contribution). We will show the both the result with and without discretization and explain it in detail there. for more discussion on discretization you can check this thread
- Kernel Width: The kernel width determines how large the neighborhood is: A small kernel width means that an instance must be very close to influence the local model, a larger kernel width means that instances that are farther away also influence the model. If not set the kernel width is 0.75 times the square root of the number of columns of the training data. Weights/Influence of the generated samples are set according to their distances to the instance we want to explain(Euclidean distance). The weights are determined by a kernel function which takes Euclidean distances and kernel width as input and outputs weight for each generated instance.
weight = np.sqrt(np.exp(-(d ** 2) / kernel_width ** 2)1. Using discretize_continuous=True
Interpreting the second record
We will explain the second record using our Model Lime. You can set number of features you want to use for explanation.
exp = explainer.explain_instance(X_train.iloc[1,:],xgb_model.predict_proba, num_features=4)
Here the intercept of the equation is 0.15, Lime prediction is 0.76 and Actual model prediction is 0.95.
We can plot impact of individual feature on model prediction.
%matplotlib inline
fig = exp.as_pyplot_figure(label=1)
Here the green bars show they have positive impact i.e. they increase the model score, while the ones in red decrease the score.
Now here discretization becomes easy to understand. For eg let’s take sex feature, Sex=Female has positive impact on the model , i.e., it increases the model score by 0.5 if rest of the feature remain unchanged. You can follow this thread to better understand the interpretation of lime result
We can also plot and save the results in a notebook format .
exp.show_in_notebook(show_all=False)
exp.save_to_file('explanation.html')
Here it shows the actual predictions, contributions, Independent variables and their corresponding value
To get coefficient of each variable and creating Lime prediction from it you can use the below command
pd.DataFrame(exp.as_list(),columns=['Feature','Contribution'])
2. Using discretize_continuous =False
Here we are not using discretization and we are predicting for third observation
explainer = lime.lime_tabular.LimeTabularExplainer(X_train.values[:,:],feature_names = X_train.columns,class_names=['not_survived','survived'],
categorical_features=categorical_features,
categorical_names=categorical_names,kernel_width=5,discretize_continuous=False,verbose=True)
#getting the lime prediction for an observation
exp = explainer.explain_instance(X_train.iloc[2,:],xgb_model.predict_proba, num_features=4)
Plotting the variable level contribution, we can see that for class and age it is not showing buckets but the effect of average change in prediction for age and P class in the local region of instance
#variable level contribution
%matplotlib inline
fig = exp.as_pyplot_figure(label=1)
Notebook style summary
pd.DataFrame(exp.as_list(),columns=['Feature','Contribution'])
The reason why discretize is preferred because interpreting coefficients for non-discretized data is tricky. The coefficients are computed using standardized values, i.e. the explanation for Age is saying something like ‘for every one standard deviation increase in Age, the prediction goes down by 0.1. However, the instance has Age=29.70, where the mean of the dataset is Age=30. So, the negative coefficient multiplied by scaled value of instance ((X- mean)/std ) will lead to a positive effect (double negative).
Thus, understanding the effect of each variable will becomes tricky
Manually Estimating Lime prediction
#calculating lime prediction value
#getting indepedent variables values
x=X_train.iloc[1,:]
#cretaing scaled version
scaled_x = (x - explainer.scaler.mean_) / explainer.scaler.scale_#setting scaled value for categorical variable to be 1
#categorical_features indices of categorical feature
scaled_x[categorical_features]=1
#extracting columns indiced
cols=[x[0] for x in exp.as_map()[1]]
#extracting their coeff
coeff=[x[1] for x in exp.as_map()[1]]
# getting prediction using pred= wixi + intercept
sum(scaled_x.iloc[cols]*coeff)+exp.intercept[1]
Conclusion
Like any other technique LIME has it limitation :
1. First, LIME explanation is unstable because it depend on a lot of parameters such as kernel width, number of features, number of samples, discretization . In certain scenarios, explanation can be easily changed by changing the kernel width
2. As mentioned earlier, LIME is a post hoc technique based on the assumption that model must be locally linear. If model is overcomplex and not linear locally, then local explanation might not be accurate. Thus, you need to find a good kernel width
3.Also, based on my personal experience , I have observed that if you repeat the sampling process, then the explanations that come out can be different.
As each time Lime might sample different observations. This type of instability makes it difficult to trust the explanations.
Even with the above limitations, LIME is a great tool to explain what your black box model is doing. It is model-agnostic, intuitive and easy to run. It can be used for any type of text, image and tabular data. If local linearity and right neighborhood is achieved , LIME can provide a good insight on the prediction of black box model . However, it should be used with great care because of the limitations mentioned above.
Thank you for reading. I hope this helps with learning the basic explainability of machine learning models through LIME. Source code can be found on GitHub. Feel free to share any questions or feedback, connect with me at LinkedIn
Reference
- https://compstat-lmu.github.io/iml_methods_limitations/introduction.html#interpretable-machine-learning
- https://www.oreilly.com/content/introduction-to-local-interpretable-model-agnostic-explanations-lime/
- https://towardsdatascience.com/understanding-model-predictions-with-lime-a582fdff3a3b
- https://towardsdatascience.com/opening-black-box-models-with-lime-beauty-and-the-beast-9daaf02f584a
- https://christophm.github.io/interpretable-ml-book/lime.html
- https://medium.com/dataman-in-ai/explain-your-model-with-lime-5a1a5867b423#:~:text=If%20you%20have%20categorical%20variables,as%20the%20instance%20being%20explained.
