Lending Club - Loan Default Modeling¶
Out[1]:
0. Setup¶
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.patches as patches
from sklearn.metrics import roc_curve,auc
from scipy import interp
In [2]:
pd.set_option('display.max_columns', 500)
In [93]:
loan = pd.read_csv("E:/Downloads/CreditNinja_ModelingChallenge/data.csv")
print(loan.shape)
loan.head()
Out[93]:
1. Data Review and Dependent Variable Definition¶
I chose loan_status as the dependent variable and turned it into a binary classification problem by setting 'Default' as '1' and other status as '0'. Since we only want to focus on the default clients, there is no need to identify the difference between 'Current' and 'Fully Paid'.
In [94]:
# create new dependent variable: default
loan.loc[loan['loan_status'] == 'Default', 'Default'] = 1
loan.loc[loan['loan_status'] != 'Default', 'Default'] = 0
# drop the original unique id and loan_status column
loan = loan.drop(['id', 'loan_status'], axis=1)
print(loan.shape)
loan.head()
Out[94]:
2. Data Cleansing¶
My data cleaning methodology and findings are listed as below.
- First, I looked at all the variables. I found that there are some variables containing information that happens after the loan underwriting. Since it may raise data leakage issues, I chose to drop them. In addition, some variables are not relevant to one's ability to pay the loan, such as the state information. I dropped there variables as well.
- Second, I found that there are four columns in total with missing values. As for those with more than 40% NAs, I chose to drop them. Also, for the emp_length column with only a few NAs, I chose to drop the rows with missing value in this column.
- Third, there are a few outliers (9999) in the
ditcolumn. However, they have already been dropped in the second step. - Fourth, I dealt with the nominal and ordinal variables in the dataset. emp_length is the only ordinal variable. I assigned different numbers to different levels accordingly to convert it into a numerical variable that can be fed into the model. As for the four nominal variables, I used the get_dummies() function to conduct one-hot encoding.
- Fifth, since there are some variables containing really large values, such as
annual_incanddit, I decided to normalize the data to ensure the model performance.
2.1 Select appropriate variables¶
First, there are several variables we would like to drop.
- last_credit_pull_d: This information happens after the loan decision and has no predictive power for our model.
- last_fico_range_high: This information happens after the loan decision and has no predictive power for our model.
- last_fico_range_low: This information happens after the loan decision and has no predictive power for our model.
- issue_d: This information happens after the loan decision and has no predictive power for our model.
- addr_state: The state of the borrower can not reflect one's ability to pay the loan.
- earliest_cr_line: The borrower's earliest reported credit line open month is not relevant one's ability to pay the loan.
In [95]:
loan = loan.drop(['last_credit_pull_d','last_fico_range_high','last_fico_range_low','addr_state','issue_d','earliest_cr_line'], axis=1)
print(loan.shape)
loan.head()
Out[95]:
2.2 Dealing with NAs¶
In [6]:
# check the NAs in each column
isnull = loan.isnull().sum().sort_values(ascending=False)/len(loan)
print('col_count:',isnull.count())
isnull
Out[6]:
In [7]:
# drop the columns with over 40% missing values
loan = loan.dropna(thresh=len(loan)*0.6, axis=1)
print(loan.shape)
loan.head()
Out[7]:
In [8]:
loan.select_dtypes(include=[np.number]).describe().T
Out[8]:
In [9]:
loan.select_dtypes(include=['object']).describe().T
Out[9]:
In [10]:
# drop the rows with missing values in the `emp_length` variable
loan = loan.dropna(axis=0)
print(loan.shape)
loan.head()
Out[10]:
2.3 Dealing with Ordinal / Nominal variables¶
The emp_length variable is an ordinal variable. Therefore, I assigned different numbers to different levels accordingly to convert it into a numerical variable that can feed the model.
In [11]:
map = {
"emp_length": {
"10+ years": 10,
"9 years": 9,
"8 years": 8,
"7 years": 7,
"6 years": 6,
"5 years": 5,
"4 years": 4,
"3 years": 3,
"2 years": 2,
"1 year": 1,
"< 1 year": 0
}
}
loan = loan.replace(map)
print(loan.shape)
loan.head()
Out[11]:
As for the four nominal variables, I used the get_dummies() function to conduct one-hot encoding.
In [12]:
n_columns = ["term","home_ownership", "verification_status", "purpose"]
dummy_df = pd.get_dummies(loan[n_columns],drop_first=True)
loan = pd.concat([loan, dummy_df], axis=1)
loan = loan.drop(n_columns, axis=1)
print(loan.shape)
loan.head()
Out[12]:
2.4 Normalize the Data¶
In [13]:
columns = loan.columns
features = columns.drop('Default')
features_s = columns.drop('Default').drop(dummy_df.columns)
from sklearn.preprocessing import StandardScaler
sc =StandardScaler()
loan[features_s] =sc.fit_transform(loan[features_s])
print(loan.shape)
loan.head()
Out[13]:
In [14]:
loan.describe().T
Out[14]:
In [17]:
loan.to_csv('loan_clean.csv')
3. Analysis¶
3.1 Feature Selection - RFE¶
Using Recursive Feature Elimination, I selected 20 out of 30 variables that are most relevant to the dependent variable.
In [18]:
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
x_val = loan[features]
y_val = loan['Default']
estimator = LogisticRegression(class_weight='balanced',solver='liblinear')
rfe = RFE(estimator=estimator,n_features_to_select=20).fit(x_val, y_val)
x_chosed = features[rfe.support_]
x_chosed
Out[18]:
3.2 Feature Selection - Pearson Correlation (Multicollinearity)¶
Based on the correlation heatmap, I dropped some variables that are highly relevant to another feature in order to avoid multicollinearity and improve the model performance.
In [19]:
colormap = plt.cm.viridis
plt.figure(figsize=(12,12))
plt.title('Pearson Correlation of Features', y=1.05, size=15)
sns.heatmap(loan[x_chosed].corr(),linewidths=0.1,vmax=1.0, square=True, cmap=colormap, linecolor='white', annot=True)
Out[19]:
In [20]:
drop_col = ['installment','fico_range_high','term_60 months','verification_status_Verified']
x_new = x_chosed.drop(drop_col)
x_new
Out[20]:
3.3 Split the Data¶
In [21]:
# Partition the data
X = loan[x_new]
y = loan["Default"]
from sklearn.model_selection import train_test_split
train_X, test_X, train_y, test_y = train_test_split(X,y,train_size=0.8, random_state = 1)
print ('Train Features: ',train_X.shape ,
'Test Features: ',test_X.shape)
print ('Train Labels: ',train_y.shape ,
'Test Labels: ',test_y.shape)
3.4 Balance the Data¶
Obviously, our dataset is not very balanced since most people would pay their loan. My solution is to use SMOTE() function to balance the training dataset.
In [22]:
# SMOTE
n_sample = train_y.shape[0]
n_pos_sample = train_y[train_y == 0].shape[0]
n_neg_sample = train_y[train_y == 1].shape[0]
print('Observations: {}; Positives: {:.2%}; Negatives: {:.2%}'.format(n_sample,
n_pos_sample / n_sample,
n_neg_sample / n_sample))
print('Features: ', train_X.shape[1])
In [23]:
from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state=1)
train_X, train_y = sm.fit_sample(train_X, train_y)
print('After SMOTE: ')
n_sample = train_y.shape[0]
n_pos_sample = train_y[train_y == 0].shape[0]
n_neg_sample = train_y[train_y == 1].shape[0]
print('Observations: {}; Positives: {:.2%}; Negatives: {:.2%}'.format(n_sample,
n_pos_sample / n_sample,
n_neg_sample / n_sample))
4. Modeling¶
4.1 Logistic Regression¶
In [24]:
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(solver='liblinear')
model.fit(train_X,train_y)
Out[24]:
In [25]:
import sklearn.metrics as sklmetrics
predict_y=model.predict(test_X)
sklmetrics.accuracy_score(test_y, predict_y)
Out[25]:
In [26]:
# Confusion Matrix
conf_mat = sklmetrics.confusion_matrix(test_y, predict_y, labels =[0,1])
conf_mat
Out[26]:
In [27]:
from sklearn.metrics import roc_auc_score
roc_auc = roc_auc_score(test_y, predict_y)
print("Area under the ROC curve : %f" % roc_auc)
In [28]:
def plot_feature_importance_coeff(model, Xnames, cls_nm = None):
imp_features = pd.DataFrame(np.column_stack((Xnames, model.coef_.ravel())), columns = ['feature', 'importance'])
imp_features[['importance']] = imp_features[['importance']].astype(float)
imp_features[['abs_importance']] = imp_features[['importance']].abs()
# Sort the features based on absolute value of importance
imp_features = imp_features.sort_values(by = ['abs_importance'], ascending = [1])
# Plot the feature importances of the forest
plt.figure()
plt.title(cls_nm + " - Feature Importance")
plt.barh(range(imp_features.shape[0]), imp_features['importance'],
color="b", align="center")
plt.yticks(range(imp_features.shape[0]), imp_features['feature'], )
plt.ylim([-1, imp_features.shape[0]])
plt.xlabel('Importance')
plt.ylabel('Feature')
plt.tight_layout()
plt.savefig(cls_nm + "_feature_imp.png", bbox_inches='tight')
plt.show()
In [29]:
plot_feature_importance_coeff(model, X.columns, cls_nm="Logistic Regression")
We found that the purpose of the specfic loan is really important for its default probability. Also, fico score in application would have a negative effect on the default probability. We should consider these features more carefully when processing an loan application.
Moreover, we found that dti, the number of inquiries in past 6 months and loan amount have a positive relationship to the default probability. It aligns with the business rules we created before the model runs. The heavier the financial burden is, the higher risk that the loan would get default.
4.2 Grid Search CV¶
In [57]:
from sklearn.model_selection import GridSearchCV
param_grid = {'C': 10.**np.arange(-5, 5),
'penalty': [ 'l1', 'l2']}
grid_search = GridSearchCV(LogisticRegression(solver='liblinear'), param_grid, cv=10)
grid_search.fit(train_X, train_y)
Out[57]:
In [31]:
print("Best parameters: {}".format(grid_search.best_params_))
print("Best cross-validation score: {:.5f}".format(grid_search.best_score_))
In [32]:
predict_y=grid_search.predict(test_X)
sklmetrics.accuracy_score(test_y, predict_y)
Out[32]:
In [33]:
# Confusion Matrix
conf_mat = sklmetrics.confusion_matrix(test_y, predict_y, labels =[0,1])
conf_mat
Out[33]:
4.3 K-fold Cross Validation¶
We used the optimal hyperparameter(C=10.0,penalty='l2') generated in the grid search above.
In [73]:
from sklearn.model_selection import cross_val_predict, KFold, cross_val_score
lr = LogisticRegression(C=10.0,penalty='l2',solver='liblinear')
kf = KFold(n_splits=10, shuffle=True)
cross_val_score(lr,train_X, train_y,cv=kf,scoring='accuracy').mean()
Out[73]:
In [84]:
fig1 = plt.figure(figsize=[12,12])
ax1 = fig1.add_subplot(111,aspect = 'equal')
ax1.add_patch(
patches.Arrow(0.45,0.5,-0.25,0.25,width=0.3,color='green',alpha = 0.5)
)
ax1.add_patch(
patches.Arrow(0.5,0.45,0.25,-0.25,width=0.3,color='red',alpha = 0.5)
)
tprs = []
aucs = []
mean_fpr = np.linspace(0,1,100)
i = 1
for train,test in kf.split(train_X, train_y):
prediction = lr.fit(train_X.iloc[train],train_y.iloc[train]).predict_proba(train_X.iloc[test])
fpr, tpr, t = roc_curve(train_y[test], prediction[:, 1])
tprs.append(interp(mean_fpr, fpr, tpr))
roc_auc = auc(fpr, tpr)
aucs.append(roc_auc)
plt.plot(fpr, tpr, lw=2, alpha=0.3, label='ROC fold %d (AUC = %0.2f)' % (i, roc_auc))
i= i+1
plt.plot([0,1],[0,1],linestyle = '--',lw = 2,color = 'black')
mean_tpr = np.mean(tprs, axis=0)
mean_auc = auc(mean_fpr, mean_tpr)
plt.plot(mean_fpr, mean_tpr, color='blue',
label=r'Mean ROC (AUC = %0.2f )' % (mean_auc),lw=2, alpha=1)
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC')
plt.legend(loc="lower right")
plt.text(0.32,0.7,'More accurate area',fontsize = 12)
plt.text(0.63,0.4,'Less accurate area',fontsize = 12)
plt.show()
The model accuracy has been improved to 61% after the cross validation. Also, we can see that the AUC is relatively stable across different folders.
Besides, a penalty can be added on false negatives since it would be more serious to classify a person would have default as not default than the opposite. It would further improve our accuracy.
Last Thoughts¶
Are there any ways to derive additional variables that would improve the model prediction accuracy?
- dti_range: Since the orignial dti variable is a continuous feature that needs to be scaled before putting into the model, we can categorize it into several groups / bins. For example, if dti > 10 and dti <= 20, this observation belongs to the dti_range_1020 group.
- annual_inc_range: The same as dti_range
- loan_pay_times: When making decisions, one of our assumption is that all the Defaults are created equal. People who becomes default in their first payment should not be treated as the same as the people who becomes default in their very last payment. With that being said, a label that can categorize different default behaviour is needed.
What variables, if any, can be used to create business rules that can be used to decline customer’s application before the model runs?
- Whether a customer has any defaults before: If a customer has many default records before (or in the last 12 months), we need to be careful when considering his/her application.
- Employment length (emp_length) & income (annual_inc): If a person has a relatively short employment length or low income, we may assume that his/her ability to pay the loan is not very strong. We can use these variable to decline customer's application before the model runs.
- Low FICO / credit score: the loan company would not want to give loan to person with low FICO / credit score.
- Low dti / Large Loan Amount: dti reflects a person's ability to pay the debts. If one has a low dti, the loan company would tend to have less confidence in his/her ability to pay the new loan. Also, a large loan amount would place a heavy financial burden on the applicant. It would result in high risk of not paying back the loan.