Synthetic Multiclass Classification Dataset
NannyML provides a synthetic dataset describing a multiclass classification problem in order to make it easier to test and document its features.
Problem Description
The dataset describes a machine learning model that tries to predict the most appropriate product for new customers applying for a credit card. There are three options. Prepaid cards for with low credit limits, highstreet cards with low credit limits and high interest rates and upmarket cards with higher credit limits and lower interest rates.
Dataset Description
A sample of the dataset can be seen below.
>>> import nannyml as nml
>>> reference, analysis, analysis_targets = nml.datasets.load_synthetic_binary_classification_dataset()
>>> display(reference.head(3))
acq_channel |
app_behavioral_score |
requested_credit_limit |
app_channel |
credit_bureau_score |
stated_income |
is_customer |
partition |
identifier |
timestamp |
y_pred_proba_prepaid_card |
y_pred_proba_highstreet_card |
y_pred_proba_upmarket_card |
y_pred |
y_true |
|
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 |
Partner3 |
1.80823 |
350 |
web |
309 |
15000 |
True |
reference |
60000 |
2020-05-02 02:01:30 |
0.97 |
0.03 |
0 |
prepaid_card |
prepaid_card |
1 |
Partner2 |
4.38257 |
500 |
mobile |
418 |
23000 |
True |
reference |
60001 |
2020-05-02 02:03:33 |
0.87 |
0.13 |
0 |
prepaid_card |
prepaid_card |
2 |
Partner2 |
-0.787575 |
400 |
web |
507 |
24000 |
False |
reference |
60002 |
2020-05-02 02:04:49 |
0.47 |
0.35 |
0.18 |
prepaid_card |
upmarket_card |
The model uses 7 features:
acq_channel: A categorical feature with 5 categories describing the acquisition channel for the new customer. Organic refers to customers brought on by the company whereas Partner1-4 referes to customers brought on by outside partners.
app_behavioral_score: A numerical feature. This score is determined by characteristics derived from how the new customer fileld and submitted their application.
requested_credit_limit: A numerical feature. The credit limit the customer selected as appropriate for their needs.
app_channel: A categorical feature with 3 categories describing how the application was submitted. It can be in-store, from the website or from a mobile device.
credit_bureau_score: A numerical feature. The credit score provided by the credit bureau that assesses the credit worthiness of the new customer. The higher the score the more creidt worthy the customer.
stated_income: A numerical feature. The yearly income of the customer as stated by them.
is_customer: A categorical feature with 2 categories describing whether the new customer has an existing relationship with the business.
The model predicts both a probability for all classes with the y_pred_proba_prepaid_card, y_pred_proba_highstreet_card, y_pred_proba_upmarket_card columns. A class prediction is also available from the y_pred column. The y_true is the Target column with the most appropriate product choice for a given customer.
There are also three auxiliarry columns that are helpful but not used by the monitored model:
identifier: A unique number referencing each new customer. This is very useful for joining the target results on the analysis dataset when we want to compare estimated with realized performace.
timestamp: A date column informing us of the date the prediction was made.
partition: The partition column tells us which Data Period each row comes from.
Metadata Extraction
The dataset’s columns are name such that the heuristics NannyML uses to extract metadata can identify them. We can see below how to extract metadata:
>>> metadata = nml.extract_metadata(
... data = reference,
... model_name='credit_card_segment',
... model_type='classification_binary',
... exclude_columns=['identifier']
>>> )
>>> metadata.is_complete()
Let’s now see the metadata that NannyML has inferred about the model.
>>> metadata.to_df()
label |
column_name |
type |
description |
|
---|---|---|---|---|
0 |
timestamp_column_name |
timestamp |
continuous |
timestamp |
1 |
partition_column_name |
partition |
categorical |
partition |
2 |
target_column_name |
y_true |
categorical |
target |
3 |
acq_channel |
acq_channel |
categorical |
extracted feature: acq_channel |
4 |
app_behavioral_score |
app_behavioral_score |
continuous |
extracted feature: app_behavioral_score |
5 |
requested_credit_limit |
requested_credit_limit |
categorical |
extracted feature: requested_credit_limit |
6 |
app_channel |
app_channel |
categorical |
extracted feature: app_channel |
7 |
credit_bureau_score |
credit_bureau_score |
continuous |
extracted feature: credit_bureau_score |
8 |
stated_income |
stated_income |
categorical |
extracted feature: stated_income |
9 |
is_customer |
is_customer |
categorical |
extracted feature: is_customer |
10 |
y_pred_proba_prepaid_card |
y_pred_proba_prepaid_card |
continuous |
extracted feature: y_pred_proba_prepaid_card |
11 |
y_pred_proba_highstreet_card |
y_pred_proba_highstreet_card |
continuous |
extracted feature: y_pred_proba_highstreet_card |
12 |
y_pred_proba_upmarket_card |
y_pred_proba_upmarket_card |
continuous |
extracted feature: y_pred_proba_upmarket_card |
13 |
prediction_column_name |
y_pred |
continuous |
predicted label |
14 |
predicted_probability_column_name |
continuous |
predicted score/probability |
For more information about specifying metadata look at Providing Metadata.