US Census Employment dataset
This page shows how US Census MA dataset was obtained and prepared to serve as an example in quickstart. Full notebook can be found in our github repository.
To find out what requirements NannyML has for datasets, check out Data Requirements.
Data Source
The dataset comes from US Census and it was obtained using folktables 1. Feature descriptions are given in PUMS documentation.
Dataset Description
The task is to predict whether an individual is employed based on features like age, education etc. The data analyzed comes from the state of Massachusetts and it covers the range from 2014 to 2018.
Preparing Data for NannyML
Fetching the Data
First we import required libraries and fetch the data with folktables:
>>> import numpy as np
>>> import pandas as pd
>>> from lightgbm import LGBMClassifier
>>> from folktables import ACSDataSource, ACSEmployment
>>> years = list(range(2014, 2019))
>>> dfs = []
>>> for year in years:
... data_source = ACSDataSource(survey_year=year, horizon='1-Year', survey='person')
... data = data_source.get_data(states=["MA"], download=True)
... features, labels, _ = ACSEmployment.df_to_numpy(data)
... df = pd.DataFrame(features)
... df.columns = ACSEmployment.features
... df[ACSEmployment.target] = labels
... df['year'] = year
... dfs.append(df)
...
>>> df = pd.concat(dfs).reset_index(drop=True)
>>> df.head()
AGEP |
SCHL |
MAR |
RELP |
DIS |
ESP |
CIT |
MIG |
MIL |
ANC |
NATIVITY |
DEAR |
DEYE |
DREM |
SEX |
RAC1P |
ESR |
year |
|
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 |
30 |
19 |
1 |
0 |
2 |
0 |
1 |
3 |
4 |
1 |
1 |
2 |
2 |
2 |
1 |
2 |
True |
2014 |
1 |
24 |
19 |
1 |
1 |
2 |
0 |
1 |
3 |
4 |
2 |
1 |
2 |
2 |
2 |
2 |
2 |
False |
2014 |
2 |
5 |
2 |
5 |
4 |
2 |
2 |
1 |
3 |
0 |
2 |
1 |
2 |
2 |
2 |
2 |
2 |
False |
2014 |
3 |
5 |
2 |
5 |
4 |
2 |
2 |
1 |
3 |
0 |
2 |
1 |
2 |
2 |
2 |
1 |
2 |
False |
2014 |
4 |
83 |
22 |
1 |
0 |
1 |
0 |
1 |
1 |
2 |
2 |
1 |
2 |
2 |
2 |
1 |
1 |
False |
2014 |
Data is fetched for each year separately and column year is created.
Descriptions of all the variables can be found in the appendix.
Defining Partitions and Preprocessing
We split the data into three partitions simulating model lifecycle: train, test and production (after deployment) data. We will use 2014 data for training, 2015 for evaluation and 2016-2018 will simulate production data.
>>> df['partition'] = None
>>> df['partition'] = np.where(df['year']==2014, 'train', df['partition'])
>>> df['partition'] = np.where(df['year']==2015, 'test', df['partition'])
>>> df['partition'] = np.where(df['year']>2015, 'prod', df['partition'])
We now define categorical and numeric features:
>>> categorical_features = ['SCHL','MAR', 'RELP', 'DIS', 'ESP', 'CIT', 'MIG', 'MIL', 'ANC', 'NATIVITY',
... 'DEAR', 'DEYE', 'DREM', 'SEX', 'RAC1P']
>>> numeric_features = ['AGEP']
Since categorical features are already encoded correctly for LGBM model (non-negative, integers-like), we don’t need
any preprocessing. We will just turn them into proper integers
. We will also rename the target column and convert
target to int
:
>>> features = numeric_features + categorical_features
>>> target_col = 'employed'
>>> df = df.rename(columns={'ESR':target_col})
>>> df[categorical_features] = df[categorical_features].astype(int)
>>> df[target_col] = df[target_col].astype(int)
Developing ML Model and Making Predictions
We will now fit a model that will be subject to monitoring (e.g. in quickstart):
>>> client_model = LGBMClassifier(random_state=1)
>>> client_model.fit(df[df['partition']=='train'][features], df[df['partition']=='train'][target_col],
... categorical_feature=categorical_features)
>>> df['prediction'] = client_model.predict(df[features])
>>> df['predicted_probability'] = client_model.predict_proba(df[features])[:,1]
Let’s turn categorical features into category
dtype
so that NannyML correctly recognizes them:
>>> for feat in categorical_features:
... df[feat] = df[feat].astype(str).astype('category')
Splitting and Storing the Data
Now we will just split the data based on partitions, drop selected columns and store it in the relevant location in NannyML repository so the data can be accessed from within the library:
>>> full_reference_data = df[df['partition']=='test'].reset_index(drop=True).drop(columns='partition')
>>> analysis_wo_targets = df[df['partition']=='prod'].reset_index(drop=True).drop(columns=[target_col, 'partition'])
>>> analysis_targets = df[df['partition']=='prod'][[target_col]].reset_index(drop=True)
>>> data_dir = '../../../nannyml/nannyml/datasets/data/'
>>> full_reference_data.to_parquet(data_dir + "employment_MA_reference.pq")
>>> analysis_wo_targets.to_parquet(data_dir + "employment_MA_analysis.pq")
>>> analysis_targets.to_parquet(data_dir + "employment_MA_analysis_target.pq", )
Appendix: Feature description
This description comes from PUMS documentation:
AGEP - age person, numeric
SCHL - Educational attainment:
bb .N/A (less than 3 years old)
01 .No schooling completed
02 .Nursery school, preschool
03 .Kindergarten
04 .Grade 1
05 .Grade 2
06 .Grade 3
07 .Grade 4
08 .Grade 5
09 .Grade 6
10 .Grade 7
11 .Grade 8
12 .Grade 9
13 .Grade 10
14 .Grade 11
15 .12th grade - no diploma
16 .Regular high school diploma
17 .GED or alternative credential
18 .Some college, but less than 1 year
19 .1 or more years of college credit, no degree
20 .Associate’s degree
21 .Bachelor’s degree
22 .Master’s degree
23 .Professional degree beyond a bachelor’s degree
24 .Doctorate degree
MAR Character 1 - Marital status:
1 .Married
2 .Widowed
3 .Divorced
4 .Separated
5 .Never married or under 15 years old
RELP Character 2 - Relationship:
00 .Reference person
01 .Husband/wife
02 .Biological son or daughter
03 .Adopted son or daughter
04 .Stepson or stepdaughter
05 .Brother or sister
06 .Father or mother
07 .Grandchild
08 .Parent-in-law
09 .Son-in-law or daughter-in-law
10 .Other relative
11 .Roomer or boarder
12 .Housemate or roommate
13 .Unmarried partner
14 .Foster child
15 .Other nonrelative
16 .Institutionalized group quarters population
17 .Noninstitutionalized group quarters population
DIS Character 1 - Disability recode:
1 .With a disability
2 .Without a disability
ESP Character 1 - Employment status of parents:
b .N/A (not own child of householder, and not child in subfamily)
1 .Living with two parents: both parents in labor force
2 .Living with two parents: Father only in labor force
3 .Living with two parents: Mother only in labor force
4 .Living with two parents: Neither parent in labor force
5 .Living with father: Father in the labor force
6 .Living with father: Father not in labor force
7 .Living with mother: Mother in the labor force
8 .Living with mother: Mother not in labor force
CIT Character 1 - Citizenship status:
1 .Born in the U.S.
2 .Born in Puerto Rico, Guam, the U.S. Virgin Islands, or the
.Northern Marianas
3 .Born abroad of American parent(s)
4 .U.S. citizen by naturalization
5 .Not a citizen of the U.S.
MIG Character 1 - Mobility status (lived here 1 year ago) - b .N/A (less than 1 year old) - 1 .Yes, same house (nonmovers) - 2 .No, outside US and Puerto Rico - 3 .No, different house in US or Puerto Rico
MIL Character 1 - Military service:
b .N/A (less than 17 years old)
1 .Now on active duty
2 .On active duty in the past, but not now
3 .Only on active duty for training in Reserves/National Guard
4 .Never served in the military
ANC Character 1 - Ancestry recode:
1 .Single
2 .Multiple
3 .Unclassified
4 .Not reported
8 .Suppressed for data year 2018 for select PUMAs
NATIVITY Character 1 - Nativity:
1 .Native
2 .Foreign born
DEAR Character 1 - Hearing difficulty:
1 .Yes
2 .No
DEYE Character 1 - Vision difficulty:
1 .Yes
2 .No
DREM Character 1 - Cognitive difficulty:
b .N/A (Less than 5 years old)
1 .Yes
2 .No
SEX Character 1 - Sex:
1 .Male
2 .Female
RAC1P Character 1 - Recoded detailed race code:
1 .White alone
2 .Black or African American alone
3 .American Indian alone
4 .Alaska Native alone
5 .American Indian and Alaska Native tribes specified; or .American Indian or Alaska Native, not specified and no other .races
6 .Asian alone
7 .Native Hawaiian and Other Pacific Islander alone
8 .Some Other Race alone
9 .Two or More Races
ESR - target:
True - employed
False - unemployed
** References **
- 1
Ding, F. et al. (2021). Retiring Adult: New Datasets for Fair Machine Learning. Advances in Neural Information Processing Systems, 34.