Building Classifiers and Regressors in teras#
teras 0.3 offers backbone models which are task independent, i.e. they are headless. Hence they can be coupled with a classification head or a regression head to build a classifier or a regressor respectively.
Let’s see how easy it is!
For the purpose of this tutorial, we’ll take on a binary classification task using the famous Adult Income dataset. And for our model, we’ll use the TabTransformerBackbone arXiv.
But, first, let’s set our backend. My personal preference is JAX, as among other things, it’s the most efficient and fun to play with. So, for this tutorial I’ll be using that!
NOTE: You must configure your Keras backend before importing teras or keras
import os
os.environ["KERAS_BACKEND"] = "jax"
We’ll take on a classification task using the famous Adult Income dataset from the UCI dataset repository.
!curl https://archive.ics.uci.edu/static/public/2/adult.zip --output adult.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 605k 0 605k 0 0 883k 0 --:--:-- --:--:-- --:--:-- 882k
!unzip adult.zip
Archive: adult.zip
inflating: Index
inflating: adult.data
inflating: adult.names
inflating: adult.test
inflating: old.adult.names
Since you’re here, you must already be familiar with the tabular machine learning ecosystem, the pandas, the sklearn etc. So, I’ll assume you understand the following boilerplate code to load and preprocess dataset.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OrdinalEncoder, Normalizer
# Load dataset
columns = ["age", "workclass", "fnlwgt", "education", "education-num",
"marital", "occupation", "relationship", "race", "sex",
"capital-gain", "capital-loss", "hours-per-week", "native-country",
">50K"]
continous_columns = ["age", "fnlwgt", "education-num", "capital-gain",
"capital-loss", "hours-per-week"]
categorical_columns = list(set(columns) - set(continous_columns))
target_column = ">50K"
df = pd.read_csv("adult.data", names=columns, header=None)
# Ordinally encode categorical values
encoder = OrdinalEncoder()
df[categorical_columns] = encoder.fit_transform(df[categorical_columns])
# Normalize continuous features
normalizer = Normalizer()
df[continous_columns] = normalizer.fit_transform(df[continous_columns])
y = df.pop(target_column)
X = df
# Split into train and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
Let’s now import the TabTransformerBackbone. In addition, we’ll also import a Classifier model class that will wrap our backbone model to give us a model ready to be trained for the classification task at hand. You can think of the backbone and the classifier class as LEGO pieces, combined they make a classification model, but they can both be combined with other different LEGO pieces made availble by teras, like the Regressor class for instance, to build a model for regression.
Anyway, let’s get to coding!
from teras.models import TabTransformerBackbone
from teras.models import Classifier
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Cell In[5], line 1
----> 1 from teras.models import TabTransformerBackbone
2 from teras.models import Classifier
ModuleNotFoundError: No module named 'teras'
Looking at the documentation of TabTransformerBackbone, we see that it requires 3 positional arguments, in the order input_dim, cardinalities and embedding_dim. It takes many other keyword arguments as well, which you can go through by looking at the documentation.
Now input_dim and embedding_dim is easy. input_dim is equal to the dimensionality of the dataset and for embedding_dim we can pass any reasonable value like 32, 64 etc.
‘So… am i going to have to compute cardinalities myself, or is there a quick way of doing it?’ you might ask. And, sure enough, unsprisingly, teras offers a handy utility function compute_cardinalities for this purpose. Let’s import it as follows
from teras.utils import compute_cardinalities
categorical_idx = [idx for idx, col in enumerate(columns)
if col in categorical_columns]
cardinalties = compute_cardinalities(X_train.values,
categorical_idx=categorical_idx)
cardinalties # A value of zero indicates a continous feature
array([ 0, 9, 0, 16, 0, 7, 15, 6, 5, 2, 0, 0, 0, 42])
Now we’re ready to instantiate our backbone.
backbone = TabTransformerBackbone(input_dim=X_train.shape[1],
cardinalities=cardinalties,
embedding_dim=32)
Let’s print backbone model’s summary to see what’s going on under the hood. This is of great help when you want to get familiar with the underlying structure.
backbone.summary()
Model: "tab_transformer_backbone"
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ Connected to ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ inputs (InputLayer) │ (None, 14) │ 0 │ - │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ categorical_extrac… │ (None, 8) │ 0 │ inputs[0][0] │ │ (CategoricalExtrac… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ tab_transformer_co… │ (None, 8, 32) │ 2,916 │ categorical_extr… │ │ (TabTransformerCol… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ transformer_encode… │ (None, 8, 32) │ 252,288 │ tab_transformer_… │ │ (TransformerEncode… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ continuous_extract… │ (None, 6) │ 0 │ inputs[0][0] │ │ (ContinuousExtract… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ flatten (Flatten) │ (None, 256) │ 0 │ transformer_enco… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ layer_normalizatio… │ (None, 6) │ 12 │ continuous_extra… │ │ (LayerNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ concatenate │ (None, 262) │ 0 │ flatten[0][0], │ │ (Concatenate) │ │ │ layer_normalizat… │ └─────────────────────┴───────────────────┴────────────┴───────────────────┘
Total params: 255,216 (996.94 KB)
Trainable params: 255,216 (996.94 KB)
Non-trainable params: 0 (0.00 B)
Let’s now plug our backbone and classifier models
model = Classifier(backbone=backbone,
num_classes=1,
activation="sigmoid")
Before training our model, we need to compile it. In this compile step, we specify the loss function and the optimizer to use for the model.
import keras
model.compile(loss=keras.losses.BinaryCrossentropy(),
optimizer=keras.optimizers.RMSprop(),
metrics=[keras.metrics.BinaryAccuracy()])
Now we’re ready to train!
history = model.fit(X_train, y_train, epochs=2,
batch_size=512, validation_split=0.1)
Epoch 1/2
46/46 ━━━━━━━━━━━━━━━━━━━━ 54s 876ms/step - binary_accuracy: 0.7196 - loss: 0.6436 - val_binary_accuracy: 0.7973 - val_loss: 0.4587
Epoch 2/2
46/46 ━━━━━━━━━━━━━━━━━━━━ 22s 470ms/step - binary_accuracy: 0.7884 - loss: 0.4669 - val_binary_accuracy: 0.7939 - val_loss: 0.4549
And just like that, teras makes it super easy to make use of tabular deep learning!
If you have any questions or run into an issue, reach us at twitter @TerasML or file an issue at teras github repository.