Feature Engineering

The art and science of transforming raw data into predictive signals.

What is Feature Engineering?

Feature Engineering is the process of using domain knowledge to create, select, and transform variables (features) from raw data that make machine learning algorithms work better. It's often considered the most impactful step in building ML models—more influential than the choice of algorithm itself.

💡 Key Insight: "Applied machine learning is basically feature engineering." – Andrew Ng

Good features can make a simple model outperform a complex model with poor features. The goal is to represent the underlying patterns in data in a way that algorithms can easily learn from.

🔢 Numerical Transformations

Raw numerical data often needs to be transformed to improve model performance and convergence.

Normalization (Min-Max Scaling)

Scales values to a range [0, 1]. Useful when you need bounded values or when using algorithms sensitive to magnitude (e.g., neural networks, KNN).

X_scaled = (X - X_min) / (X_max - X_min)

Standardization (Z-Score)

Centers data around mean=0 with std=1. Preferred for algorithms assuming normally distributed data (e.g., Linear Regression, SVM).

X_scaled = (X - mean) / std

Log Transform

Reduces right-skewness in data. Common for income, price, or count data with long tails.

X_log = np.log1p(X)  # log1p handles zeros

Binning / Discretization

Converts continuous variables into categorical bins. Useful for capturing non-linear relationships (e.g., age groups: 0-18, 19-35, 36-55, 55+).

df['age_group'] = pd.cut(df['age'], bins=[0, 18, 35, 55, 100], labels=['child', 'young', 'adult', 'senior'])

🏷️ Categorical Encoding

Machine learning algorithms require numerical input. Categorical variables must be encoded properly.

Method	When to Use	Example
One-Hot Encoding	Low cardinality, no ordinal relationship	Color: Red→[1,0,0], Blue→[0,1,0]
Label Encoding	Ordinal categories (e.g., Low < Medium < High)	Low→0, Medium→1, High→2
Target Encoding	High cardinality, uses target variable mean	City → Avg. target for that city
Frequency Encoding	High cardinality, preserves popularity info	Category → Count / Total
Binary Encoding	Medium-high cardinality, memory efficient	Label → Binary representation

⚠️ Warning: Avoid using Label Encoding for non-ordinal categories in linear models—it implies an artificial order that can mislead the model.

📅 Date/Time Feature Engineering

Datetime columns are goldmines of information. Extract these features to capture temporal patterns.

Basic Extractions

Year, Month, Day
Hour, Minute, Second
Day of Week (0=Monday)
Week of Year
Quarter

Derived Features

Is Weekend (Sat/Sun)
Is Holiday
Is Business Hour (9-17)
Days Since Event
Time Until Next Event

# Cyclical Encoding for periodic features (e.g., hour, day of week)
# Prevents the model from treating midnight (23) as far from 1am (1)
import numpy as np

df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)

df['day_sin'] = np.sin(2 * np.pi * df['dayofweek'] / 7)
df['day_cos'] = np.cos(2 * np.pi * df['dayofweek'] / 7)

📝 Text Feature Engineering

Text data requires special handling to convert unstructured information into numerical features.

Basic Text Statistics

Character count, word count, sentence count
Average word length
Number of uppercase letters, punctuation, digits
Number of stopwords, unique words

TF-IDF (Term Frequency-Inverse Document Frequency)

Weights words by importance: frequent in a document but rare across documents = high score.

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(max_features=1000)
X_tfidf = vectorizer.fit_transform(texts)

Word Embeddings (Word2Vec, GloVe, FastText)

Dense vector representations that capture semantic meaning. "King - Man + Woman ≈ Queen"

from gensim.models import Word2Vec
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)
vector = model.wv['machine']

Sentence Embeddings (SBERT, OpenAI)

Modern approach: embed entire sentences/documents into dense vectors for semantic similarity.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(["This is a sentence", "Another one"])

🔗 Feature Interactions

Sometimes the combination of two features is more predictive than either alone.

# Arithmetic interactions
df['price_per_sqft'] = df['price'] / df['sqft']
df['rooms_per_person'] = df['rooms'] / df['occupants']
df['age_income'] = df['age'] * df['income']

# Polynomial features (for capturing non-linear relationships)
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X[['feature_a', 'feature_b']])

💡 Tip: Use domain knowledge to guide feature interactions. For real estate: price per sqft is meaningful. For e-commerce: clicks / views = conversion rate.

🎯 Feature Selection

More features ≠ better model. Irrelevant or redundant features can hurt performance. Selection methods include:

Filter Methods

Statistical tests independent of model

Correlation matrix
Chi-squared test
Mutual Information
ANOVA F-test

Wrapper Methods

Use model performance to select

Forward Selection
Backward Elimination
Recursive Feature Elimination (RFE)

Embedded Methods

Built into model training

L1 Regularization (Lasso)
Tree-based importance
XGBoost feature importance

# Example: Feature Importance with Random Forest
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train)

# Get feature importances
importances = pd.DataFrame({
    'feature': X_train.columns,
    'importance': model.feature_importances_
}).sort_values('importance', ascending=False)

print(importances.head(10))

🕳️ Handling Missing Values

Missing data is common. The strategy depends on the nature of the missingness.

Strategy	When to Use	Code
Drop rows	Few missing values, sufficient data	`df.dropna()`
Mean/Median imputation	Numerical, MCAR	`df.fillna(df.mean())`
Mode imputation	Categorical	`df.fillna(df.mode()[0])`
KNN Imputer	Uses similar rows to fill	`KNNImputer(n_neighbors=5)`
Add "is_missing" indicator	Missingness itself is informative	`df['col_missing'] = df['col'].isna()`

✅ Best Practices

Do ✓

Fit transformers on training data only
Use pipelines to prevent data leakage
Document your feature engineering steps
Validate with cross-validation
Use domain knowledge

Don't ✗

Fit scalers on full dataset (data leakage!)
Use target info in feature creation (leakage!)
Create too many features without selection
Ignore feature distributions
Overlook multicollinearity