Feature Engineering That Still Beats Deep Learning on Tabular Data

The current state of machine learning often creates the illusion that architectural complexity is a direct proxy for performance. In natural language processing and computer vision, this is largely true; the shift from handcrafted filters to deep residual networks and transformers has fundamentally changed what machines can perceive. However, the practitioner working with relational databases, financial transactions, or user logs quickly discovers that tabular data is a different beast. Unlike the spatial locality of pixels or the sequential coherence of words, tabular features often lack a natural global structure. A column for user_age and a column for transaction_amount do not have a geometric relationship that a standard convolutional layer or self-attention mechanism can inherently exploit without massive amounts of data.

In production environments, the promise of "end-to-end" deep learning for tables frequently falls short of Gradient Boosted Decision Trees (GBDTs) like XGBoost, LightGBM, or CatBoost. The reason is rarely the algorithm itself, but rather the data representation. While deep learning models struggle with the non-smooth, discontinuous nature of tabular decision boundaries, manual feature engineering allows the practitioner to inject domain-specific inductive biases directly into the model. By transforming raw signals into meaningful ratios, temporal deltas, and aggregated statistics, we provide the model with the high-level concepts it would otherwise take millions of extra rows to learn. This article explores the specific engineering techniques that maintain the performance gap over deep learning.

The structural limitations of deep learning for tables

Neural networks are universal function approximators, but they are biased toward smooth functions. In a typical tabular dataset, the relationship between a feature and the target is often piecewise or contains sharp thresholds. For example, a credit score threshold at 620 might trigger a binary change in loan eligibility. A multilayer perceptron (MLP) requires multiple layers and specific activations to approximate this simple step function, whereas a decision tree can isolate it with a single split. This fundamental difference in how models partition the feature space explains why trees are more robust to the noise and outliers common in CSV-style data.

Another significant hurdle is feature scale and distribution. Neural networks are notoriously sensitive to the scale of input features, necessitating meticulous normalization or standardization. In contrast, tree-based models are invariant to monotonic transformations of the input. Whether a feature is expressed as x or log(x), the tree will produce the same split point. This robustness reduces the preprocessing overhead and prevents the model from being led astray by skewed distributions or unscaled continuous variables. When we talk about feature engineering, we are often talking about simplifying the search space for the optimization algorithm.

Finally, the rotational invariance of neural networks is actually a disadvantage in the tabular domain. A neural network treats a rotation of the coordinate system as fundamentally the same problem. However, in tabular data, the individual axes (columns) carry unique, non-interchangeable meanings. The relationship between zip_code and income is not something that should be rotated or projected into a latent space without careful consideration. By manually engineering features, we preserve the semantic identity of the data while highlighting the interactions that matter most for the specific prediction task.

Temporal aggregations and windowing

Time is often the most important dimension in tabular data, yet deep learning models frequently struggle to capture long-term dependencies without complex recurrent or transformer architectures that are difficult to tune. Feature engineering allows us to flatten temporal dynamics into a single row using windowed aggregations. Instead of feeding a sequence of transactions into a 1D-CNN, we can calculate the mean, std, and count of transactions over the last 1, 7, and 30 days. This provides the model with a clear view of both short-term volatility and long-term trends.

Beyond simple averages, we should look at velocity and acceleration in user behavior. A feature representing the ratio of the current transaction amount to the average transaction amount over the last 90 days is a powerful signal for fraud detection. We call these "recency, frequency, monetary" (RFM) features. Implementing these requires efficient data pipelines, often utilizing window functions in SQL or transform methods in pandas. The key is to avoid data leakage by ensuring that the aggregation window only includes data points that occurred strictly before the target event's timestamp.

We also need to consider time-since-event features. For instance, days_since_last_login or seconds_since_page_refresh provide a sense of urgency or staleness that raw timestamps cannot. These features turn absolute time into relative duration, which is much easier for a gradient boosting model to correlate with an outcome like churn. When deep learning attempts to learn these through embeddings or positional encodings, it often requires a magnitude more data to reach the same level of precision that a simple subtraction operation provides during the feature engineering phase.

Encoding categorical variables for tree-based models

High-cardinality categorical features—those with hundreds or thousands of unique values like product_id or merchant_name—are a common roadblock. One-hot encoding these features creates sparse, high-dimensional matrices that slow down tree construction and lead to overfitting. Deep learning models use embedding layers to map these to continuous vectors, but this adds significant complexity to the model architecture and deployment pipeline. A more efficient alternative for GBDTs is Target Encoding (or Mean Encoding).

Target encoding replaces a categorical label with the average value of the target variable for that category. For a binary classification problem, merchant_id='Amazon' might be replaced by 0.02, representing a 2% fraud rate for that merchant. To prevent overfitting, specifically target leakage, it is vital to use K-fold smoothing or leave-one-out techniques. This collapses a high-dimensional space into a single, highly informative continuous feature that the tree can split on efficiently.

Count encoding is another underrated technique. Simply replacing a category with its frequency in the dataset helps the model distinguish between common and rare occurrences. In many real-world scenarios, rare categories behave differently than frequent ones. For example, a rare browser version might be more correlated with bot traffic. By providing the count as a feature, the model doesn't have to learn the frequency through multiple splits; it is given the information directly.

Cross-feature interactions and domain-specific ratios

While deep learning layers are designed to learn interactions between features automatically, they are not always successful at finding the specific mathematical relationships that define a domain. In finance, the debt-to-income ratio is more predictive than either debt or income in isolation. In e-commerce, click-through rate (clicks divided by impressions) is the standard metric for relevance. Creating these ratios manually ensures that the model focuses on the most mathematically sound relationships from the start.

We can also use automated methods to find interactions, such as creating polynomial features or using a DecisionTreeRegressor to find the most important pairs of features. However, the most effective interactions usually come from understanding the physics or the business logic of the problem. If you are predicting house prices, the product of square_footage and quality_score is a logical interaction. If you are predicting flight delays, the interaction between origin_airport and weather_condition is essential.

One specific technique involves using groupby operations to create context-aware features. For example, price_relative_to_category_average compares an item's price to the average price within its specific category. This tells the model if an item is expensive or cheap relative to its peers, which is a much stronger signal than the raw price. This type of engineering effectively "centers" the data around meaningful clusters, making the optimization landscape much flatter for the booster.

Handling missing values and outliers

Deep learning models generally require complete data, forcing practitioners to use imputation methods like mean, median, or K-Nearest Neighbors. Imputation introduces bias and can mask the fact that data is missing for a specific reason (e.g., a user declining to provide their income). Feature engineering allows us to treat "missingness" as a signal itself. By creating a binary indicator feature is_missing_income, we allow the model to learn the specific behavior associated with users who withhold information.

Tree-based models have a native advantage here: they can learn a default direction for missing values during the split process. However, engineering still plays a role. If a feature is missing because of a sensor failure, we might want to impute it with a value that signifies an error state. If it is missing because of a logic flow (e.g., days_since_last_claim for a new customer), we might replace it with a sentinel value like -1 or a very large number that the tree can easily isolate.

Outliers also require careful handling. While trees are robust to outliers in the features, they can still be affected by outliers in the target variable (especially in regression). Engineering the target through a log(y + 1) transformation can stabilize the variance and prevent the model from being dominated by a few extreme values. On the input side, clipping or winsorizing extreme values at the 1st and 99th percentiles can prevent the model from learning noise in the tails of the distribution.

The efficiency trade-offs of engineering vs. architecture

A common argument for deep learning is that it saves the time spent on manual engineering. This is often a false economy. The time saved in feature creation is frequently lost to hyperparameter tuning, architecture search, and the infrastructure required to train and serve heavy neural models. A well-engineered GBDT model can often be trained on a single CPU in minutes, whereas a tabular transformer might require a GPU and hours of compute time for similar results.

In production, latency is a critical factor. Calculating 50 engineered features in a streaming pipeline (using Redis for stateful lookups) and running a LightGBM inference takes only a few milliseconds. In contrast, passing data through a 10-layer deep network involves significantly more floating-point operations. For high-throughput systems like ad-bidding or high-frequency trading, the simplicity of feature engineering combined with fast tree inference is the only viable path.

Furthermore, the debuggability of engineered features is vastly superior. When a model makes a wrong prediction, you can inspect the specific features—like total_spent_last_hour—and determine if the data pipeline is broken or if the feature itself is no longer predictive. In a deep learning model, the error is buried in a latent representation that is almost impossible to interpret without specialized tools like SHAP or Integrated Gradients, which even then provide only an approximation of the truth.

Feature engineering is the process of mapping the raw data into a space where the relationship with the target becomes simpler, more linear, or more separable, reducing the burden on the model architecture.

Common mistakes in feature engineering

The most frequent and damaging mistake is data leakage. This occurs when information from the future or information that directly contains the target variable is included in the training features. For example, if you are predicting if a customer will churn in month 6, including their total spend in month 6 as a feature is leakage. Leakage leads to inflated cross-validation scores and disastrous real-world performance. Always ensure your feature generation logic respects the temporal boundary of the prediction point.

Another mistake is feature proliferation without selection. It is tempting to create every possible ratio and aggregation, but this leads to the curse of dimensionality. Redundant features increase the chance of overfitting and make the model harder to maintain. Use feature importance scores and permutation importance to prune your feature set. If two features are 99% correlated, keep the one that is more interpretable or easier to compute.

Lastly, practitioners often forget to account for covariate shift. Features that were predictive in the training set might change their distribution in production. For example, a feature based on a specific marketing campaign ID will become useless once that campaign ends. Engineering features that are more general—like days_since_campaign_start instead of the campaign_id itself—can make the model more resilient to these shifts over time.

What to practise this week

To master these techniques, you must move beyond the standard Titanic or Iris datasets and work with time-series or transactional data where engineering truly shines. Focus on building pipelines that can handle high-cardinality data and temporal shifts.

• Take a transactional dataset and implement five different windowed aggregations (e.g., 7-day rolling mean, 30-day max) using the pandas or polars rolling API.
• Implement a Target Encoder from scratch, including a smoothing parameter to handle categories with very few observations.
• Compare the performance of an XGBoost model using raw features versus one using the same features plus five domain-specific ratios you've identified.
• Practice identifying data leakage by intentionally introducing a future-dated feature and observing how it artificially inflates your validation metrics.
• Build a feature selection pipeline using Recursive Feature Elimination (RFE) to reduce a large feature set down to the most impactful 20%.

Ultimately, the goal of feature engineering is to make the learning task as easy as possible for the algorithm. While deep learning will continue to evolve, the ability to translate domain knowledge into numerical features remains the most valuable skill in a data scientist's toolkit when dealing with the structured data that runs the world's businesses.