A feature cross is a technique in machine learning where two or more individual features (variables) are combined to create a new, composite feature. This new feature captures interactions between the original features that may not be apparent when each is considered separately. Feature crosses are especially useful when certain combinations of feature values provide important signals for a predictive model, but those signals are hard to capture with the original features alone.
For example, suppose a dataset contains two categorical features: “city” and “month.” Individually, each feature might impact a model’s prediction, but the combination (such as “Boston in January” vs. “Boston in July”) could reveal unique patterns—like seasonal trends in sales or weather behavior. By crossing these features, a new feature is created representing every possible pair of city and month, allowing the model to learn these specific interactions.
Feature crosses can be created in several ways. The most common method is to concatenate the values of two or more features, forming a new categorical value. In deep learning, feature crosses are sometimes implemented using embedding layers, which map high-cardinality categorical crosses into a dense, continuous space. In tree-based models like decision trees and random forests, feature crosses can improve performance by explicitly representing these interactions in the feature set.
The primary benefit of feature crossing is that it allows models to learn non-linear relationships between variables without requiring the model itself to be highly complex. In traditional linear models, feature crosses expand the hypothesis space by giving the model access to higher-order interactions. However, feature crossing can also increase the dimensionality of the data, especially when crossing high-cardinality features. This can lead to increased computational requirements and a higher risk of overfitting if not managed carefully.
Feature crosses are often created as part of the broader process of feature engineering, where data scientists manually construct new features to help models learn more effectively. Some modern machine learning frameworks, such as TensorFlow’s Wide & Deep models, automate feature crossing as part of the model architecture. This helps combine the memorization ability of wide models (which benefit from many explicit feature crosses) with the generalization power of deep neural networks.
In summary, a feature cross is a practical method for surfacing important interactions in your data, making it easier for machine learning models to capture complex relationships. Used thoughtfully, feature crosses can boost model accuracy, especially in problems where combinations of categorical or discrete features are significant.
