ngosangns Knowledge Base

XGBoost Tips And Tricks

#technology#xgboost#machine-learning#kaggle

Updated: 2026-09-24

Source

Source: XGBoost Tips and Tricks — Chris Deotte (cdeotte), Kaggle Writeup, 11/27/2025. Deotte writes from competition work, and the writeup mixes his own results with numbers from NVIDIA's team. The 250x and 25x speedups for multiple GPUs come from NVIDIA's runs, not from the author's cluster. QuantileDMatrix notes refer to the v2.0 and v3.0 releases. Parameter defaults and ranges are as the writeup states them. Feature engineering gets more room than tuning, and cuDF groupby runs up to ~50x faster. The closing list repeats the main point of each section in one line.

Overview

The article distills years of experience using XGBoost to win Kaggle competitions and ship real-world models, split into 5 areas: data science foundations, XGBoost fundamentals, building/optimizing models, scaling to large data, and deployment/inference.

Data Science Foundations (apply to every problem, not just XGB)

  • Fast experimentation: the key to success is to shorten the preprocess → feature engineering → train → infer → evaluate loop as much as possible, so more ideas can be tried each day. The #1 way to speed up is running on GPU instead of CPU — use cuDF/cuML to accelerate dataframe operations and model train/infer; for XGBoost you just add "device": "cuda".
  • Reliable internal validation: use KFold to make use of all the training data when evaluating. What matters is designing the folds so they correctly mimic the real relationship between train and test — for example, if the test set contains patients never seen before, use GroupKFold by patient; if the test is a time series occurring after train, validation must also split by time in the same way.
  • EDA (Exploratory Data Analysis): the deeper your understanding of the data and the feature–target relationships, the easier it is to design suitable feature engineering and model architecture.

XGBoost Fundamentals

XGBoost is essentially an ensemble of decision trees, where each subsequent tree is fit to correct the errors (residuals) of the previous trees. Two properties to remember:

  • A decision tree only cares about the ordering of numeric values, not their specific distribution.
  • A decision tree cannot extrapolate beyond the range of values seen during training.

There are 2 main APIs for using XGBoost:

  • Native Python API (xgb.DMatrix + xgb.train): more advanced features (learning rate that changes per round, callbacks, continued/incremental training...) but more complex for beginners.
  • Scikit-Learn API (XGBRegressor/XGBClassifier with .fit/.predict): convenient to use with sklearn pipelines/GridSearchCV, no need to build a DMatrix yourself, but it doesn't expose all of the native API's advanced features.

Building and Optimizing Models

A nearly free baseline: one of XGBoost's strengths is that you can train right away without preprocessing — leave missing values, categorical, and numeric columns as they are — whereas many other models require imputing/encoding/normalizing first. The usual flow: run KFold, create a DMatrix in each fold (with enable_categorical=True), train with early_stopping_rounds so it stops at the right time, then take the out-of-fold predictions to evaluate (e.g., AUC).

Hyperparameters — no need to worry too much: the default parameter set is already quite good; you only need to adjust a few main "knobs":

  • objective, eval_metric — define the problem type.
  • learning_rate — start at ~0.1, then lower it after optimizing everything else to squeeze out extra performance.
  • device: cuda — turn on the GPU for medium/large datasets to speed things up.
  • The two most important knobs are max_depth (try 3 to 12, default 6) and colsample_bytree (try 0.3 to 0.9, default 0.8), along with subsample (default 0.8). Tuning just the 2 knobs max_depth and colsample_bytree can already reach more than 95% of XGBoost's achievable performance.
  • Only when you want to squeeze out a bit more performance should you go deeper into regularization (min_child_weight, gamma, lambda, alpha, scale_pos_weight...) or other parameters (grow_policy, max_leaves, tree_method, max_bin), tuned by hand or with Optuna.

Feature engineering is where the biggest difference is made: the author spends most of the time here instead of tuning hyperparameters. The most powerful technique is creating many new categorical features and then encoding them, especially by grouping by a categorical column and then aggregating a statistic of a numeric column (mean, quantile, histogram bins...). When the aggregated statistic is the target itself, the technique is called Target Encoding — be careful to avoid leakage. Common feature transformation directions: binning numeric→categorical, combining/splitting columns, then encoding with one-hot/label/target/count encoding. Many recent Kaggle wins came from feature engineering alone (binning, combining columns, groupby-aggregate, target encoding); cuDF speeds up groupby by up to ~50x, allowing thousands of feature ideas to be tried faster.

Scaling XGBoost to Large Data

Three main techniques:

  1. Reduce dtypes to the smallest size needed to save RAM/VRAM.
  2. QuantileDMatrix (XGBoost v2.0/v3.0) in place of a regular DMatrix, allowing training on larger datasets without increasing RAM/VRAM thanks to better memory management; ExtMemQuantileDMatrix pushes this limit even further (using the same custom data loader/iterator).
  3. Dask-XGBoost to use multiple GPUs at once: create a Dask LocalCluster/Client, and use DaskDMatrix and xgb.dask.train/xgb.dask.predict in place of the single-GPU API.

According to the article, by combining dtype reduction + QuantileDMatrix + Dask-XGBoost, NVIDIA's team achieved 250x speedup (4 GPUs vs 1 CPU) and 25x (4 GPUs vs 20 CPUs), making experiments faster and winning many RecSys competitions.

Deployment and Inference

  • NVIDIA cuML Forest Inference Library (FIL): load a trained model (model.ubj/model.json) into ForestInference to speed up GPU inference; you can call optimize(batch_size=...) to auto-tune for the actual batch size before predict/predict_proba.
  • Refit on Full Data: after finding the optimal hyperparameters with KFold, retrain a single model on 100% of the training data (instead of keeping the K models from K-fold). A model that uses 100% of the data is typically better than one that only saw (K-1)/K of the data, and serving needs just 1 model instead of K. The number of training rounds should be scaled by the ratio K/(K-1) relative to the optimal number found during KFold. This is a common trick for boosting leaderboard scores on Kaggle.

Quick Summary

  • Data science foundations: fast experimentation (prefer GPU) + validation that correctly mimics the train/test relationship.
  • XGBoost fundamentals: only cares about the ordering of numeric values; doesn't extrapolate beyond the trained range.
  • Build & optimize: the 2 most important knobs are max_depth and colsample_bytree; investing time in feature engineering (especially categorical/target encoding) pays off more than tuning hyperparameters.
  • Scale: reduce dtype → DMatrix → QuantileDMatrix → ExtMemQuantileDMatrix → Dask-XGBoost when multiple GPUs are needed.
  • Deploy & inference: use cuML FIL to speed up predict; refit the final model on 100% of the training data.

See also: Loss Functions · Monitoring Tracking · Data Analytics Tools