Hyperparameters are configuration settings that control how a machine learning algorithm learns or how complex its model can become. Unlike model parameters, which are learned from training data, hyperparameters are usually selected by the developer or an automated search procedure.
Examples include the learning rate in gradient descent, the number of neighbors in k-nearest neighbors, the maximum depth of a decision tree, the number of trees in a random forest, and the regularization strength in linear models. The appropriate values depend on the dataset and learning task.
Hyperparameter tuning evaluates different configurations to identify those that perform well on validation data. Grid search tests every combination in a predefined set of values. Random search samples configurations from specified distributions and can explore large search spaces more efficiently when only a few hyperparameters strongly influence performance. Bayesian optimization uses previous evaluation results to guide the selection of promising configurations.
Cross-validation is often used to obtain more reliable estimates during tuning, especially when the dataset is limited. However, repeatedly selecting configurations based on the same validation data can lead to overfitting to the validation set. Nested cross-validation or a separate final test set can help provide a more independent performance estimate.
Hyperparameter tuning should use sensible search ranges and appropriate evaluation metrics. Computational cost, model complexity, reproducibility, and the practical requirements of deployment should also be considered.
Hyperparameter Tuning
Hyperparameter tuning is the process of selecting suitable settings that control a machine learning algorithm's behavior before or during training. It helps improve validation performance and model generalization.
Explanation
Variables
| Symbol | Meaning |
|---|---|
η | Learning rate, a hyperparameter controlling the size of optimization updates |
k | Number of neighbors in a k-nearest neighbors model or number of folds in cross-validation, depending on context |
d | Maximum tree depth when used as a decision-tree hyperparameter |
λ | Regularization strength in some machine learning models |
Example or application
A developer trains a decision tree classifier and wants to choose its maximum depth. They test depths of 3, 5, 8, and 12 using five-fold cross-validation. The configuration with depth 8 achieves the strongest average validation F1-score and is selected for further evaluation on an untouched test set.
Text
Hyperparameter tuning selects suitable algorithm settings that are not learned directly as ordinary model parameters. Examples include learning rate, tree depth, number of neighbors, and regularization strength. Grid search tests predefined combinations, random search samples configurations, and Bayesian optimization uses previous results to guide future evaluations. Validation data or cross-validation is used to compare configurations, while a separate test set should be reserved for final evaluation.
Key Points
Hyperparameters control how a learning algorithm behaves.
Model parameters are learned from data; hyperparameters are selected separately.
Grid search evaluates predefined combinations.
Random search samples configurations from a search space.
Bayesian optimization uses earlier results to guide later trials.
Cross-validation can provide more reliable tuning estimates.
Repeated validation-based selection can overfit the validation data.
The final test set should not be used as a tuning dataset.
Applications
Selecting neural network learning rates
Choosing decision tree depth
Optimizing random forest configurations
Selecting regularization strength
Improving support vector machine performance
Tuning machine learning pipelines
Learning Outcomes
Distinguish model parameters from hyperparameters.
Identify common hyperparameters in machine learning algorithms.
Explain grid search, random search, and Bayesian optimization.
Describe the role of validation data and cross-validation in tuning.
Practice Questions
What is the difference between a model parameter and a hyperparameter?
How does random search differ from grid search?
Why should the test dataset remain separate during hyperparameter tuning?
Related Concepts
Model Training
Cross-Validation
Regularization
Learning Rate
Model Selection
Common Mistakes
Confusing learned model weights with manually selected hyperparameters.
Choosing configurations based on test-set performance.
Searching excessively large parameter ranges without a clear strategy.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Bergstra, J., and Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. Springer.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning