10 Hyperparameter optimization frameworks.

Original article was published by Sivasai Yadav Mudugandla on Artificial Intelligence on Medium

10 Hyperparameter optimization frameworks.

Tune your Machine Learning models with open-source optimization libraries

Image by Jörg Felix from Pixabay


Hyper-parameters are the parameters used to control the behavior of the algorithm while building the model. These parameters cannot be learned from the regular training process. They need to be assigned before training the model.

Example: n_neighbors (KNN), kernel (SVC) , max_depth & criterion (Decision Tree Classifier) etc.

Hyperparameter optimization or tuning in machine learning is the process of selecting the best combination of hyper-parameters that deliver the best performance.

Various automatic optimization techniques exist, and each has its own strengths and drawbacks when applied to different types of problems.

Example: Grid Search, Random Search, Bayesian Search, etc.

Scikit-learn is one of the frameworks we could use for Hyperparameter optimization, but there are other frameworks that could even perform better.

  1. Ray-Tune
  2. Optuna
  3. Hyperopt
  4. mlmachine
  5. Polyaxon
  6. BayesianOptimization
  7. Talos
  9. Scikit-Optimize
  10. GPyOpt

1. Ray-Tune

Tune is a Python library for experiment execution and hyperparameter tuning at any scale.

Key Features

  1. Launch a multi-node distributed hyperparameter sweep in less than ten lines of code.
  2. Supports any machine learning framework, including PyTorch, XGBoost, MXNet, and Keras.
  3. Choose among the state of the art algorithms such as Population Based Training (PBT), BayesOptSearch, HyperBand/ASHA.
  4. Tune’s Search Algorithms are wrappers around open-source optimization libraries such as HyperOpt, SigOpt, Dragonfly, and Facebook Ax.
  5. Automatically visualize results with TensorBoard.

#Tune for Scikit Learn

Installation: pip install ray[tune] tune-sklearn

2. Optuna

Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning.

Key Features

  1. Easy parallelization
  2. Quick visualization
  3. Efficient optimization algorithms
  4. Lightweight, versatile, and platform-agnostic architecture
  5. Pythonic search spaces

Installation: pip install optuna

3. Hyperopt

Hyperopt is a Python library for serial and parallel optimization over awkward search spaces, which may include real-valued, discrete, and conditional dimensions.

Hyperopt currently it supports three algorithms :

Key Features

  1. Search space (you can create very complex parameter spaces)
  2. Persisting and restarting (you can save important information and later load and then resume the optimization process)
  3. Speed and Parallelization (you can distribute your computation over a cluster of machines)

Installation: pip install hyperopt

4. mlmachine

mlmachine is a Python package that facilitates clean and organized notebook-based machine learning experimentation and accomplishes many key aspects of the experimentation life cycle.

mlmachine performs Hyperparameter Tuning with Bayesian Optimization on multiple estimators in one shot and includes functionality for visualizing model performance and parameter selections.

A well explained article on mlmachine.

Installation: pip install mlmachine

5. Polyaxon

Polyaxon is a platform for building, training, and monitoring large scale deep learning applications. It makes a system to solve reproducibility, automation, and scalability for machine learning applications.

The way Polyaxon performs hyperparameter tuning is by providing a selection of customizable search algorithms. Polyaxon supports both simple approaches such as random search and grid search, and provides a simple interface for advanced approaches, such as Hyperband and Bayesian Optimization, it also integrates with tools such as Hyperopt, and provides an interface for running custom iterative processes. All these search algorithms run in an asynchronous way, and support concurrency and routing to leverage your cluster(s)’s resources to the maximum.

Key Features

  1. Easy-to-use: Polyaxon’s Optimization Engine is a built-in service and can be used easily by adding a matrix section to your operations, you can run hyperparameter tuning using the CLI, client, and the dashboard.
  2. Scalability: Tuning hyperparameters or neural architectures requires leveraging a large amount of computation resources, using Polyaxon you can run hundreds of trials in parallel and intuitively track their progress.
  3. Flexibility: Besides the rich built-in algorithms, Polyaxon allows users to customize various hyperparameter tuning algorithms, neural architecture search algorithms, early stopping algorithms, etc.
  4. Efficiency: We are intensively working on more efficient model tuning from both system-level and algorithm level. For example, leveraging early feedback to speedup tuning procedure.

Installation: pip install -U polyaxon

6. Bayesian Optimization

Bayesian Optimization is another framework that is a pure Python implementation of Bayesian global optimization with Gaussian processes. This is a constrained global optimization package built upon Bayesian inference and Gaussian process, that attempts to find the maximum value of an unknown function in as few iterations as possible. This technique is particularly suited for the optimization of high-cost functions, situations where the balance between exploration and exploitation is important.

Installation: pip install bayesian-optimization

7. Talos

Talos radically changes the ordinary Keras workflow by fully automating hyperparameter tuning and model evaluation. Talos exposes Keras functionality entirely and there is no new syntax or templates to learn.

Key Features

  1. Single-line optimize-to-predict pipeline talos.Scan(x, y, model, params).predict(x_test, y_test)
  2. Automated hyperparameter optimization
  3. Model generalization evaluator
  4. Experiment analytics
  5. Pseudo, Quasi, and Quantum Random search options
  6. Grid search
  7. Probabilistic optimizers
  8. Single file custom optimization strategies

Installation: pip install talos


SHERPA is a Python library for hyperparameter tuning of machine learning models.

It provides:

  1. hyperparameter optimization for machine learning researchers
  2. a choice of hyperparameter optimization algorithms
  3. parallel computation that can be fitted to the user’s needs
  4. a live dashboard for the exploratory analysis of results.

Installation: pip install parameter-sherpa

9. Scikit-Optimize

Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It implements several methods for sequential model-based optimization. skopt aims to be accessible and easy to use in many contexts. Scikit-Optimize provides support for tuning the hyperparameters of ML algorithms offered by the scikit-learn library, so-called hyperparameter optimization.

The library is built on top of NumPy, SciPy and Scikit-Learn.

Installation: pip install scikit-optimize

10. GPyOpt

GPyOpt is a tool for optimization (minimization) of black-box functions using Gaussian processes. It has been implemented in Python by the group of Machine Learning (at SITraN) of the University of Sheffield.GPyOpt is based on GPy, a library for Gaussian process modeling in Python. It can handle large data sets via sparse Gaussian process models.

  1. Bayesian optimization with arbitrary restrictions
  2. Parallel Bayesian optimization
  3. Mixing different types of variables
  4. Tuning scikit-learn models
  5. Integrating the model hyper parameters
  6. External objective evaluation

Installation: pip install gpyopt

Thank you for reading!

Any feedback and comments are, greatly appreciated!

Some of my other posts you may find interesting,