Guide to Scalable and Robust Bayesian Optimization with Dragonfly
Dragonfly, an open-source python framework for scalable and robust Bayesian optimization, is developed by researchers from Carnegie Mellon University, Pittsburgh : Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas Poczos, Eric P. Xing. The paper was submitted to Journal of Machine Learning Research in April 2020 titled “ Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly. Machine Learning tasks are cast as black-box problems, where we have given some input and black-box does some processing and outputs the value for the given input. To enhance the performance of such a black-box system, we tend to optimize it to its optimum. An example of it is Hyperparameter Tuning where we search for the best parameters to find the best model for a given task. According to domain knowledge, this searching is often done by grid search or random search or manual settings(according to the domain knowledge). Formally, we define a black box as a function f : X → R over some domain X and try to find the optimum value of f via evaluating it repeatedly. Each evaluation, in this case, is expensive and requires huge computation. That’s when Bayesian Optimization(BO) comes into the picture. BO maximises the function f with a few evaluations by calculating the next point’s evaluation via previous observation. In probability terms, in order to determine the next evaluation, given the current evaluations as a point to start for the next evaluation, BO uses posterior. BO also showed some promising results in some hyperparameters tasks. Optimization tasks in today’s applications are posed to new threats that are difficult to tackle by traditional approaches. Dragonfly is primarily focused on scalable Bayesian Optimization and its robustness. • Scalability: This includes handling higher dimensional domains, multi-fidelity evaluations, optimising over neural network architectures, handling parallel evaluations. Apart from this, Dragonfly also contains some evolutionary algorithms that are capable of optimizing complex domains. • Robustness: Includes a set of values for a set of acquisition choices and model parameters instead of one for the entire optimization process. Hence, lowering the computational cost and adding robustness to the optimization system. Requirements & Installation Dragonfly is supported by Python 2(>=2.7) and Python 3(>=3.6) as well. It is fully compatible with Mac, Linux & Windows: You can install dragonfly via PyPI. pip install dragonfly-opt -v To check the installation of Dragonfly by following code : #installation testing from dragonfly import minimise_function # The first argument below is the function, the second is the domain, and the third is the budget. min_val, min_pt, history = minimise_function(lambda x: x ** 4 - x**2 + 0.1 * x, [[-10, 10]], 10); print(min_val, min_pt) Quick Start with Dragonfly Dragonfly is supported by command line, python and ask-tell mode. This guide will contain all of the three approaches to dragonfly. 1. Dragonfly with Command line Since, we can run command line on Colab Notebook, this demo contains the code for colab command line. Dragonfly can be used explicitly by calling dragonfly-script.py or by importing python code through maximize_function in the main library. Examples are given here in this directory. If you have installed dragonfly via pip, you have to clone the git repository to access the examples. To use dragonfly through the command line, specify the optimization problem and the optimization parameters(can be specified in JSON (recommended) or protocol buffer format). You can check  examples/synthetic/branin/config.json and examples/synthetic/branin/config.pb for examples. Then, specify the optimisation parameters in an options file, in the format shown in examples/options_example.txt. Basic Usage : The example shown below is the Branin benchmark for global optimization. %cd examples !python /content/dragonfly/bin/dragonfly-script.py --config synthetic/branin/config.json --options options_files/options_example.txt !python /content/dragonfly/bin/dragonfly-script.py --config synthetic/branin/config.pb --options options_files/options_example.txt Minimization : By default setting, dragonfly is set to maximize the function. But we can change that setting by usingthe  max_or_min flag. You can check the example here. !python dragonfly-script.py --config synthetic/branin/config.json --optionsoptions_files/options_example_for_minimisation.txt Multi-fidelity Optimisation : Multi fidelity version of Branin benchmark is shown below #multi-fidelity optmization !python dragonfly-script.py --config synthetic/branin/config_mf.json --options options_files/options_example.txt You can find all types of optimization that can be run through cli here. 2. Dragonfly with Python Code API’s for dragonfly are initialized in dragonfly/__init__.py and are defined in the dragonfly/apis folder. For example of using dragonfly API in python: from dragonfly import minimise_function, maximise_function func = lambda x: x ** 4 - x**2 + 0.1 * x domain = [[-10, 10]] max_capital = 100 min_val, min_pt, history = minimise_function(func, domain, max_capital) print(min_val, min_pt) max_val, max_pt, history = maximise_function(lambda x: -func(x), domain, max_capital) print(max_val, max_pt) Here, func : It is the optimization function which is to be maximized. domain : Over which the above func needs to be maximized. max_capital : capital available for optimization(max. Number of available functions depends on cost or other constraints). maximise_function : Returns the history with optimum value and the optimum point. history.query_points : Represents the points evaluated by algorithm history.query_vals : It contains the function values. You can check other examples as well : Branin Example for maximizationHartmann BenchmarkExample of discrete euclidean domains Example of Constrained Hartmann Benchmarkborehole constrained exampleuse case for random forest regression and gradient boosted regression A demo of neural architecture search using the NASBOT algorithm on an MLP   (Multi-Layer Perceptron) architecture search problem Similarly, you call use multiple objective optimization API. The basic structure of it is shown below : from dragonfly import multiobjective_maximise_functions pareto_max_values, pareto_points, history = multiobjective_maximise_functions(funcs, domain, max_capital) where, funcs :  list of functions to be maximised, pareto_values :  Pareto optimal function values pareto_points :  The corresponding points in domain. You can check multi-objective branin example and multi-objective hartmann example. You can check all other optimization available in dragonfly for python here. 3. Dragonfly with Ask-tell It  enables step-by-step optimization by calculating the next point to be evaluated in a loop of BO. Two main components of this interface are: A function caller  EuclideanFunctionCaller : Used when domain is limited to euclidean spacesCPFunctionCaller : else use this callerAn optimizer.  Here, <domain> is replaced by Euclidean or CP(depending on the domain).<domain>GPBandit<domain>GAOptimiser<domain>RandomOptimiser An example of it is shown below: from dragonfly.exd import domains from dragonfly.exd.experiment_caller import CPFunctionCaller, EuclideanFunctionCaller from dragonfly.opt import random_optimiser, cp_ga_optimiser, gp_bandit max_capital = 100 objective = lambda x: x[0] ** 4 - x[0]**2 + 0.1 * x[0] domain = domains.EuclideanDomain([[-10, 10]]) func_caller = EuclideanFunctionCaller(None, domain) opt = gp_bandit.EuclideanGPBandit(func_caller, ask_tell_mode=True) #call initialise() on the created optimizer to begin using the interface opt.initialise() for i in range(max_capital): #ask returns the next point to be evaluated in a numpy array. x = opt.ask() y = objective(x) print("x:", x, ", y:", y) #Once the point is evaluated by calling the objective function, #tell the point back to the optimiser opt.tell([(x, y)]) The above example represents the maximization problem. For minimization, take the negative of the objective function and rest of the procedure is same. Check all the details here. Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with DragonflyK. Kandasamy et al.Python Library: https://t.co/xfyMkVs6Z3Docs: https://t.co/vf8jXVkP5o pic.twitter.com/ZMlOGvyhRT— Brandon Amos (@brandondamos) March 18, 2019 Conclusion In this post, we have given an overview of Dragonfly, an optimized framework for  Bayesian Optimization that is scalable and robust as well. This framework is built keeping in mind the optimization of expensive optimization functions. It outperformed many existing systems available for BO. Colab Notebook Dragonfly Demo Official codes, docs & Tutorial are available at : GithubResearch PaperDocumentationExamples
Dragonfly, an open-source python framework for scalable and robust Bayesian optimization, is developed by researchers from Carnegie Mellon University, Pittsburgh : Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas Poczos, Eric P. Xing. The paper was submitted to Journal of Machine Learning Research in April 2020 titled “ […]
Dragonfly, an open-source python framework for scalable and robust Bayesian optimization primarily focused on scalability & robustness
["Bayesian optimisation", "evolutionary algorithm", "hyperparameter optimisation", "hyperparameter tuning", "optimisation algorithms", "Python framework"]
["hyperparameter optimisation", "NumPy", "machine learning", "TPU", "Python framework", "AI", "neural network", "ML", "Bayesian optimisation", "hyperparameter tuning", "Colab", "Ray", "RAG", "optimisation algorithms", "Python", "evolutionary algorithm"]
["AI", "machine learning", "ML", "neural network", "Ray", "Colab", "NumPy", "RAG", "TPU", "Python"]
A data science enthusiast and a post-graduate in Big Data Analytics. Creative and organized with an analytical bent of mind.