Automated Algorithm Design (AAD)

Introduction

Algorithms play a critical role in solving real-world problems. However, designing effective algorithms often involves a tedious trial-and-error process that requires expert knowledge.

Automated algorithm design has long been a goal for both researchers and practitioners, aiming to streamline this process and reduce reliance on manual intervention.

aad_tasks

A general automated algorithm design task can be formally defined as follows:

Given:

  • Problem Instance Distribution \(P^I\): A probability distribution representing the target problem instances (e.g., the Traveling Salesman Problem, TSP). In practice, this distribution is typically unknown. Therefore, we usually use \(InstS\): A set of problem instances, which are samples drawn from \(P^I\).

  • Algorithm Space \(A^S\): An algorithm space, representing the set of all possible algorithms under consideration.

  • Performance Metric \(f(a, i)\): A performance metric that evaluates the effectiveness of an algorithm \(a \in A^S\) on a specific problem instance \(i\).

Goal:

Objective: Find an algorithm \(a \in A^S\) that optimizes its expected performance over the problem instances. This can be expressed as:

\[F(a) := \mathbb{E}_{i \sim P^I} [f(a, i)].\]

In practice, since the true distribution $P^I$ is often unavailable, the goal is approximated using the available instances in $InstS$:

\[F(a) := \sum_{i \in InstS} f(a, i).\]

Representative Work from Optima Group

Learning for Optimization

Deep learning and various machine learning methodologies are employed for the automated design of algorithms. A particularly prominent area of focus is Neural Combinatorial Optimization (NCO), which involves training deep learning models to acquire heuristics for solving combinatorial optimization problems in an end-to-end manner. In our recent research, we have advanced multi-task cross-problem learning techniques specifically tailored for vehicle routing problems (VRPs) [1], explored prompt learning strategies for cross-distribution vehicle routing scenarios [2], and examined the supervised learning paradigm in this context [3]. Additionally, we are engaged in the development of Machine Learning-assisted Algorithm Design. Notably, we have introduced a machine learning-enhanced multiobjective solver designed to address complex real-world industrial routing and packing challenges [4].

Large Language Model (LLM)-driven AAD

Evolution of Heuristics (EoH)：A New Algorithm Design Paradigm

The Evolution of Heuristics [5] concept stems from our foundational work, AEL [6], which appeared online in Nov 2023. It is the first to integrate large language models within an evolutionary computation framework, serving as designers for automated algorithm design. Contrasted with a) traditional hand-crafted design methods and b) conventional machine learning-assisted algorithm design, it offers several key advantages:

  • Automation: It drastically minimizes the need for manual design and eliminates the requirement for model training.
  • Efficiency: It enables the parallel and efficient evolution of new algorithms and heuristics.
  • Performance: It provides novel insights into target algorithm design tasks and frequently generates new state-of-the-art (SOTA) heuristics.

eoh

We have successfully applied it on combinatorial optimization problems [5], acquisition function design for Bayesian optimization [7], attack strategy design in image adversary attack [8], turbulence model design for computational fluid dynamics [9],
and automated data augmentation for long-tailed problems [10]. In [11], we highlight the significance of evolutionary search in automated algorithm/heuristic/program search.

MEoH [12]: We propose the first LLM-based multi-objective heuristic search framework, Multi-objective Evolution of Heuristic (MEoH), which integrates LLMs in a zero-shot manner to generate a non-dominated set of heuristics to meet multiple design criteria. We design a new dominance-dissimilarity mechanism for effective population management and selection, which incorporates both code dissimilarity in the search space and dominance in the objective space.

CoEvo [13]: We propose a framework that utilizes LLMs in an evolutionary search methodology, augmented by a dynamic knowledge library that integrates and refines insights in an open-ended manner. This approach aims to tackle the dual challenges of efficiently navigating complex symbolic representation spaces and leveraging both existing and newly generated knowledge to foster open-ended innovation. By enabling LLMs to interact with and expand upon a knowledge library, we facilitate the continuous generation of novel solutions in diverse forms such as language, code, and mathematical expressions.

LLM4AD Platform

πŸš€LLM4AD is an open-source Python-based Platform leveraging Large Language Models (LLMs) for Automated Algorithm Design (AD). πŸ”₯It has won the second price in IEEE CIS FLAME Technical Challenge 2024. πŸ”₯It has been listed as one of the suggested integrations by DeepSeek.

  • Visit our GitHub Repo
  • Please refer to the paper for detailed information, including the overview, methodology, and benchmark results.
  • LLM4AD is accompanied with Documents and Examples materials to support users and developers to easily test, build and deploy their own LLM4AD applications and conduct research.
  • LLM4AD was originally developed for optimization tasks. The framework is versatile enough to be used in other areas, including machine learning, science discovery, game theory and engineering design.

llm4ad

Surveys / Tutorials / Special Sessions

Tutorial

Survey

llm4ad_survey

References

[1] Fei Liu, Xi Lin, Zhenkun Wang, Qingfu Zhang, Tong Xialiang, and Mingxuan Yuan. β€œMulti-task learning for routing problem with cross-problem zero-shot generalization.” SigKDD 2024.

[2] Fei Liu, Xi Lin, Weiduo Liao, Zhenkun Wang, Qingfu Zhang, Xialiang Tong, and Mingxuan Yuan. β€œPrompt learning for generalized vehicle routing.” IJCAI 2024.

[3] Shunyu Yao, Xi Lin, Jiashu Wang, Qingfu Zhang, and Zhenkun Wang. β€œRethinking Supervised Learning based Neural Combinatorial Optimization for Routing Problem.” ACM Transactions on Evolutionary Learning, 2024.

[4] Fei Liu, Qingfu Zhang, Qingling Zhu, Xialiang Tong, and Mingxuan Yuan. β€œMachine learning assisted multiobjective evolutionary algorithm for routing and packing.” IEEE Transactions on Evolutionary Computation, 2024.

[5] Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. β€œEvolution of Heuristics: Towards Efficient automated Algorithm Design Using Large Language Model.” ICML 2024. (Oral)

[6] Fei Liu, Xialiang Tong, Mingxuan Yuan, and Qingfu Zhang. β€œAlgorithm evolution using large language model.” arXiv:2311.15249, 2023.

[7] Yiming Yao, Fei Liu, Ji Cheng, and Qingfu Zhang. β€œEvolve Cost-aware Acquisition Functions using Large Language Models.” PPSN 2024. (Best Paper Nomination)

[8] Ping Guo, Fei Liu, Xi Lin, Qingchuan Zhao, and Qingfu Zhang. β€œL-autoda: Large language models for automatedally evolving decision-based adversarial attacks.” GECCO (Companion) 2024.

[9] Yu Zhang, Kefeng Zheng, Fei Liu, Qingfu Zhang, and Zhenkun Wang. β€œAutoTurb: Using large language models for automated algebraic turbulence model discovery.” Physics of Fluids, 2025.

[10] Pengkun Wang, Zhe Zhao, HaiBin Wen, Fanfu Wang, Binwu Wang, Qingfu Zhang, and Yang Wang. β€œLLM-AutoDA: Large Language Model-Driven automated Data Augmentation for Long-tailed Problems.” NeurIPS 2025.

[11] Rui Zhang, Fei Liu, Xi Lin, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. β€œUnderstanding the importance of evolutionary search in automated heuristic design with large language models.” PPSN 2024.

[12] Shunyu Yao, Fei Liu, Xi Lin, Zhichao Lu, Zhenkun Wang, and Qingfu Zhang. β€œMulti-objective evolution of heuristic using large language model.” AAAI 2025. (Oral)

[13] Ping Guo, Qingfu Zhang, and Xi Lin. β€œCoEvo: Continual Evolution of Symbolic Solutions Using Large Language Models.” arXiv:2412.18890, 2024.

[14] Fei Liu, Yiming Yao, Ping Guo, Zhiyuan Yang, Xi Lin, Xialiang Tong, Mingxuan Yuan, Zhichao Lu, Zhenkun Wang, and Qingfu Zhang. β€œA Systematic Survey on Large Language Models for Algorithm Design.” arXiv:2410.14716, 2024