Draft 5/3/2023

Main single agent machine learning algorithms

Most of the machine learning methods described below are in more detail in [1].

Algorithm Description Potential for Multi-agent System

Decision Trees Decision tree learning is a method for approximating discrete function by a decision tree. In the nodes of trees are attributes and in the leaves are values of discrete function. The decision tree can be rewritten in a set of if-then rules. Trees learning methods are popular inductive inference algorithms, mostly used for variety of classification tasks (for example for diagnosing medical cases). For tree generation there is often used entropy as information gain measure of the attribute. The best-known methods are ID3, C4.5, etc.

There are papers appearing about Multi-agent tree learning problem, e.g. [2]. It mentioned that the new algorithm “is being applied to several knowledge discovery problems in molecular biology and network-based intrusion detection.

Neural Networks Neural networks learning methods provide a robust approach to approximating real-valued, discrete-valued and vector-valued functions. The well-known algorithm, Back-Propagation, uses gradient descent to tune network parameters to best fit to training set with input-output pair. This method is inspired by neurobiology. It imitates function of brain, where many neurons are inter-connected. The instances are represented by many input-output pairs. NN learning is robust to errors in training data and has been successfully applied to problems such as speech recognition, face recognition, etc.

There are papers which try to apply NN in MAS. For instance, in [3], “a neural network based multi-agent, especially hierarchically organized, information retrieval system” was presented. This multi-agent approach was motivated both analytically and experimentally.

Bayesian Methods

Bayesian reasoning provides a probabilistic approach to inference. Bayesian reasoningprovides the basis for learning algorithms that directly manipulate with probabilities, as well as a framework for analyzing the operation of other algorithms. Bayesian learning algorithm that calculates explicit probabilities for hypothesis, such us the naive Bayes, are among the most practical approaches to certain type of learning problems. Bayes classifier is competitive with other ML algorithms in many cases. For example for learning to classify text documents, the naive Bayes classifier is one of the most effective classifiers.

The papers appeared in this area started from 1993, e.g. [4-6]. These three papers have the same first author. So, there is potential of succeed of this algorithm in MAS, but not clear evidence yet.

Reinforcement Learning

Reinforcement learning solves the task -- how the agent (that can sense and act inenvironment) can learn to choose optimal actions to reach its goal. Each time the agentperforms an action in its environment, a trainer may provide a reward or penalty to indicatethe conveniency of the resulting state. For example, when agent is trained to play a gamethen trainer might provide a positive reward when the game is won, negative reward when it is lost, and zero reward in other states. The task of agent is to learn from this delayed reward,

There was a paper [7], in which the authors adopt general-sum stochastic games as a framework for multiagent reinforcement learning. Their work extends previous work by Littman on zero-sum stochastic games to a broader

Draft 5/3/2023

to choose sequences of actions that produce the greatest cumulative reward. An algorithm that can acquire optimal control strategies from delayed reward is called Q-learning. This method can solve the problems like learning to control mobile robot, learning to optimize operations in factories, learning to plan therapeutic procedures, etc.

framework. They design a multiagent Q-learning method under this framework, and prove that it converges to a Nash equilibrium under specified conditions. (This paper was cited for 41 times.)

Inductive Logic Programming

Inductive logic programming has its roots in concept learning from examples, a relatively straightforward form of induction. The aim of concept learning is to discover, from a given set of pre-classified examples, a set of classification rules with high predictive power. The theory of ILP is based on proof theory and model theory for the first order predicate calculus. Inductive hypothesis formation is characterized by techniques including inverse resolution, relative least general generalisations, inverse implication, and inverse entailment. This method can be used for creating logical programs from training data set. The final program should be able to generate that data back. The creating logical programs is very dependent on task complexity. In many cases this method is not usable without many restrictions posed on the final program. With success ILP is mostly used in Data Mining for finding rules in huge databases.

In recent years, a number of logic programming-based proposals to agents have been put forward [8]. In [8], a comprehensive survey of computational logic-based agents and multi-agent systems was provided. (This survey was cited for 21 times.)

Case-Based Reasoning

Case-Based Reasoning (CBR) is a lazy learning algorithm that classifies new queryinstance by analyzing similar instances while ignoring instances that are very different from the query. This method holds all previous instances in case memory. The instances/cases can be represented by values, symbols, trees, various hierarchical structures or other structures. It is non-generalization approach. The CBR works in the cycle: case retrieval -reuse - solution testing - learning. This method is inspired by biology, concretely by human reasoning using knowledge from old similar situations. This learning method is also known as Learning by Analogy. CBR paradigm covers a range of different methods. Widely used is Instance-Based Reasoning (IBR) algorithm that differs from general CBR mainly in representing instances. The representation of the instances is simple, usually it is vector of numeric or symbolic values. Instance-based learning includes k-Nearest Neighbors (k-NN) and Locally Weighted Regression (LWR) methods.

The Multi-agent Systems Lab (Dep. of computer science at the Univ. of Massachusetts at Amherst) has a project – “CBR in a MAS” [9]. They have now started investigating case-based learning and have implemeted a Distributed Case-Based Learning System for multi-agent path planning and are in process of evaluating it.

Support VectorMachines

Support Vector Machines (SVM) has become very popular method for classification andoptimization at the recent time. SVMs were introduced by Vapnik et al. in 1992. Thismethod combines two main ideas. The first one is concept of an optimum linear marginclassifier, which constructs a separating hyperplane that maximizes distances to the training point. The second one is concept of a kernel. In its simplest form, the kernel is a function which calculates the dot product of two training vectors. Kernels calculate these dot product

Based on the search on web, there is not any evidence that SVM has been applied to MAS. SVMs were successfully applied in classification and regression problems. So there are two possible ways: 1)

Draft 5/3/2023

in feature space, often without explicitly calculating the feature vectors, operating directly on the input vectors instead. When we use feature transformation, which reformulates input vector into new features, the dot product is calculated in feature space, even if the new feature space has higher dimensionality. So the linear classifier is unaffected. Margin maximization provides a useful trade off with classification accuracy, which can easily lead to overfitting of the training data. SVM are well applicable to solve learning tasks where the number of attributes is large with respect to the number of training examples.

describe MAS into several classification and regression problems; 2) apply kernel and large margin concept into MAS condition.

Genetic Algorithms

Genetic algorithms provide a learning method motivated by an analogy to biologicalevolution. The search for an appropriate hypothesis begins with a population of initial hy-pothesis. Members of the current population give rise to the next generation population byoperations such as selection, crossover and mutation. At each step, a collection of hypothesis called the current population is updated by replacing some fraction of the population by off-springs of the most fit current hypothesis. Genetic algorithms have been applied successfully to a variety of learning tasks and optimization problems. For example, Genetic algorithms can be used in other ML methods, such as Neural Network or Instance-Based Reasoning for optimal parameters setting.

There are research groups undertakes research in applying GA in MAS: 1. The Evolutionary Computation Research Group in the Department of Computer Studies at Napier University undertakes research in evolutionary computation and its applications. Its emphasis is on evolving multi- agent systems and self-adaptation in genetic algorithms with applications in timetabling, logic minimisation and control systems engineering.2. The Evolutionary Computing Group at UWE, Bristol, undertakes applied research in collaboration with other organisations, and performs research in evolutionary computing, artificial life, and multi-agent systems.There are also papers about the research in this area, e.g. [10] (2 citations).