# Multi-Agent Active Inference ## Overview Multi-agent active inference extends the free energy principle (FEP) from single-agent settings to systems of interacting agents, each minimizing their own variational free energy while sharing an environment. This framework addresses how multiple self-organizing systems coordinate behavior, develop shared representations, communicate, and evolve collective intelligence. Rather than treating multi-agent interaction as an externally designed optimization problem, multi-agent active inference shows how coordination emerges naturally from the imperative to minimize surprise at individual and collective scales. ## Shared Generative Models ### The Problem of Other Minds Each agent in a multi-agent system maintains its own generative model of the world. A central challenge is that the "world" now includes other agents, each of which is itself a free-energy-minimizing system. Agent i must therefore maintain a generative model that includes models of other agents' generative models -- a recursive structure that grounds theory of mind. The generative model for agent i in a multi-agent setting takes the form: ```math P_i(o_i, s, \theta_{-i} | m_i) = P_i(o_i | s) P_i(s | a_i, a_{-i}) P_i(a_{-i} | \theta_{-i}) P_i(\theta_{-i}) ``` where: - `o_i` are observations for agent i - `s` represents shared environmental states - `theta_{-i}` are parameters of other agents' generative models - `a_i` and `a_{-i}` are actions of agent i and other agents respectively - `m_i` is agent i's generative model ### Shared vs. Complementary Models Agents in a multi-agent system may have: 1. **Shared generative models**: Agents possess structurally similar models of the environment, enabling mutual prediction and coordination. This is common in species with shared evolutionary history. 2. **Complementary generative models**: Agents possess different but compatible models that, when combined, provide more complete coverage of environmental dynamics. This supports division of cognitive labor. 3. **Hierarchically nested models**: Lower-level agents' models are embedded within higher-level organizational models, as seen in cellular collectives, social insect colonies, and institutional hierarchies. The degree of model sharing can be quantified through the mutual information between agents' sufficient statistics: ```math I(\mu_i; \mu_j) = D_{KL}[P(\mu_i, \mu_j) || P(\mu_i) P(\mu_j)] ``` where `mu_i` and `mu_j` are the sufficient statistics (beliefs) of agents i and j. ## Theory of Mind via Hierarchical Inference ### Inference About Other Agents Active inference provides a natural account of theory of mind: understanding others' mental states is simply inference about the hidden causes of their observed behavior. Agent i infers the beliefs, preferences, and policies of agent j by treating agent j's actions as observations generated by j's (hidden) generative model. This can be formalized as hierarchical inference: ```math Q_i(\theta_j) = \arg\min_{Q} D_{KL}[Q(\theta_j) || P_i(\theta_j | o_i^{(j)})] ``` where `o_i^{(j)}` are agent i's observations of agent j's behavior, and `theta_j` encodes j's preferences, beliefs, and policies. ### Levels of Intentional Depth Multi-agent active inference naturally accommodates multiple levels of intentional reasoning: - **Level 0 (reactive)**: Agent models environment without distinguishing other agents from objects - **Level 1 (mentalizing)**: Agent models others as intentional agents with beliefs and preferences - **Level 2 (recursive)**: Agent models others' models of itself ("I think that you think that I...") - **Level k**: Arbitrary depth of recursive belief modeling The depth of recursive modeling is bounded by computational resources and can be understood through the lens of model complexity. The free energy principle suggests agents will adopt the simplest model (lowest level) that adequately predicts others' behavior, adding recursive depth only when it reduces prediction error enough to justify the complexity cost. ### Sophistication and Epistemic Depth Sophisticated inference (Friston et al., 2021) extends standard active inference to account for how an agent's actions will change its own future beliefs. In the multi-agent case, this becomes: ```math G_i(\pi_i) = \mathbb{E}_{Q_i}[\ln Q_i(s_{\tau} | \pi_i) - \ln P_i(o_{\tau}, s_{\tau} | \tilde{\pi})] ``` where `tilde{pi}` denotes the joint policy space including other agents' anticipated responses to agent i's policy `pi_i`. ## Communication as Inference ### Generalized Communication In the FEP framework, communication between agents is itself a form of active inference. When agent i communicates with agent j, agent i selects actions (signals) that are expected to align agent j's beliefs with states of affairs relevant to both agents. Communication minimizes the divergence between agents' posterior beliefs: ```math a_i^{comm} = \arg\min_{a_i} \mathbb{E}_{Q_i}[D_{KL}[Q_j(s | o_j, a_i) || Q_i(s)]] ``` This formulation unifies several communication phenomena: 1. **Ostensive communication**: Signals selected to reduce the receiver's uncertainty about the sender's communicative intent 2. **Pragmatic communication**: Actions chosen to align beliefs about action-relevant states 3. **Phatic communication**: Signals that maintain channel reliability and social connection (reducing uncertainty about the relationship itself) ### Language as Shared Generative Model Natural language can be understood as a shared generative model that enables efficient alignment of beliefs between agents. Linguistic conventions function as shared priors that constrain the space of possible interpretations: ```math P(meaning | utterance, context) \propto P(utterance | meaning) P(meaning | context) ``` The evolution of linguistic structure reflects the collective minimization of communicative free energy across a population of agents over cultural timescales. ## Multi-Agent Coordination Through Aligned Free Energy ### Collective Free Energy When multiple agents share an environment, a natural quantity emerges: the collective free energy of the group. This can be decomposed as: ```math F_{collective} = \sum_i F_i + F_{interaction} ``` where `F_i` is the individual free energy of agent i, and `F_{interaction}` captures the coupling between agents' belief systems through their shared environment. Alternatively, collective free energy can be expressed as: ```math F_{collective} = D_{KL}[\prod_i Q_i(s) || P(o_1, ..., o_n, s)] + \text{const} ``` This highlights that collective free energy is minimized when agents' beliefs are both individually accurate and mutually consistent. ### Coordination Without Central Control A key insight of multi-agent active inference is that coordination can emerge without any central controller or explicit coordination mechanism. When agents share an environment, their individual free energy minimization creates implicit coordination through: 1. **Stigmergy**: Agents modify the shared environment, which other agents then sense, creating indirect communication through environmental traces 2. **Behavioral coupling**: One agent's actions become part of another agent's observations, creating sensorimotor loops that span multiple agents 3. **Belief alignment**: Shared sensory access to common environmental states drives convergence of posterior beliefs ### Synchronization as Free Energy Minimization The spontaneous synchronization of coupled oscillators (Kuramoto model) can be recast as multi-agent free energy minimization. Each oscillator minimizes surprise about the phase signals it receives from neighbors, naturally producing phase-locked states that correspond to collective free energy minima. ## Nash Equilibria as Non-Equilibrium Steady States (NESS) ### Game-Theoretic Connection A profound connection exists between game theory and multi-agent active inference: Nash equilibria in games correspond to non-equilibrium steady states (NESS) of the coupled dynamical system formed by multiple free-energy-minimizing agents. At a Nash equilibrium, no agent can reduce its free energy by unilaterally changing its policy. Formally, for each agent i: ```math F_i(\pi_i^*, \pi_{-i}^*) \leq F_i(\pi_i, \pi_{-i}^*) \quad \forall \pi_i ``` This connects to the FEP through the observation that a NESS is characterized by the existence of a steady-state density that agents' dynamics preserve. The NESS condition: ```math \nabla \cdot (f(x) p_{ss}(x)) = 0 ``` when decomposed across agents, yields exactly the Nash equilibrium conditions when the flow `f` is identified with free energy gradient descent. ### Beyond Classical Nash Multi-agent active inference extends beyond classical game theory in several ways: 1. **Bounded rationality**: Agents minimize free energy under computational constraints, yielding quantal response equilibria rather than exact Nash equilibria 2. **Belief-dependent payoffs**: In active inference, the "payoff" is negative free energy, which depends on beliefs as well as outcomes 3. **Dynamic games**: The temporal structure of active inference naturally handles extensive-form games through policy selection over time 4. **Incomplete information**: Uncertainty about other agents' types is handled through hierarchical inference rather than requiring common knowledge assumptions ### Correlated Equilibria and Shared Priors When agents share priors (e.g., through cultural learning or common evolutionary history), the resulting equilibria correspond to correlated equilibria rather than Nash equilibria. Shared priors act as a correlation device: ```math P(a_i, a_j) = \sum_s P(a_i | s) P(a_j | s) P(s) ``` where the shared state `s` mediates correlation between agents' actions without requiring explicit communication. ## Game-Theoretic Formulations ### Active Inference Games An active inference game is defined by: - A set of agents `I = {1, ..., n}` - For each agent i: a generative model `m_i`, observations `o_i`, and policy space `Pi_i` - A shared environment with state dynamics `P(s_{t+1} | s_t, a_1, ..., a_n)` The solution concept is a set of policies `{pi_i^*}` such that each agent's policy minimizes its expected free energy given its beliefs about other agents' policies: ```math \pi_i^* = \arg\min_{\pi_i} G_i(\pi_i | Q_i(\pi_{-i})) ``` ### Cooperative vs. Competitive Settings **Cooperative games**: When agents share prior preferences (similar `P(o)` terms in their generative models), free energy minimization drives cooperation. Joint planning emerges as each agent's model of others' goals aligns with its own. **Competitive games**: When prior preferences conflict, agents' free energy minimization creates adversarial dynamics. Each agent must model the other's strategic reasoning, driving deeper recursive inference. **Mixed-motive games**: Most real-world scenarios involve partial alignment of preferences. Multi-agent active inference naturally handles these through the balance between shared and distinct components of agents' generative models. ### Evolutionary Game Theory Connection Over evolutionary timescales, the generative models themselves are subject to selection. The evolutionary stable strategy (ESS) corresponds to a generative model that, when common in a population, cannot be invaded by any alternative model: ```math F(m^* | m^*) \leq F(m | m^*) \quad \forall m \neq m^* ``` This connects Maynard Smith's ESS concept to the FEP: evolutionary stable generative models are those that minimize free energy in the ecological niche defined by a population of similar agents. ## Social Learning ### Learning From Others Social learning in the active inference framework occurs when one agent uses observations of another agent's behavior to update its own generative model. This is more efficient than individual learning when: ```math D_{KL}[Q_i(s | o_i, o_i^{(j)}) || P(s)] < D_{KL}[Q_i(s | o_i) || P(s)] ``` That is, when incorporating observations of another agent's behavior reduces the divergence between the agent's beliefs and the true state of affairs. ### Imitation and Emulation - **Imitation**: Directly copying observed action sequences. Corresponds to adopting another agent's policy without inferring the underlying generative model. - **Emulation**: Inferring the goals or preferences behind observed behavior and generating one's own policy to achieve similar outcomes. Requires deeper generative model inference. Active inference predicts that agents should prefer emulation when they have sufficient computational resources, as it transfers more generalizable knowledge. Imitation is preferred when the complexity cost of full model inference outweighs its benefits. ### Conformity and Innovation The tension between conformity (adopting group priors) and innovation (maintaining individual priors) can be modeled as a precision-weighting problem: ```math Q_i(s) \propto P_{individual}(s)^{\pi_{ind}} \cdot P_{social}(s)^{\pi_{soc}} ``` where `pi_{ind}` and `pi_{soc}` are precision parameters controlling the relative influence of individual and social evidence. The optimal balance depends on the volatility of the environment: stable environments favor conformity, volatile environments favor individual exploration. ## Cultural Evolution Through Shared Priors ### Cumulative Cultural Evolution Cultural evolution in the FEP framework is the process by which shared priors (generative models) evolve across generations through social learning. Unlike genetic evolution, cultural evolution allows: 1. **Lamarckian inheritance**: Acquired model updates can be transmitted to the next generation through teaching and demonstration 2. **Horizontal transfer**: Models can spread within a generation through communication 3. **Cumulative improvement**: Each generation can build on previous generations' model refinements ### Cultural Attractors Certain configurations of shared generative models act as cultural attractors -- states in the space of possible cultural configurations that are stable under the dynamics of social learning and transmission: ```math \frac{d}{dt} P_{culture}(\theta) = -\nabla_\theta F_{collective}(\theta) + \eta(t) ``` Cultural attractors correspond to local minima of collective free energy, explaining why certain cultural forms (kinship systems, musical scales, narrative structures) recur independently across human societies. ### Institutions as Shared Generative Models Social institutions can be understood as collectively maintained generative models that coordinate expectations and behavior across large groups. Institutions encode: - **Shared state spaces**: Common ontologies for categorizing social reality - **Transition models**: Shared expectations about how social states evolve - **Preference structures**: Collectively endorsed norms and values - **Precision assignments**: Shared assessments of which information sources are reliable The stability of institutions reflects their status as free energy minima for the collective: institutional change requires sufficient perturbation to escape the basin of attraction of current shared priors. ## Formal Properties ### Convergence Under mild regularity conditions (ergodicity, finite state spaces, connected communication structure), multi-agent belief updating converges to a consensus: ```math \lim_{t \to \infty} D_{KL}[Q_i(s, t) || Q_j(s, t)] = 0 \quad \forall i, j ``` The rate of convergence depends on network topology, with well-connected networks converging faster but potentially sacrificing diversity of beliefs needed for collective intelligence. ### Scalability The computational complexity of multi-agent active inference scales with the depth of recursive modeling. Mean-field approximations (assuming independence between agents' policies) reduce complexity from exponential to polynomial, at the cost of ignoring strategic correlations. ### Robustness Multi-agent active inference systems exhibit robustness to individual agent failure because coordination emerges from distributed free energy minimization rather than centralized control. The loss of any individual agent shifts the collective NESS but does not destroy the coordination mechanism. ## Applications ### Neuroscience - Social brain hypothesis: cortical expansion driven by demands of multi-agent inference - Mirror neurons as implementing inference about others' generative models - Autism spectrum as atypical precision weighting in social inference ### Artificial Intelligence - Multi-agent reinforcement learning through shared generative models - Emergent communication in artificial agent populations - Cooperative robotics via distributed active inference ### Social Science - Institutional design as engineering collective free energy landscapes - Market dynamics as multi-agent belief updating - Political polarization as divergence of shared generative models ### Ecology - Collective behavior in animal groups (flocking, schooling, swarming) - Symbiotic relationships as coupled free energy minimization - Ecosystem stability through multi-species active inference ## Key References - Friston, K., & Frith, C. (2015). A duet for one. Consciousness and Cognition, 36, 390-405. - Vasil, J., et al. (2020). A world unto itself: Human communication as active inference. Frontiers in Psychology, 11, 417. - Kaufmann, R., et al. (2021). The active inference approach to ecological perception. Frontiers in Psychology, 12. - Friston, K., et al. (2022). Designing ecosystems of intelligence from first principles. arXiv preprint. - Albarracin, M., et al. (2022). Epistemic communities under active inference. Entropy, 24(4), 476. - Da Costa, L., et al. (2020). Active inference on discrete state-spaces. Journal of Mathematical Psychology, 99. ## Cross-References - [[active_inference|Active Inference]] - Single-agent foundation - [[social_cognition|Social Cognition]] - Neural basis of social understanding - [[social_learning|Social Learning]] - Learning from conspecifics - [[knowledge_base/mathematics/expected_free_energy|Expected Free Energy]] - Policy selection criterion - [[knowledge_base/mathematics/markov_blankets|Markov Blankets]] - Boundary conditions for agents - [[biology/niche_construction|Niche Construction]] - Environmental modification by agents - [[philosophy/enactivism|Enactivism]] - Participatory sense-making - [[knowledge_base/systems/circular_causality|Circular Causality]] - Reciprocal agent-environment dynamics