Texture Generation with Neural Cellular Automata

Catex is an experimental project focused on generating textures using Neural Cellular Automata (NCA), a bio-inspired approach mimicking the emergent behavior of natural systems. By leveraging the self-organizing principles found in nature, Catex seeks to explore dynamic, decentralized intelligence.

The ultimate goal is to create robust, adaptable AI that mirrors how brains and natural networks process information.

The Motivation Behind Catex

Inspired by the complexity and adaptability of biological systems like the brain and social structures of ants and bees, we sought to create an AI model that adhered to similar principles: emergence, self-organization, and dynamic adaptability. Early on, we explored self-organizing maps (Kohonen maps) to mimic cortical maps, with a strong focus on vision tasks. However, a challenge arose when classifying textures which is a critical aspect of visual processing.

Neural Cellular Automata (NCA) caught our attention because of their ability to simulate complex patterns through simple rules applied consistently across cells, which mirrored the brain’s neural structures. This led us to experiment with NCA for texture generation and, eventually, texture classification.

The simplicity of the algorithm and its emergent behavior aligned perfectly with our biological constraints.

What Are Cellular Automata?

Cellular Automata (CA) are simple, grid-based systems where each cell follows a set of rules that dictate its behavior based on its current state and the states of its neighboring cells. This simple setup can lead to complex, emergent behaviors over time, where patterns, structures, or behaviors evolve from these interactions.

The key features of cellular automata include:

• Grid of Cells: Each cell has a state: "alive" or "dead."

• Neighborhood: A cell’s behavior is determined by its neighbors.

• Rules: A set of predefined rules that dictate how cells update their state based on their neighborhood state.

For instance, Conway’s Game of Life is a classic example of cellular automata, where simple rules about living or dying based on neighbors can lead to the emergence of intricate and evolving patterns.

In cellular_automata.py, the structure is defined, showing how perception and evolution modules work for NCAs:

class Perception(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
        # Define perception filters
        ...
 
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Execute perception actions
        ...
 
class Evolution(torch.nn.Module):
    def __init__(self, ns: int, ng: int) -> None:
        super().__init__()
        # Define network layers
        ...
 
class CellularAutomata(torch.nn.Module):
    def __init__(self, ns: int, ng: int) -> None:
        super().__init__()
        self.perception = Perception()
        self.evolution = Evolution(ns, ng)
 
    def forward(self, s: torch.Tensor, xg: torch.Tensor) -> torch.Tensor:
        xs = self.perception(s)
        combined_input = torch.cat([xs, xg], dim=1)
        ds = self.evolution(combined_input)
        return s + ds * update_mask

What Are Neural Cellular Automata?

Neural Cellular Automata (NCA) extend the idea of traditional cellular automata by integrating neural networks into the system. Unlike regular CA, where the rules are predefined and static, NCA learns its rules through a neural network, allowing it to adapt and perform more complex tasks like image generation or texture creation.

In a Neural Cellular Automata:

• Cells are not just binary or simple states; they have more complex states represented by vectors or multidimensional values.

• Learned Rules: The transition rules for each cell are learned through a neural network, allowing the system to discover patterns and solutions that are optimized for specific tasks (e.g. generating textures).

• Dynamic Evolution: Each cell continuously updates its state based on both its local neighbors and the information learned from a global objective (e.g. generating a specific pattern).

This approach makes NCA highly adaptable and capable of self-organizing into complex structures, much like biological cells in living organisms.

Key Features of Catex

• Emergent Behavior for Texture Generation: Inspired by the self-organizing principles of biological systems, Catex employs NCA to generate textures from scratch. This process evolves over time, leading to complex and intricate patterns.

• (NEW) Goal-Oriented Cellular Automata: Overcome the typical limitation of Cellular Automata by introducing hidden, fixed channels, optimized to generate specific targets. This opens up possibilities for zero-shot texture generation, where the system can create unseen textures by combining or interpolating existing patterns.

class Goal(torch.nn.Module):
    def __init__(self, n_goal: int, embed_dim: int) -> None:
        super().__init__()
        self.embedding = torch.nn.Embedding(n_goal, embed_dim)
        self.norm = torch.nn.LayerNorm(embed_dim)
        self.ffwd = torch.nn.Sequential(
            torch.nn.Linear(embed_dim, 4 * embed_dim),
            torch.nn.ReLU(),
            torch.nn.Linear(4 * embed_dim, embed_dim),
        )
 
    def forward(self, i: torch.Tensor) -> torch.Tensor:
        x = self.embedding(i)
        x = self.ffwd(self.norm(x)) + x
        return x

Both CA and goal parameters are optimized together in a training loop:

• The CA parameters focus on evolving the cellular states effectively to match the intricate details of the textures.
• The goal parameters aim to encapsulate the target's defining features and provide the necessary objective for the CA to follow.

Through iterations, seperate optimizers are used, ensuring both local alignment (cell-level update) and global coherence (goal-oriented generation) are achieved effectively.

Here is how the training is orchestrated in goal.py:

def train(
    ca: catex.model.CellularAutomata,
    goal: catex.model.Goal,
    loss_f: callable,
    n_exp: int,
    n_epoch: int,
):
    optimizer = torch.optim.Adam(ca.parameters(), lr=1e-3)
    g_optimizer = torch.optim.Adam(goal.parameters(), lr=1e-3)
 
    for i in tqdm.trange(n_epoch):
        for n in range(n_exp):
            # Encode goals
            g = torch.Tensor(4 * [n]).to(torch.int64).to(device)
            xg = einops.repeat(goal(g), "b g -> b g h w", h=h, w=w)
 
            # Pick up starting states
            x = pool[n, batch_idx]
            ...
 
            # Evolve state and compute losses
            for k in range(step_n):
                x = ca(x, xg)
            loss = loss_f(x, n)
            loss.backward()
 
            # Update CA parameters
            for p in ca.parameters():
                p.grad /= p.grad.norm() + 1e-8
            optimizer.step()
            optimizer.zero_grad()
 
            # Update goal parameters
            for p in goal.parameters():
                p.grad /= p.grad.norm() + 1e-8
            g_optimizer.step()
            g_optimizer.zero_grad()
            ...

Under the Hood: Technical Overview

Catex is built on the principle of dynamic state updates, where each cell continuously evolves from an initial state toward a target, driven by local interactions and global objectives:

• Neural Cellular Automata: The core algorithm mimics the self-organizing nature of cellular automata, where simple rules applied at a local level lead to complex global behaviors. These behaviors can generate detailed textures, evolving from random initial states.

• Loss Functions: The model leverages a combination of VGG-style loss and Sliced Wasserstein loss, which is crucial for high-fidelity texture synthesis.

• Goal-Oriented Footprints: By embedding additional hidden channels into the CA model, Catex creates a footprint that enables it to generate specific patterns, allowing for variation across textures by modifying initial conditions.

• Dynamic, Bio-Inspired Algorithms: Reflecting principles from neural and biological systems, Catex’s NCA adapts over time, making it an ideal framework for tasks requiring flexibility, such as texture classification and generation.

Catex demonstrates its ability to produce textures:

1. Initial State: The system starts with a random configuration of cells.
2. Evolution: The cells continuously evolve under the NCA rules, influenced by the target texture’s footprint.
3. Final Output: The result is a texture, demonstrating the power of emergent, self-organized behavior in generating complex patterns.

Experiments with Goal-Oriented NCA

In exploring the capabilities of goal-oriented Neural Cellular Automata (NCA), I conducted a series of experiments to understand how the system can generate distinct textures based on embedded footprints.

Experiment 1: Texture Generation from Footprints

This experiment involved two distinct texture footprints. By initializing the grid with these footprints, the NCA successfully generated the designated textures on their respective sides of the grid. The left side of the grid started with texture 1's footprint, while the right side began with texture 2's footprint. The grid evolved to accurately replicate each texture in its respective section.

Experiment 2: Interpolation

Beyond visual outputs, I explored the potential of footprint interpolation. By blending the two footprints, the NCA produced a hybrid texture that seamlessly combined elements from both patterns.

Experiment 3: Analysis of Footprint Embeddings with t-SNE

In addition to generating textures, an analysis of the footprint embeddings was conducted using t-SNE to investigate their utility for texture classification. This analysis provided insights into the distinctiveness and clustering potential of different texture patterns within the system's learned representation.

While these latter experiments didn't yield direct visible results, they offer promising pathways for future exploration in texture adaptation and classification using NCAs.

Scalability and Adaptability of Cellular Automata

Cellular Automata (CA) are inherently scalable, allowing for easy upscaling or downscaling of textures due to their local update rules. Beyond traditional 2D grids, CAs can adapt to various geometries—like triangular or hexagonal grids—if perception kernels are defined for them. This flexibility makes CAs applicable to diverse spatial domains and use cases.

Target

Generated (upscaled)

Driving Idea Behind Goal-Oriented Cellular Automata

Imagine each texture as a living organism. The goal embeddings function like DNA, providing the blueprint that defines the texture's unique identity. These instructions guide the form and characteristics across the entire texture.

The NCA update rules are akin to the universal laws of cellular processes, operating consistently to evolve textures through local interactions. Just as cellular machinery drives biological functions according to DNA, NCA rules refine the texture based on the goal blueprint.

Together, the goal embedding (DNA) and NCA update rules (cellular machinery) collaboratively shape complex textures, mirroring the growth and adaptation seen in living organisms. This synergy enables Catex to dynamically generate diverse patterns.

Why Catex Matters

Catex represents a bridge between biology and AI, embracing nature’s inherent intelligence. By following bio-inspired principles, Catex opens up new possibilities for dynamic, adaptable systems that can self-organize, much like natural systems do. This project doesn’t just generate textures—it pioneers a new way of thinking about how AI systems can emulate the brain’s versatility and resilience, making it a promising step toward the future of decentralized AI.

References

• Original Inspiration: The Catex project was inspired by the research on texture generation using self-organizing principles, as detailed in "Textures Synthesized through Self-Organization".

• Self-Organizing Maps: For foundational knowledge on self-organizing maps, refer to "The self-organizing map", which explains how these neural networks mimic brain processes for pattern recognition.

• Conway's Game of Life: An introduction to cellular automata using "Conway’s Game of Life", which illustrates basic concepts of rule-based cell behavior leading to complex patterns.

• Neural Cellular Automata: For insights into neural network-based cellular automata, see "Growing Neural Cellular Automata" by Mordvintsev et al., which explores how neural cellular automata can grow structures and patterns.

• Loss Functions for Image Synthesis: The VGG-style loss is discussed in "Image Style Transfer Using Convolutional Neural Networks", a key study in style transfer and texture synthesis.

• Sliced Wasserstein Loss: To understand the theoretical underpinnings of high-dimensional distributions' comparisons in texture synthesis, review "Sliced Wasserstein Generative Models"