Advancing NeSy AI with Abstract Urban Simulation

NeurIPS'24 D&B Track

Bowen Li1, Zhaoyu Li2, Qiwei Du3, Jinqi Luo4, Wenshan Wang1, Yaqi Xie1, Simon Stepputtis1, Chen Wang3, Katia Sycara1, Pradeep Ravikumar1, Alexander Gray5, Xujie Si2, Sebastian Scherer1,*
1Carnegie Mellon University, 2University of Toronto, 3University at Buffalo, 4University of Pennsylvania 5Centaur AI Institute
Official arXiv Video Code, Data, Models

LogiCity is a new simulator and benchmark for NeSy AI. It simulates dynamic urban environments with flexible abstractions. It features Inductive Abstract Reasoning, where the most recent LLM (GPT-4o) falls behind human.

Abstract

Recent years have witnessed the rapid development of Neuro-Symbolic (NeSy) AI systems, which integrate symbolic reasoning into deep neural networks. However, most existing benchmarks for NeSy AI fail to provide long-horizon reasoning task with complex multi-agent interaction. Furthermore, they are usually constrained by fixed and simplistic logical rules over limited entities, making them inadequate for capturing real-world complexities.

To address these crucial gaps, we introduce LogiCity, the first simulator based on customizable first-order logic (FOL) for an urban-like environment with multiple dynamic agents. LogiCity models diverse urban elements using semantic and spatial concepts, such as IsAmbulance(X) and IsClose(X,Y). These concepts are used to define FOL rules governing the behavior of various agents. Since the concepts and rules are abstractions, they can be universally applied to cities with any agent compositions, facilitating the instantiation of diverse scenarios. Besides, a key benefit of our LogiCity is its support for user-configurable abstractions, enabling customizable simulation complexities for logical reasoning.

To explore various aspects of NeSy AI, we introduces two tasks, one features long-horizon sequential decision-making, and the other focuses on one-step visual reasoning, varying in difficulty and agent behaviors. Our extensive evaluation using LogiCity reveals the advantage of NeSy frameworks in abstract reasoning. Moreover, we highlight the significant challenges of handling more complex abstractions in long-horizon multi-agent reasoning scenarios or under high-dimensional, imbalanced data. With the flexible design, various features, and newly raised challenges, we believe LogiCity represents a pivotal step for advancing the next generation of NeSy AI.

Contribution

  • We propose a pioneer abstraction-based dynamic city simulator LogiCity for NeSy AI, which is flexible and scalable for various reasoning tasks.
  • We develop Sequential Decision Making and Visual Reasoning tasks in LogiCity to evaluate the compositional generalization capability of various models.
  • We conduct exhaustive experiments and demonstrate the advantages of NeSy AI in compositional generalization, while also revealing the new reasoning challenges introduced by LogiCity.

    • Simulation Examples


    [1] An entity will Stop if it is CollingClose with another entity, OR, if it is AtIntersection with another entity InIntersection, OR, if it is AtIntersection with another HigherPriority entity AtIntersection.


    Simulation Rendering


    By virtue of the foundamental generative models, LogiCity supports a diverse range of rendering styles.

    Safe Path Following (SPF)


    Safe Path Following (SPF) requires an algorithm to control an agent in LogiCity, navigating it to the goal while maximizing the trajectory reward. Since the ego agent could meet with complex situations along the way and needs to smartly plan to maximize trajectory return, this task features long-horizon reasoning with multiple dynamic agents.


    Note that SPF also features Different Train/Test Agent Sets.

    Description of the image

    Hard mode visualization

    The FOL rule and a training example in the hard mode for the SPF task is shown below.


    Clause Set
    Stop(X):- Not(IsAmbulance(X)), Not(IsOld(X)), IsAtInter(X), IsInInter(Y).
Stop(X):- Not(IsAmbulance(X)), Not(IsOld(X)), IsAtInter(X), IsAtInter(Y), HigherPri(Y, X).
...
    Example training episode
    Testing episode 1

    Visual Action Prediction (VAP)


    Visual Action Prediction (VAP) focuses on reasoning with high-dimensional data, requiring models to predict the actions of all agents from an RGB image. The challenge here lies in sophisticated abstract reasoning with high-level perceptual noise.

    Description of the image

    Qualitative comparison between NLM and GNN in the hard mode of VAP task. We display the grounded clauses, where the involved entities are marked with boxes in corresponding colors. Correct predictions are shown in gree, while the wrong one is in red.