FlowSync is a Multi-Agent Deep Reinforcement Learning (MARL) framework for adaptive traffic signal control.
Each intersection is modeled as an agent that cooperatively learns to optimize signal timing to minimize global traffic congestion.

A key innovation is a neighbor-state attention mechanism, enabling agents to dynamically weigh neighboring intersections’ states and learn coordinated behaviour.

Built with Python, TensorFlow/Keras, and validated using SUMO (Simulation of Urban MObility).

We formulate Traffic Signal Control (TSC) as a Multi-Agent Markov Decision Process (MDP):

$$ (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma) $$

• $\mathcal{I} = {1, 2, \dots, N}$ — each agent $i$ is an intersection.

• $s = (s_1, s_2, \dots, s_N)$
• Each agent observes only its local state $s_i$.

• $A_i = {\text{Keep Phase}, \text{Change Phase}}$

• Determined by the SUMO simulator: $P(s' \mid s, a)$

• Each agent receives local reward $r_i = R(s, a, s')$

• $\gamma \in [0, 1]$

Objective:

$$ \pi_i^{*} = \arg\max_{\pi_i} \mathbb{E}\left[\sum_{k=t}^{\infty} \gamma^{k-t} r_k \right] $$

Each agent’s local state combines visual, tabular and neighbor information:

Penalty-based reward designed to reduce congestion:

$$ r_i = -\alpha_1 \sum \text{queue} - \alpha_2 \sum \text{wait} - \alpha_3 \sum\left(1 - \frac{v}{v_{\max}}\right) - \alpha_4 \cdot \text{flicker} + \alpha_5 \sum v_{\text{left}} $$

Weights $\alpha$ are set in conf/grid_2x2/sumo_agent.conf.

We approximate the optimal action-value function:

Training minimizes the TD loss:

$$ L(\theta) = \mathbb{E}_{(s,a,r,s')}\left[\left(y_i - Q(s,a;\theta)\right)^2\right] $$

with

$$ y_i = r + \gamma \max_{a'} Q(s', a'; \theta^{-}) $$

Inputs: visual map, local vectors, neighbor states.

Local feature extraction

• Visual Encoder: small CNN (32 filters @ 8×8, 16 filters @ 4×4) → $v_{\text{map}}$
• Vector Encoder: dense layers on concatenated per-lane vectors → $v_{\text{local}}$
• Local embedding: concatenate $v_{\text{map}}$ and $v_{\text{local}}$ → dense → $e_{\text{local}}$

Coordination via attention

• Form neighbor embeddings then apply dot-product attention:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^{T}}{\sqrt{d_k}}\right)V $$

Output: $c_{\text{neighbor}}$.

Q-value head

• Concatenate $e_{\text{local}}$ and $c_{\text{neighbor}}$, pass through two dense layers, output 2 linear units for {Keep, Change}.

Agents were trained on the grid_2x2 scenario (data/grid_2x2/grid.sumocfg).

Observations

• Learning trend: average reward improves over training (penalties reduced).
• Agent specialization: some intersections handle more traffic; all agents still show learning improvement.

sudo apt install sumo sumo-tools
pip install tensorflow numpy traci pandas matplotlib

cd data/grid_2x2/
netconvert --node-files=grid.nod.xml --edge-files=grid.edg.xml --output-file=grid.net.xml

Edit runexp.py:

sumoBinary_nogui = "/usr/bin/sumo"
sumoBinary_gui   = "/usr/bin/sumo-gui"
setting_memo     = "grid_2x2"

Run:

python runexp.py

For headless runs, ensure sumo_cmd_str points to the nogui binary.

python plot_results.py

This generates rewards_over_time.png from the latest memories.txt.

This implementation builds on the ideas in the IntelliLight work:

Hua Wei*, Guanjie Zheng*, Huaxiu Yao, Zhenhui Li, IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control, KDD 2018. IntelliLight repository

FlowSync/
├── conf/
│   └── grid_2x2/
├── data/
│   └── grid_2x2/
├── models/
│   ├── deeplight_agent.py
│   ├── network_agent.py
│   └── agent.py
├── runexp.py
├── traffic_light_dqn.py
├── plot_results.py
├── rewards_over_time.png
└── README.md

• Built on IntelliLight architecture (Wei et al., 2018)
• Inspired by MARL attention-based coordination frameworks
• Simulation powered by SUMO