Enemy AI State Machine Design: A Practical Guide for Roguelikes

Who this helps: Developers building enemy AI for roguelike games, programmers working on boss phase systems, and system designers who want a reusable finite state machine architecture that works across Phaser, Unity, Godot, or any engine.

---

The Problem

Your enemy needs to detect the player, chase, attack, and flee. A simple if-else chain works for two or three states, but as behaviors grow — boss phase transitions, stagger reactions, search behavior after losing line of sight — the code becomes unmanageable. State transitions turn into infinite loops. Frame-rate-dependent timers cause inconsistent behavior across devices.

Core goal: Model enemy behavior as a sequence of state transitions, keep each state self-contained, and define transition rules explicitly.

---

Why It Matters

• Maintainability: Independent states let you add new behaviors without touching existing code.
• Debugging: Displaying the current state on screen makes AI issues immediately visible.
• Difficulty scaling: When each state exposes tunable parameters, difficulty becomes a data problem rather than a code problem.
• Boss design: A hierarchical state machine (HFSM) structures phase transitions so boss AI remains systematic rather than chaotic.

---

Core Principle

Enemy AI is a system that transitions between a finite set of states. Each state owns its own entry, update, and exit logic. Transition rules are triggered by the detection system.

State transition graph:

                   +--------+
                   |  IDLE  |<-------------------+
                   +--------+                     |
                        | patrolDelay             | arrived
                   +----v----+                     |
                   | PATROL  |---------------------+
                   +----+----+
                        | player detected
                   +----v----+
                   | ALERT   | (warning indicator, brief reaction delay)
                   +----+----+
                        | reactionTime elapsed
                   +----v----+     player lost
                   | CHASE   |------------------+
                   +--+--+--+                  |
            player in |  | out of sight        |
            range     |  +---------+           |
         +----v----+   |           |      +----v---+
         | ATTACK  |   |           |      | SEARCH |
         +--+--+--+   |           |      +----+---+
            |  | fleeHP           |           | timeout
   +----v---+  |                |           |
   |  FLEE  |<-+                |           +----v---+
   +---+----+                   |           | RETURN |
       | flee done               |           +----+---+
       +-------------------------+----------------+

---

How To Apply It

Step 1: Track Detection as a Meter, Not a Boolean

Replace canSeePlayer booleans with a continuous detection value that ramps up when the player is in sight and decays when hidden.

// Detection system
enum DetectionLevel { UNAWARE, SUSPICIOUS, ALERTED, HOSTILE }

function updateDetection(entity, player, dt) {
  if (canSee(entity, player)) {
    entity.detectionMeter += 30 * dt;
  } else if (hear(entity, player)) {
    entity.detectionMeter += 15 * dt;
  } else {
    entity.detectionMeter -= 5 * dt;
  }
  entity.detectionMeter = clamp(entity.detectionMeter, 0, 100);

  if (entity.detectionMeter >= 80) entity.level = DetectionLevel.HOSTILE;
  else if (entity.detectionMeter >= 50) entity.level = DetectionLevel.ALERTED;
  else if (entity.detectionMeter >= 20) entity.level = DetectionLevel.SUSPICIOUS;
  else entity.level = DetectionLevel.UNAWARE;
}

This approach prevents flickering between states when the player briefly breaks line of sight.

Step 2: Build a Boss AI with a Hierarchical State Machine

Bosses need macro-level phases (intro, combat, enrage) each containing micro-level state machines (idle, attack patterns, reposition).

class BossStateMachine {
  private phase: string;
  private actionFSM: StateMachine;

  checkPhaseTransition(boss) {
    const ratio = boss.health / boss.maxHealth;
    if (ratio <= 0.33 && this.phase === 'PHASE2') {
      this.transitionTo('PHASE3', boss);
    }
  }

  transitionTo(newPhase, boss) {
    this.phase = newPhase;
    boss.invincible(2.0);          // Brief invuln during transition
    this.actionFSM = getActionFSM(newPhase);
  }

  getCurrentStateName(): string {
    return this.actionFSM?.currentName || 'IDLE';
  }
}

Step 3: Expose Difficulty as Data

When difficulty is data, you change the feel of your game by editing a table, not rewriting logic.

Step 4: Engine Integration

The state pattern is engine-agnostic. The only difference is where you call update:

Phaser 3:

update(time: number, delta: number) {
  const dt = delta / 1000;
  this.enemies.forEach(e => e.fsm.update(e, dt));
}

Unity:

void Update() {
  fsm.Update(Time.deltaTime);
}

Godot:

func _process(delta):
    fsm.update(delta)

---

Common Mistakes

---

Checklist

• [ ] Does your state interface define three methods: enter, update, exit?
• [ ] Are global transitions (death, stagger) evaluated before state-local logic?
• [ ] Does the detection meter decay smoothly rather than reset instantly?
• [ ] Is there a minimum dwell time to prevent transition flicker?
• [ ] Are all timers deltaTime-based, not frame-count-based?
• [ ] Does every state have a fallback or timeout?
• [ ] Can you debug by printing the current state name at runtime?
• [ ] Are vision, speed, and aggression parameters externalized to config/data?

---

References

1. Nystrom, Robert. Game Programming Patterns. State Pattern chapter. gameprogrammingpatterns.com
2. Millington, Ian. Artificial Intelligence for Games. Chapters 5–6.
3. GDC Talk: "Building a Better Boss." (Specific year/speaker pending verification.)
4. Yu, Derek. "Designing Spelunky." GDC 2017. GDC Vault

---