EC
TILT
RedisWARNINGCriticalSentinel / High AvailabilityHIGH confidence

Sentinel has entered TILT mode due to a system clock anomaly

Production Risk

High — while in TILT mode, automatic failover is disabled; a master failure becomes a manual incident.

What this means

Redis Sentinel enters TILT mode when it detects that the system clock has moved backward or has ticked too fast (more than 2 seconds between two consecutive timer firings). In TILT mode, Sentinel suspends all active monitoring decisions — it continues to collect information but will not trigger failovers, will not declare masters as down (ODOWN), and will not promote replicas.

Why it happens
  1. 1The host system clock was adjusted backward (e.g., NTP correction, VM clock drift after a snapshot/resume).
  2. 2The system was under such heavy load that Sentinel's timer fired more than 2 seconds late.
  3. 3A virtual machine was paused and resumed, causing a large jump in perceived time.
  4. 4Running Sentinel on a heavily overloaded host sharing CPU with other workloads.
How to reproduce

A VM hosting a Sentinel node is resumed from a snapshot; the system clock jumps several seconds, triggering TILT mode.

trigger — this will error
trigger — this will error
# Query Sentinel for its current state
SENTINEL info sentinel

# Check if Sentinel is in TILT mode
SENTINEL masters

expected output

# sentinel
sentinel_masters:1
sentinel_tilt:1
sentinel_tilt_since_seconds:42
sentinel_running_scripts:0

Fix 1

Wait for TILT mode to clear automatically

WHEN The clock anomaly was a one-time event (e.g., a single NTP correction)

Wait for TILT mode to clear automatically
# TILT mode clears automatically after 30 seconds of normal timer behaviour
# Monitor with:
SENTINEL info sentinel
# Wait until sentinel_tilt:0

Why this works

Sentinel exits TILT mode on its own after 30 consecutive seconds without detecting a clock anomaly.

Fix 2

Investigate and fix the underlying clock issue

WHEN TILT mode recurs or persists

Investigate and fix the underlying clock issue
# Check NTP sync status on the Sentinel host
timedatectl status
ntpstat

# Check system load that may delay Sentinel's timer
top -b -n 1 | head -20

Why this works

Fixing the root cause (clock sync, host overload, VM scheduling) prevents recurrence.

Fix 3

Ensure Sentinel runs on a dedicated, lightly loaded host

WHEN TILT is triggered by host overload rather than clock drift

Ensure Sentinel runs on a dedicated, lightly loaded host
# sentinel.conf — no specific setting, but host isolation helps
# Run Sentinel on a host with predictable CPU scheduling
# Avoid co-locating with CPU-intensive workloads

Why this works

Predictable CPU scheduling ensures Sentinel's internal timer fires on time, preventing false TILT triggers.

What not to do

✕ Ignore recurring TILT events

Repeated TILT mode means Sentinel is not protecting your cluster; a real master failure during TILT will go undetected and no automatic failover will occur.

✕ Disable NTP on Sentinel hosts to prevent clock jumps

An unsynchronised clock leads to far worse drift over time; use ntpd/chrony with gradual slewing instead of disabling time sync.

Version notes
Redis Sentinel (all versions)

TILT mode has been present since early Sentinel versions as a safety mechanism against clock anomalies.

Redis 7.x

No change to TILT behaviour; the 30-second recovery window and 2-second threshold remain the same.

Sources
Official documentation ↗

Redis Sentinel documentation — TILT mode

Content generated with AI assistance and reviewed for accuracy. Found an error? hello@errcodes.dev

← All Redis errors