detecting-database-deadlocks

Database Deadlock Detector

Overview

Detect, analyze, and prevent database deadlocks in PostgreSQL, MySQL, and MongoDB by examining lock wait graphs, parsing deadlock log entries, identifying the application code paths that cause lock ordering conflicts, and implementing preventive patterns.

Prerequisites

• Database credentials with access to lock monitoring views (pg_locks, INNODB_LOCK_WAITS)
• psql or mysql CLI for executing diagnostic queries
• PostgreSQL: log_lock_waits = on and deadlock_timeout = 1s configured
• MySQL: innodb_print_all_deadlocks = ON for deadlock logging to error log
• Access to database error logs for deadlock event parsing
• Application source code access for identifying lock-inducing code paths

Instructions

1. Check for currently blocked transactions and their blockers:

   • PostgreSQL: SELECT blocked.pid AS blocked_pid, blocked.query AS blocked_query, blocking.pid AS blocking_pid, blocking.query AS blocking_query FROM pg_stat_activity blocked JOIN pg_locks bl ON bl.pid = blocked.pid JOIN pg_locks bl2 ON bl2.locktype = bl.locktype AND bl2.relation = bl.relation AND bl2.pid != bl.pid JOIN pg_stat_activity blocking ON blocking.pid = bl2.pid WHERE NOT bl.granted
   • MySQL: SELECT * FROM information_schema.INNODB_LOCK_WAITS

2. Parse recent deadlock events from database logs:

   • PostgreSQL: Search logs for ERROR: deadlock detected entries, which include the two conflicting queries and the lock types
   • MySQL: Run SHOW ENGINE INNODB STATUS\G and examine the LATEST DETECTED DEADLOCK section
   • Extract: transaction IDs, queries involved, tables and rows locked, and which transaction was rolled back

3. Construct the lock wait graph from the deadlock log. Map which transaction held which lock and which lock each transaction was waiting for. The circular dependency reveals the deadlock cycle. Identify the specific rows or index ranges involved.

4. Trace the deadlocking queries back to application code. Use Grep to find the SQL statements in the codebase and identify the transaction boundaries (BEGIN/COMMIT blocks or ORM transaction decorators). Map the full sequence of operations within each transaction.

5. Identify the root cause pattern:

   • Opposite lock ordering: Transaction A locks row 1 then row 2; Transaction B locks row 2 then row 1. Fix by ensuring consistent lock ordering.
   • Index gap locks (MySQL): UPDATE/DELETE on non-existent rows creates gap locks that conflict. Fix by adding the target row first or using READ COMMITTED isolation.
   • Foreign key lock escalation: INSERT into child table acquires shared lock on parent row, conflicting with UPDATE on parent. Fix by locking parent first explicitly.
   • Implicit lock promotion: SELECT with FOR UPDATE followed by UPDATE promotes shared to exclusive lock. Fix by acquiring the exclusive lock upfront.

6. Implement deadlock prevention strategies:

   • Enforce consistent lock ordering: always lock tables/rows in alphabetical or ID order within transactions
   • Minimize transaction duration: move non-database operations (API calls, file I/O) outside the transaction
   • Use SELECT ... FOR UPDATE NOWAIT or SKIP LOCKED to fail fast instead of waiting
   • Reduce transaction isolation level from SERIALIZABLE to READ COMMITTED where possible

7. Add retry logic for deadlock victims. When the database aborts a transaction due to deadlock, catch the error (PostgreSQL error code 40P01, MySQL error code 1213) and retry the entire transaction up to 3 times with a short random delay.

8. Monitor deadlock frequency over time. Create a query or script that counts deadlock events per hour from the database logs. Alert when deadlock frequency exceeds the baseline by more than 3x.

9. For persistent deadlocks on specific tables, consider advisory locks (pg_advisory_lock() in PostgreSQL) to serialize access to contended resources at the application level, avoiding database-level lock contention entirely.

10. Document all identified deadlock patterns, root causes, and fixes in a deadlock analysis report for the development team.

Output

• Lock wait graph visualization showing the circular dependency between transactions
• Deadlock analysis report with root cause, affected queries, and code paths
• Code fix recommendations with before/after transaction ordering examples
• Retry logic implementation for deadlock victim transactions
• Monitoring queries/scripts for tracking deadlock frequency trends

Error Handling

Error	Cause	Solution
PostgreSQL error 40P01: deadlock detected	Circular lock dependency between transactions	Implement retry logic; fix lock ordering in application code; reduce transaction scope
MySQL error 1213: Deadlock found when trying to get lock	InnoDB detected circular wait in lock wait graph	Enable innodb_print_all_deadlocks; analyze SHOW ENGINE INNODB STATUS; implement retry logic
Lock wait timeout (not deadlock)	Transaction holding lock too long, exceeding lock_wait_timeout	Investigate the blocking transaction; increase timeout or implement NOWAIT; optimize the long-running transaction
Phantom deadlocks in monitoring	Transient lock waits resolved before deadlock detection runs	Increase monitoring frequency; use database deadlock log instead of snapshot queries; set deadlock_timeout lower
Deadlock frequency increases after schema change	New index or constraint creates additional lock targets	Analyze new lock patterns with EXPLAIN and pg_locks; adjust transaction scope to avoid locking new index entries

Examples

Classic opposite-ordering deadlock in an order processing system: Transaction A processes order 100 (locks order row), then updates inventory for product 50 (waits for inventory lock). Transaction B processes order 200 with product 50 (locks inventory row), then updates order 100 status (waits for order lock). Fix: always lock inventory first, then order, regardless of the business flow.

MySQL gap lock deadlock on a queue table: Two workers concurrently DELETE FROM job_queue WHERE status = 'pending' LIMIT 1. InnoDB gap locks on the index range conflict even though the workers target different rows. Fix: use SELECT ... FOR UPDATE SKIP LOCKED to skip already-locked rows, or add unique job IDs and target specific rows.

Foreign key deadlock between parent and child inserts: Concurrent transactions inserting into order_items (child) acquire shared locks on orders (parent) for FK validation. A third transaction updating orders requires an exclusive lock and deadlocks with the shared FK locks. Fix: explicitly SELECT ... FOR UPDATE on the parent order row before inserting child items.

Resources

• PostgreSQL deadlock detection: https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-DEADLOCKS
• MySQL InnoDB deadlocks: https://dev.mysql.com/doc/refman/8.0/en/innodb-deadlocks.html
• PostgreSQL lock monitoring: https://wiki.postgresql.org/wiki/Lock_Monitoring
• Advisory locks in PostgreSQL: https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS
• InnoDB lock types explained: https://dev.mysql.com/doc/refman/8.0/en/innodb-locking.html