Source: https://medium.com/@bugfreeai/system-design-interview-with-a-meta-staff-engineer-designing-a-task-scheduler-1a5041b4860e The note follows the interview section by section. Requirements discuss recurring and ad-hoc tasks, resource constraints and optimization goals. Scalability covers a message queue, priority queues, task splitting and dynamic worker scaling. Fault tolerance covers detection, retries with exponential backoff and escalation. Design considerations then weigh consistency, availability, performance and fairness. The main components section names the scheduler service, the queue, the worker pool and the database. Kafka, RabbitMQ and AWS SQS are the queue options named, and Prometheus, Grafana, PagerDuty and Slack handle monitoring and alerts. Five best practices close the note.
Overview
The article describes a mock system design interview with a Senior Engineer (E6) from Meta, focused on designing a Task Scheduler.
Requirements Discussion
Task Types
Clarify which kinds of tasks the system will handle:
- Recurring tasks: jobs that repeat on a schedule (e.g., daily backups, weekly reports)
- Ad-hoc (one-time) tasks: tasks executed once (e.g., handling a user request)
Resource Constraints
Identify the resources needed to execute tasks:
- Runtime
- Network bandwidth
- CPU consumption
- Memory
- Other resources
Optimization Goals
Determine which parts of the system need optimizing:
- Reduce latency
- Increase throughput
- Ensure resource fairness between tasks
- Ensure availability
Scalability
Parallel Task Processing With a Message Queue
Use a message queue to:
- Decouple task creation from task execution
- Distribute tasks across many worker nodes
- Prevent bottlenecks
Technologies that can be used:
- Kafka
- RabbitMQ
- AWS SQS
Handling Complex Tasks
For tasks that consume many resources or take a long time:
Priority Queue
- Prioritize small, fast tasks
- Ensure important tasks are processed first
Splitting Tasks by Type
- Split tasks by type or resource requirements
- Separate complex tasks from simple ones
- Prevent large tasks from blocking small ones
Dynamic Worker Scaling
- Automatically add workers when a surge in heavy tasks is detected
- Ensure the system has enough resources to handle high load
Fault Tolerance
Failure Detection and User Notification
When a task fails, the system should:
- Record the error in detail
- Notify the user promptly
Tools that can be used:
- Monitoring: Prometheus, Grafana
- Alerting: PagerDuty, Slack notifications
Retry Mechanism
For transient failures (e.g., network problems, temporarily unavailable resources):
- Retry the task a configurable number of times
- Apply an exponential backoff strategy to avoid overloading the system
Benefits of exponential backoff:
- Gives the system time to recover
- Reduces simultaneous pressure on the failing service
- Avoids the "service comes back up and dies again" situation caused by being flooded too quickly
Handling Persistent Failures
If a task keeps failing after many retries, the system needs:
- Escalation: move the task to manual handling or alert the operations team
- Isolation: separate the failing task from the main queue so it doesn't affect other tasks
- Detailed error logs: provide detailed logs to help users diagnose the problem
- Manual management: let users reschedule or cancel failed tasks
Main System Components
Scheduler Service
- Manages task schedules
- Creates tasks and pushes them onto the queue
Message Queue
- Stores tasks waiting to be processed
- Ensures task durability
Worker Pool
- Worker nodes that process tasks
- Can scale dynamically based on load
Database
- Stores task metadata
- Stores execution history
- Stores task state
Monitoring & Alerting
- Monitors system performance
- Alerts on errors or high load
Design Considerations
Consistency
- Ensure tasks are not lost or duplicated
- Handle race conditions when multiple workers pick up the same task
Availability
- Replication for critical components
- Automatic failover when problems occur
Performance
- Optimize latency from task creation to the start of execution
- Optimize throughput (tasks processed per second)
Fairness
- Ensure tasks are processed fairly
- Avoid starving some tasks of resources
Best Practices
- Start with an MVP: approach the problem MVP-first, solve the core design, then expand
- Clarify constraints: clarify the constraints on actors (locks, race conditions, chosen protocol) before considering scalability
- Use exponential backoff: apply a smart retry mechanism with exponential backoff to avoid overloading the system
- Monitor from the start: set up monitoring and alerting from the beginning so problems can be detected and handled early
- Separate task types: split tasks by type so large tasks don't block small ones
See also: Kafka DLQ And Retry · Retry And Circuit Breaker · Outbox Pattern