How would you implement a distributed task queue in Go without using an external framework?
Use Redis lists (BLPOP/RPUSH) or Kafka topics as the queue, goroutines as workers, and implement heartbeat/lease mechanisms for at-least-once delivery with a dead letter queue for failures.
Redis-based task queue
Production considerations
At-least-once delivery: use RPOPLPUSH to move task to processing list, remove after success
Idempotency: store processed task IDs in DB with a UNIQUE constraint to handle duplicate delivery
asynq library: production-grade Redis-based queue with retries, scheduling, and dashboard
Kafka with segmentio/kafka-go: durable, ordered, replayable — prefer over Redis for high-volume event streams
Dead letter queue: capture failed tasks for manual inspection and replay after bug fixes