Distributed architecture with a message queue, worker nodes, and a distributed database, handling 100 GB of log data per day