How Dropbox uses automated data center operations to reduce server outage and downtime

Today, in a blog post, Dropbox explained how the Prilo system used by the team has automated most of the processes of the company, that were previously manually attended to by Dropbox personnel. Pirlo is used by Dropbox in two main areas- validate and configure network switches and ensure the reliability of servers before entering production. This has, in turn, helped Dropbox to safely manage their physical infrastructure operations with ease.

Pirlo consists of a distributed MySQL-backed job queue built by Dropbox itself, using primitives like gRPC, service discovery, and our managed MySQL clusters.

Switch provisioning at Dropbox is handled by the TOR STarter which is a Pirlo component. The TOR Starter validates and configures switches in Dropbox datacenter server racks, PoP server racks, and at the different layers of the data center fabric; responsible to connect racks in the same facility together.

Server provisioning and repair validation is handled by Pirlo Server Validation.

All new servers arriving at the company are validated using this component. Repaired servers are also validated before they are transitioned back into production.

Pirlo has automated these manual processes at Dropbox and has led to a reduction in downtime, outages, and inefficiencies associated with the incomplete or erroneous fixing of the systems. By reducing manual work, employees can now focus their attention to more value adding jobs.

Before using Pirlo, the above tasks had to be performed by operations engineers and subject matter experts who used various server error logs to take appropriate actions to fix failed hardware. After applying the remediation actions, the engineer would send the machine back into production by sending the server to Dropbox re-imaging system. If the remediation actions didn’t fix the system or properly prepare it for re-imaging, the server would be sent back to the operations engineer for additional fixing. This would end up consuming a lot of the operation engineer's time as well as company resources.

Operating engineers who used Pirlo system steadily increased their output by 40+%. The automation of manual tasks allowed engineers to address more issues in the same amount of time.

You can head over to Dropbox’s official blog to explore the workings of Pirlo and how it benefited the organization.