Azure DevOps report: How a bug caused ‘sqlite3 for Python’ to go missing from Linux images

Yesterday, Youhana Naseim the Group Engineering Manager at Azure Pipelines provided a post-mortem of the bug, due to which a sqlite3 module in the Ubuntu 16.04 image for Python went missing from May 14th.

The Azure DevOps team identified the bug on May 31st and fixed it on June 26th. Naseim apologized to all the affected customers for the delay in detecting and fixing the issue.

https://twitter.com/hawl01475954/status/1134053763608530945

https://twitter.com/ProCode1/status/1134325517891411968

How Azure DevOps team detected and fixed the issue

The Azure DevOps team upgraded the versions of Python, which were included in the Ubuntu 16.04 image with M151 payload. These versions of Python’s build scripts consider sqlite3 as an optional module, hence the builds were carried out successfully despite the missing sqlite3 module.

Naseim says that,

“While we have test coverage to check for the inclusion of several modules, we did not have coverage for sqlite3 which was the only missing module.”

The issue was first reported by a user who received the M151 deployment containing the bug via the Azure Developer Community on May 20th. But the Azure support team escalated, only after receiving more reports during the M152 deployment on May 31st. The support team then proceed with the M153 deployment, after posting a workaround for the issue, as the M152 deployment would take at least 10 days. Further, due to an internal miscommunication, the support team didn’t start the M153 deployment to Ring 0 until June 13th.

[box type="shadow" align="" class="" width=""]To safeguard the production environment, Azure DevOps rolls out changes in a progressive and controlled manner via the ring model of deployments. [/box]To safeguard the production environment, Azure DevOps rolls out changes in a progressive and controlled manner via the ring model of deployments.

The team then resumed deployment to Ring 1 on June 17th and reached Ring 2 by June 20th. Finally, after a few failures, the team fully deployed the M153 deployment by June 26th.

Azure’s future workarounds to deliver timely fixes

The Azure team has set out plans to make improvements to their deployment and hotfix processes with an aim to deliver timely fixes. Their long term plan is to provide customers with the ability to choose to revert to the previous image as a quick workaround for issues introduced in new images. The detailed medium and short plans are as given below:

Medium-term plans

• Add the ability to better compare what changed on the images to catch any unexpected discrepancies that our test suite might miss.
• Increase the speed and reliability of deployment process.

Short term plans

• Build a full CI Pipeline for image generation for verifying images daily.
• Add test coverage for all modules in the Python standard library including sqlite3.
• Improving the support team's communication with the support team to escalate issues more quickly.
• Add telemetry, so it would be possible to detect and diagnose issues more quickly.
• Implement measures, which will enable reverting to prior image versions quickly and mitigate issues faster.

Visit the Azure Devops status site for more details.

Read More

Ubuntu has decided to drop i386 (32-bit) architecture from Ubuntu 19.10 onwards

Xubuntu 19.04 releases with latest Xfce package releases, new wallpapers and more

Ubuntu 19.04 Disco Dingo Beta releases with support for Linux 5.0 and GNOME 3.32