How Grab Improved Mobile App Stability by Safely Handling Surge in AI-Generated Code Changes

The growing adoption of AI coding assistants has accelerated development and increased the volume of code changes per release, bringing more opportunities for defects and crashes, says Grab. To address these challenges, they implemented a layered defense strategy to prevent, detect, and contain crashes throughout the development and release lifecycle.

Grab observes that "AI is speeding up how fast code is written; the layered defense is what makes that speed safe":

At [Grab's] scale, crashes leaking into production are an ongoing risk. A high volume of merge requests lands every week across many product teams, many shipping features behind flags that are switched on independently of releases. Any one of those changes can introduce a crash.

Beyond degrading the user's experience, a mobile app's crash reduces crash-free session rates and, as a consequence, the app's Google Play ranking. It also has an increased cost because not all users enable automatic updates, meaning that "a mobile crash that ships stays on users' devices" for longer. Furthermore, not all crashes come with clean, symbolicated stack trace, which means that "finding the cause means cross-referencing release timelines, flag rollouts, and session metadata".

Grab's layered defense follows a shift-left strategy aimed at catching crashes as early as possible through a series of safeguards along the development cycle phases: pre-merge through pre-production, release control, and post-release response.

During the pre-merge to pre-production phase, the goal is prevention, which Grab describes as its "primary line of defense". To achieve this, Grab engineers use static code analyzers such as Android Lint and SwiftLint to catch issues before code even merges via platform-specific, community-based, and custom lint rules; automated and manual testing, including unit tests, UI tests, and quality assurance (QA) checks; and Grab Early Access (GEA), a pre-production gate where early builds are distributed to internal users and actively monitored for crashes:

A crash that only a handful of internal users see in GEA would, at production scale, translate into a substantial spike in the crash rate, extra support load, and potentially a hotfix. Catching these crashes before they ever reach production users makes GEA one of our most important stability gates.

During the release phase, Grab uses staged rollout and feature flags to control production exposure to crashes. Staged rollout enables a new app version to reach only a small percentage of users initially, so engineers can monitor its crashes and possibly pause the rollout if an abnormal crash rate is detected. Features flags, in turn, allow teams to activate features gradually while monitoring their impact on crash rates. If a feature causes crash rates to exceed predefined thresholds, it be disabled immediately without requiring a new release.

During the post-release phase, Grab focuses on rapid incident response using mechanisms to detect, investigate, and contain production crashes. These include real-time crash monitoring using Firebase Crashlytics and BigQuery to identify anomalies as soon as a build goes live; automated triage and enriched diagnostics to aggregate crash metadata and stack traces, helping teams to map crashes to specific PRs; and feedback loops and post-mortem analysis to extend lint rules and test coverage.

There is much more to Grab's layered defense approach than can be covered here. Be sure to read the full article for all the details.