xsk<p><a href="https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">status.cloud.google.com/incide</span><span class="invisible">nts/ow5i3PPK96RduMcb1SsW</span></a> </p><p>How to say that you phased rolled out untested, null pointer exception causing codebase, in a vital and global policy system that has also obviously never been benchmarked for time to full recovery - the initial updates had no ETA even though the root cause was apparently known from the first 10 mins - in 5 paragraphs.</p><p>And the scary part ? no mention of violations of any coding practices and code reviews, nothing to base a policy system on policy schemata or versions, nor unit tests that would have stopped a phased rollout.</p><p>For some reason they picked to announce static analysis improvements ( to avoid null pointer exceptions ), mandatory feature flags ( in order to not have to wait 20-30 minutes for CI/CD ) and modularization ( blast radius control ), which should have been the baseline anyway, no process changes.</p><p>Very embarrassing, I hope whoever decided to remove guardrails and took actions to eventually effectively nullify the phased roll out process, presses on the brakes and starts realizing that they need to push for the stability/credibility currency increase once again...</p><p>"Successful null release" </p><p><a href="https://social.treehouse.systems/tags/gcp" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>gcp</span></a> <a href="https://social.treehouse.systems/tags/gcpdown" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>gcpdown</span></a></p>