discussion: Delete Ci::BuildMetadata after Ci::Pipeline archival

The Problem

is articulated more broadly in the parent epic: &13886

This issue is to track implementation of a specific approach to reduce the volume of TOAST data consumed by records on ci_builds_metadata

This Specific Proposal

Was articulated with time window options in Enable pipeline archival on Gitlab.com (#521005 - closed):

Once a pipeline is archived and retry is disabled, we can clear the config_options and config_variables columns that from ci_builds_metadata that &13886. Those columns represent the overwhelming majority of the 8.31TiB of TOAST data we currently store on that table.

Disabling retry on these builds over 1 year old would allow use to clear the config_options and config_variables column data. This is not customer-provided data, it's a cache of parsed YAML configuration (the YAML is the canonical customer data) that we use to execute the build and its retries. If retry isn't available, there's no further use.

Enable archival after 1 month

We can get ~8TiB of disk space (roughly immediately) and permanently reduce the growth rate back by archiving (disable play and retry) after 1 month.

Reducing this to 1 month potentially means this table will never be a major disk space consumption factor again.

Considered Alternative: We can get ~4TiB of disk space (roughly immediately) and permanently reduce the growth rate by archiving (disable play and retry) after 1 year.

Engineering's Recommendation: We agreed that we should push for archiving after 1 month. This is a more aggressive plan, but if the idea is good, I think it's valuable to take the full step now, instead of announcing 1 year now and then another reduction to 1 month in the future. As a communication strategy, I think we should make one decisive policy announcement.