Make audit table retention period configurable, supporting much longer timeframes
Problem to solve
As an enterprise customer, we have to demonstrate to auditors that certain processes were followed (in great detail) at least every year, and often have data retention policies for this data that extends well beyond 1 year. Currently GitLab audit event data is purged from the DB after 1 year ( https://docs.gitlab.com/ee/api/events.html#event-time-period-limit ). This makes the audit table (UI and API) an unreliable source for auditing workflows in GitLab.
The reason given for purging data after 1 year relates to user contribution statistics, however this use-case is for enterprise audits which have different requirements.
Further details
This especially affects customers in Banking, Medical, and government industries.
As an auditor, I should be able to pull all relevant events for key workflows for an entire year, even if I am conducting the audit 6 months after the end of the time period being audited. Additionally I should be able to go back 2 years later to verify or revisit an audit to verify findings.
Proposal
Make the audit table retention policy configurable, and ensure performance testing ensures reliable performance for at least 3 years retention for our largest enterprise customers.
Ideally we would support longer time frames, including "never purge."
What does success look like, and how can we measure that?
Audit table retention is configurable, and large customers do not experience performance problems.
Links / references
https://docs.gitlab.com/ee/api/events.html#event-time-period-limit
/cc @jeremy_