ProjectsGrades.grades_for using vulnerability_statistics.traversal_ids

What does this MR do and why?

Implement ProjectsGrades.grades_for using vulnerability_statistics.traversal_ids and archived columns introduced in !177591 (merged).

• Add new scopes to Vulnerabilities::Statistic: by_group (including subgroups) and by_group_excluding_subgroups.
• Change .grades_for to iterate over vulnerable objects. If the vulnerable is a group and the remove_cross_join_from_vulnerabilities_projects_grade feature flag is enabled for that group, then it uses the new by_group and by_group_excluding_subgroups scopes.
• Add a new DB index: index_vulnerability_statistics_on_traversal_ids.
• Remove unused constant: BATCH_SIZE.
• Remove unused rspec context: when multiple batches are required.

This addresses performance issues for large groups such as gitlab-org. See #508631 (comment 2264738914)

The performance issue was caused by Resolve cross-join in ProjectsGrade.grades_for ... (!172720 - merged), which aimed at solving a cross-database join, and that was behind the same feature flag.

Note: Vulnerabilities::Statistic.by_group includes subgroups to consistent with Vulnerabilities::Read.by_group.

References

Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

SQL queries

group excluding subgroups

!177892 (comment 2297674778)

SELECT "vulnerability_statistics"."letter_grade", array_agg(project_id) project_ids
FROM "vulnerability_statistics"
WHERE "vulnerability_statistics"."traversal_ids" = '{9970}'
AND "vulnerability_statistics"."archived" = FALSE
GROUP BY "vulnerability_statistics"."letter_grade"

query plan with index_vulnerability_statistics_on_traversal_ids_and_letter_grade: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/35206/commands/109190

group excluding subgroups, letter grade filter

!177892 (comment 2297676948)

SELECT "vulnerability_statistics"."letter_grade", array_agg(project_id) project_ids
FROM "vulnerability_statistics"
WHERE "vulnerability_statistics"."traversal_ids" = '{9970}'
AND "vulnerability_statistics"."archived" = FALSE
AND "vulnerability_statistics"."letter_grade" = 0
GROUP BY "vulnerability_statistics"."letter_grade"

query plan with index_vulnerability_statistics_on_traversal_ids_and_letter_grade: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/35206/commands/109189

group including subgroups

!177892 (comment 2297678679)

SELECT "vulnerability_statistics"."letter_grade", array_agg(project_id) project_ids
FROM "vulnerability_statistics"
WHERE "vulnerability_statistics"."traversal_ids" >= '{9970}'
AND "vulnerability_statistics"."traversal_ids" < '{9971}'
AND "vulnerability_statistics"."archived" = FALSE
GROUP BY "vulnerability_statistics"."letter_grade"

query plan with index_vulnerability_statistics_on_traversal_ids_and_letter_grade: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/35206/commands/109181

group including subgroups, letter grade filter

!177892 (comment 2297680145)

SELECT "vulnerability_statistics"."letter_grade", array_agg(project_id) project_ids
FROM "vulnerability_statistics"
WHERE "vulnerability_statistics"."traversal_ids" >= '{9970}'
AND "vulnerability_statistics"."traversal_ids" < '{9971}'
AND "vulnerability_statistics"."archived" = FALSE
AND "vulnerability_statistics"."letter_grade" = 3
GROUP BY "vulnerability_statistics"."letter_grade"

query plan with index_vulnerability_statistics_on_traversal_ids_and_letter_grade: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/35206/commands/109180

Related to #503387 (closed)