AI Usage Dashboard

Update:

This was our workflowproblem validation issue.

Here is the workflowsolution validation epic with mapping of the problems and possible solutions for adding "AI Impact" to Category:Value Stream Management. More feedback is welcome in that epic.

Customer request from @mfinch:

Do you know if there is anyway to see how many people are utilizing the coding suggestions api beta in an org or even by repo which have it used for?

Idea:

• new Value Stream management report for users of AI features and impacts to productivity metrics
• Start with just how many are using what AI features.
• Metric ideas
  • Number of AI actions
  • Estimations of time saved (faster deployments, faster time to resolve MRs, others?0
  • % of users using AI features
  • Comparison of users using AI vs none

Existing VSM dashboard: https://gitlab.com/groups/gitlab-org/-/analytics/dashboards/value_streams_dashboard?query=gitlab-org/gitlab

Take these existing metrics and filter them based on user's using AI features

We already have service ping datapoints

these events AND ((event_category = 'Llm::ExecuteMethodService' --tracking for all but Code Suggestions AND event_action = 'execute_llm_method') OR event_category IN ('API::Ml::AiAssist', 'API::CodeSuggestions')) --Code Suggestions has slightly different tracking

*/
SELECT 
  *,
  f.value['schema']::VARCHAR  AS context_data_schema,
  f.value['data']             AS context_data
FROM common_mart.mart_behavior_structured_event, 
lateral flatten(input => TRY_PARSE_JSON(contexts), path => 'data') f
WHERE app_id = 'gitlab_ai_gateway' AND behavior_at >= '2023-09-01'


) , suggestions_with_user_id AS ( SELECT \*, context_data\['gitlab_global_user_id'\] as gitlab_global_user_id, context_data\['gitlab_realm'\]::VARCHAR as gitlab_realm FROM suggestions_with_context WHERE context_data_schema like 'iglu:com.gitlab/code_suggestions_context/jsonschema/%' )https://docs.gitlab.com/ee/user/ai_features.html

Really rough vision of how this could manifest itself in VSM