[go: up one dir, main page]
Skip to content

Resolve "Update GET /projects API to filter by projects with code embeddings"

Tian Gaorequested to merge
547112-API_to_filter_by_projects_with_code_embeddings into master

What does this MR do and why?

As part of Semantic search: chat with your codebase (&16910), the Language Server will use the GET /projects API to fetch projects to be displayed on the /include <repository> menu

We need to support a new parameter on this API to allow filtering by projects with code embeddings.

  • Update the ProjectsFinder to accept a boolean with_code_embeddings_indexed parameter, and filter by p_ai_active_context_code_repositories.state=ready (see discussion)
  • Update the GET /projects to accept a boolean with_code_embeddings_indexed and pass it on to the ProjectsFinder

References

Related to #547112 (closed)

https://docs.gitlab.com/api/projects/#list-projects

Screenshots or screen recordings

Before After

How to set up and validate locally

p = Project.last
g = p.group
c=Ai::ActiveContext::Connection.create(name: "connection_1", adapter_class: 'Ai::ActiveContext::Adapters::BaseAdapter', active: true, options: { 'url' => 'https://example.com', 'token' => 'secret' }, prefix: 'test')
ens = Ai::ActiveContext::Code::EnabledNamespace.create!(namespace: g, active_context_connection: c)
rep = Ai::ActiveContext::Code::Repository.create!(project: p, enabled_namespace: ens, connection_id: ens.connection_id)
rep.update(state: :ready)

[15] pry(main)> Feature.enabled?(:allow_with_code_embeddings_indexed_projects_filter)
=> true

no code embedding filters: Screenshot_2025-07-04_at_4.36.14_PM

with code embedding filters: Screenshot_2025-07-11_at_12.40.10_PM

Screenshot_2025-07-11_at_12.44.20_PM

[62, 71] in /Users/tiangao/tgao_gitlab/gitlab-development-kit/gitlab/ee/app/finders/ee/projects_finder.rb
   62:       # only apply the code_embeddings_indexed filter when
   63:       # 1. feature flag `allow_with_code_embeddings_indexed_projects_filter` is enabled AND
   64:       # 2. params[:with_code_embeddings_indexed] is present and is true AND
   65:       # 3. project_ids_relation is an array AND
   66:       # 4. project_ids_relation is not empty
=> 67:       items.with_ready_active_context_code_repository_project_ids project_ids_relation
   68:     end
   69: 
   70:     def by_hidden(items)
   71:       params[:include_hidden].present? ? items : items.not_hidden
(byebug) project_ids_relation
["1000000"]
(byebug)

DB query plan

when query with: "http://127.0.0.1:3000/api/v4/projects?with_code_embeddings_indexed=true"

analysis link: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/41098/commands/126345

SELECT
    "projects"."id",
    "projects"."name",
    "projects"."path",
    "projects"."description",
    "projects"."created_at",
    "projects"."updated_at",
    "projects"."creator_id",
    "projects"."namespace_id",
    "projects"."last_activity_at",
    "projects"."import_url",
    "projects"."visibility_level",
    "projects"."archived",
    "projects"."avatar",
    "projects"."merge_requests_template",
    "projects"."star_count",
    "projects"."merge_requests_rebase_enabled",
    "projects"."import_type",
    "projects"."import_source",
    "projects"."approvals_before_merge",
    "projects"."reset_approvals_on_push",
    "projects"."merge_requests_ff_only_enabled",
    "projects"."issues_template",
    "projects"."mirror",
    "projects"."mirror_last_update_at",
    "projects"."mirror_last_successful_update_at",
    "projects"."mirror_user_id",
    "projects"."shared_runners_enabled",
    "projects"."runners_token",
    "projects"."build_allow_git_fetch",
    "projects"."build_timeout",
    "projects"."mirror_trigger_builds",
    "projects"."pending_delete",
    "projects"."public_builds",
    "projects"."last_repository_check_failed",
    "projects"."last_repository_check_at",
    "projects"."only_allow_merge_if_pipeline_succeeds",
    "projects"."has_external_issue_tracker",
    "projects"."repository_storage",
    "projects"."repository_read_only",
    "projects"."request_access_enabled",
    "projects"."has_external_wiki",
    "projects"."ci_config_path",
    "projects"."lfs_enabled",
    "projects"."description_html",
    "projects"."only_allow_merge_if_all_discussions_are_resolved",
    "projects"."repository_size_limit",
    "projects"."printing_merge_request_link_enabled",
    "projects"."auto_cancel_pending_pipelines",
    "projects"."service_desk_enabled",
    "projects"."cached_markdown_version",
    "projects"."delete_error",
    "projects"."last_repository_updated_at",
    "projects"."disable_overriding_approvers_per_merge_request",
    "projects"."storage_version",
    "projects"."resolve_outdated_diff_discussions",
    "projects"."remote_mirror_available_overridden",
    "projects"."only_mirror_protected_branches",
    "projects"."pull_mirror_available_overridden",
    "projects"."jobs_cache_index",
    "projects"."external_authorization_classification_label",
    "projects"."mirror_overwrites_diverged_branches",
    "projects"."pages_https_only",
    "projects"."external_webhook_token",
    "projects"."packages_enabled",
    "projects"."merge_requests_author_approval",
    "projects"."pool_repository_id",
    "projects"."runners_token_encrypted",
    "projects"."bfg_object_map",
    "projects"."detected_repository_languages",
    "projects"."merge_requests_disable_committers_approval",
    "projects"."require_password_to_approve",
    "projects"."max_pages_size",
    "projects"."max_artifacts_size",
    "projects"."pull_mirror_branch_prefix",
    "projects"."remove_source_branch_after_merge",
    "projects"."marked_for_deletion_at",
    "projects"."marked_for_deletion_by_user_id",
    "projects"."autoclose_referenced_issues",
    "projects"."suggestion_commit_message",
    "projects"."project_namespace_id",
    "projects"."hidden",
    "projects"."organization_id"
FROM
    "projects"
    INNER JOIN "p_ai_active_context_code_repositories" ON "p_ai_active_context_code_repositories"."state" = 10
        AND "p_ai_active_context_code_repositories"."project_id" = "projects"."id"
    INNER JOIN "ai_active_context_connections" "active_context_connection" ON "active_context_connection"."id" = "p_ai_active_context_code_repositories"."connection_id"
        AND "active_context_connection"."active" = TRUE
WHERE (EXISTS (
        SELECT
            1
        FROM
            "project_authorizations"
        WHERE
            "project_authorizations"."user_id" = 1
            AND (project_authorizations.project_id = projects.id))
        OR projects.visibility_level IN (0, 10, 20))
AND "projects"."pending_delete" = FALSE
AND "projects"."hidden" = FALSE
/* loading for inspect */
LIMIT 11
/*application:web,correlation_id:01JZ4D596E06E5N9VX5HE01P1K,endpoint_id:GET /api/:version/projects,db_config_database:gitlabhq_development,db_config_name:main,line:/app/finders/projects_finder.rb:69:in `execute'*/

when query without ?with_code_embeddings_indexed=true : "http://127.0.0.1:3000/api/v4/projects"

SELECT
    "projects"."id",
    "projects"."name",
    "projects"."path",
    "projects"."description",
    "projects"."created_at",
    "projects"."updated_at",
    "projects"."creator_id",
    "projects"."namespace_id",
    "projects"."last_activity_at",
    "projects"."import_url",
    "projects"."visibility_level",
    "projects"."archived",
    "projects"."avatar",
    "projects"."merge_requests_template",
    "projects"."star_count",
    "projects"."merge_requests_rebase_enabled",
    "projects"."import_type",
    "projects"."import_source",
    "projects"."approvals_before_merge",
    "projects"."reset_approvals_on_push",
    "projects"."merge_requests_ff_only_enabled",
    "projects"."issues_template",
    "projects"."mirror",
    "projects"."mirror_last_update_at",
    "projects"."mirror_last_successful_update_at",
    "projects"."mirror_user_id",
    "projects"."shared_runners_enabled",
    "projects"."runners_token",
    "projects"."build_allow_git_fetch",
    "projects"."build_timeout",
    "projects"."mirror_trigger_builds",
    "projects"."pending_delete",
    "projects"."public_builds",
    "projects"."last_repository_check_failed",
    "projects"."last_repository_check_at",
    "projects"."only_allow_merge_if_pipeline_succeeds",
    "projects"."has_external_issue_tracker",
    "projects"."repository_storage",
    "projects"."repository_read_only",
    "projects"."request_access_enabled",
    "projects"."has_external_wiki",
    "projects"."ci_config_path",
    "projects"."lfs_enabled",
    "projects"."description_html",
    "projects"."only_allow_merge_if_all_discussions_are_resolved",
    "projects"."repository_size_limit",
    "projects"."printing_merge_request_link_enabled",
    "projects"."auto_cancel_pending_pipelines",
    "projects"."service_desk_enabled",
    "projects"."cached_markdown_version",
    "projects"."delete_error",
    "projects"."last_repository_updated_at",
    "projects"."disable_overriding_approvers_per_merge_request",
    "projects"."storage_version",
    "projects"."resolve_outdated_diff_discussions",
    "projects"."remote_mirror_available_overridden",
    "projects"."only_mirror_protected_branches",
    "projects"."pull_mirror_available_overridden",
    "projects"."jobs_cache_index",
    "projects"."external_authorization_classification_label",
    "projects"."mirror_overwrites_diverged_branches",
    "projects"."pages_https_only",
    "projects"."external_webhook_token",
    "projects"."packages_enabled",
    "projects"."merge_requests_author_approval",
    "projects"."pool_repository_id",
    "projects"."runners_token_encrypted",
    "projects"."bfg_object_map",
    "projects"."detected_repository_languages",
    "projects"."merge_requests_disable_committers_approval",
    "projects"."require_password_to_approve",
    "projects"."max_pages_size",
    "projects"."max_artifacts_size",
    "projects"."pull_mirror_branch_prefix",
    "projects"."remove_source_branch_after_merge",
    "projects"."marked_for_deletion_at",
    "projects"."marked_for_deletion_by_user_id",
    "projects"."autoclose_referenced_issues",
    "projects"."suggestion_commit_message",
    "projects"."project_namespace_id",
    "projects"."hidden",
    "projects"."organization_id"
FROM
    "projects"
WHERE (EXISTS (
        SELECT
            1
        FROM
            "project_authorizations"
        WHERE
            "project_authorizations"."user_id" = 1614863
            AND (project_authorizations.project_id = projects.id))
        OR projects.visibility_level IN (0, 10, 20))
AND "projects"."pending_delete" = FALSE
AND "projects"."hidden" = FALSE
/* loading for inspect */
ORDER BY
    "projects"."id" DESC
LIMIT 11

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #547112 (closed)

Edited by Tian Gao

Merge request reports