ES DSL Design

Design, Architecture & Migration

Architecture

The Elasticsearch DSL stack introduces a layered, declarative architecture that replaces hard-coded JSON builders and mappings with a unified DSL pipeline.

flowchart TD
    %% Entry
    A["Application / Indexing Worker<br/>(e.g. Elastic::Indexer)"] --> B["Search::Elastic::References::<Model>"]

    %% Feature flag
    B -- "FF :use_dsl = ON" --> C["References::<Model>DslReference"]
    B -- "FF OFF" --> D["References::<Model> (Legacy)"]

    %% DSL path
    C --> REF["Search::Elastic::Dsl::Models::<Model> < Base"]
    REF --> BUILDERS
    BUILDERS --> CORE
    REF --> TYPE
    REF -.-> ES["Elasticsearch Index"]

    %% Builders
    subgraph BUILDERS["Search::Elastic::Dsl::Builders::*"]
        JB["JsonBuilder<br/>→ builds indexed document"]
        MB["MappingBuilder<br/>→ builds ES mapping tree"]
        PB["ProxyBuilder<br/>→ enriches records"]
        SB["SettingsBuilder<br/>→ merges model + default settings"]
    end

    %% Core DSL
    subgraph CORE["Core DSL Modules"]
        F["Field DSL<br/>→ defines field registry"]
        SV["SchemaVersions<br/>→ current_version logic"]
        ST["Settings<br/>→ evaluates settings block"]
    end

    %% Type bridge
    subgraph TYPE["Search::Elastic::Types::DslType"]
        T1[".for(:model)<br/>→ binds DSL model"]
        T2["mappings / settings / index_name<br/>→ delegate to DSL model"]
    end

Layer Responsibilities

Layer	Purpose
Application / Worker	Triggers indexing (e.g. Elastic::Indexer)
Reference layer	Entry point for a model’s indexing logic; gated by feature flag
DslReference	Adapter delegating to the new DSL model
Dsl::Models::	Defines index schema declaratively via field, settings, schema_versions
Builders	Generate JSON, mappings, and settings from model DSL definitions
Core DSL	Shared logic for field registry, versioning, and settings evaluation
Types::DslType	Provides typed access to mappings and settings for ES type inspection

---

Builder ↔️ Core DSL Interaction

The Builders layer consumes the definitions from the Core DSL modules to construct the actual Elasticsearch artifacts.

flowchart TD
    %% === BUILDERS LAYER ===
    subgraph BUILDERS["Search::Elastic::Dsl::Builders::*"]
        JB["JsonBuilder<br/>–––––––––––––––––––––––––––––––<br/>Builds Elasticsearch JSON documents for indexing.<br/>Applies field definitions, compute rules, and version filters."]
        MB["MappingBuilder<br/>–––––––––––––––––––––––––––––––<br/>Generates Elasticsearch mappings.<br/>Includes only fields active for the current schema version."]
        PB["ProxyBuilder<br/>–––––––––––––––––––––––––––––––<br/>Creates enriched runtime proxies combining raw records<br/>and preloaded enrichment data for indexing."]
        SB["SettingsBuilder<br/>–––––––––––––––––––––––––––––––<br/>Builds index settings including analyzers,<br/>tokenizers, and normalization filters."]
    end


    %% === CORE DSL LAYER ===
    subgraph CORE["Search::Elastic::Dsl::* (Core Modules)"]
        F["Field DSL<br/>–––––––––––––––––––––––––––––––<br/>Central definition of all fields:<br/>type, compute, preload, default, and conditions."]
        SV["SchemaVersions<br/>–––––––––––––––––––––––––––––––<br/>Defines available schema versions and<br/>resolves which one is currently active."]
        ST["Settings<br/>–––––––––––––––––––––––––––––––<br/>Defines reusable analyzer, filter,<br/>and tokenizer configuration for indices."]
    end


    %% === RELATIONSHIPS ===
    JB --> F
    JB --> SV
    MB --> F
    MB --> SV
    PB --> F
    SB --> ST


    %% === STYLING ===
    classDef builder fill:#e8f1ff,stroke:#3366cc,stroke-width:1px,color:#000;
    classDef core fill:#f2faff,stroke:#66b3ff,stroke-width:1px,color:#000;
    classDef title fill:none,stroke:none,font-weight:bold;

    class BUILDERS,JB,MB,PB,SB builder;
    class CORE,F,SV,ST core;

Flow Summary:

• JsonBuilder queries the Field DSL registry to construct the indexed document.
• MappingBuilder reads both Field and SchemaVersions for dynamic mapping trees.
• SettingsBuilder merges runtime settings using the Settings DSL.
• The output of all builders produces the final Elasticsearch artifacts: JSON docs, mappings, and settings.

---

Other Flows

All other indexing, search, and read flows remain unchanged.

• Existing search APIs, indexers, and background workers continue to use the Search::Elastic::References::<Model> interface.
• The only addition is the feature-flag bridge that toggles between legacy and DSL implementations.
• No schema or runtime dependency changes occur when the flag is disabled.

---

Migration Strategy

1. Add scope-aware resolver in Search::Elastic::References::Vulnerability

def self.for_scope(scope)
  resolved = scope.is_a?(Project) || scope.is_a?(Group)
  if resolved && Feature.enabled?(:use_dsl, resolved)
    ::Search::Elastic::References::DslReference
  else
    self
  end
end

2. **Update reference usage in Vulnerability and Vulnerabilities::Read (smiliarly to **

# ee/app/models/ee/vulnerability.rb
def elastic_reference
  klass = ::Search::Elastic::References::Vulnerability.for_scope(project)
  klass.new(id, Vulnerabilities::Read.generate_es_parent(project)).serialize
end

# ee/app/models/vulnerabilities/read.rb
def elastic_reference
  ::Search::Elastic::References::Vulnerability
    .for_scope(project)
    .serialize(self)
end

The for_scope method ensures the correct reference class (Vulnerability or VulnerabilityDsl) is used per project or group, based on the feature flag.

3. Keep constants shared Both Vulnerability and VulnerabilityDsl define identical constants (DOC_TYPE, INDEX_NAME, etc.), so existing calls remain valid.

4. Leave other usages unchanged Files using .index, .DOC_TYPE, or static field lists do not need modification.

   • ::Search::Elastic::Types::Vulnerability.index_name can stay unchanged initially.

5. Feature flag rollout FF scoped per project or group. Default off (legacy path), enable gradually, then switch fully once validated.