Support dynamic job dependencies in needs:parallel:matrix

Summary

Enable variable expansion for matrix variables in needs:parallel:matrix to support dynamic job dependencies based on matrix values.

Problem

Currently, when using parallel:matrix to create multiple jobs, you cannot dynamically reference the matrix variables in needs:parallel:matrix. This forces users to manually create separate jobs for each matrix permutation, which doesn't scale well.

Validated Solution

Based on the PoC in Draft: PoC: Implement matrix expressions (!202964 - closed), we will implement matrix expressions using the $[[ matrix.VARIABLE_NAME ]] syntax.

Syntax

unit-test:
  stage: test
  script:
    - echo "Running tests for $OS-$ARCH-$ENV"
  needs:
    - job: build
      parallel:
        matrix:
          - OS: $[[ matrix.OS ]]
            ARCH: $[[ matrix.ARCH ]]
            ENV: $[[ matrix.ENV ]]
  parallel:
    matrix:
      - OS: ["ubuntu", "alpine", "centos"]
        ARCH: ["amd64", "arm64"]
        ENV: ["dev", "prod"]

How it works

• Matrix expressions in needs:parallel:matrix use the $[[ matrix.VARIABLE_NAME ]] syntax to reference variables from the current job's matrix
• Each parallel job will resolve its own matrix values when determining dependencies
• This allows 1:1 mapping between matrix jobs across stages without manual duplication

Why not just expand variables?

We're introducing the $[[ matrix.VAR ]] syntax instead of expanding $VAR because of our long-term initiative to provide a more structured and predictable system than variables.

1. Improved UX: Matrix variables are not regular CI/CD variables, and treating them as such would create confusion about variable precedence and scope. The CI expression matrix context makes it explicit that you're referencing matrix values, not environment variables or another type of CI/CD variable
2. Future consistency: Following the job inputs work, users will be able to use ${{ matrix.VAR }} in CI keywords evaluated by Runner (like script), where matrix variables are currently available

Original Description

I am aware of 255311. Requested to be a separate issue by @dhershkovitch here.

Proposal

Expand variables that are added to the job via parallel:matrix when used in needs:parallel:matrix. This is very useful for pipelines with a matrix that creates N jobs in e.g. stage build and subsequently also needs N jobs in a later stag e e.g. test, where each test-job depends on a specific build-job from the matrix. Consider the following example:

.parallel-strat:
  parallel:
    matrix:
      - TARGET: ["linux", "windows"]

build-job:
  parallel: !reference [.parallel-strat, parallel]

# this would be great but it's not working, because `${TARGET}` is not expanded
test-job:
  parallel: !reference [.parallel-strat, parallel]
  needs:
    - job: build-job
      parallel:
        matrix:
          - TARGET: ${TARGET}

# status quo, create a dedicated job for every possible value of TARGET
test-job-linux:
  needs:
    - job: build-job
      parallel:
        matrix:
          - TARGET: "linux"

test-job-windows:
  needs:
    - job: build-job
      parallel:
        matrix:
          - TARGET: "windows"

As you can imagine, the current behavior does not scale well. The example above only considers two stages that need the N jobs and a single matrix variable. If there are multiple stages and/or multiple matrix variables, things get worse pretty f ast.

Currently, in a scenario where I have N jobs and M stages that all need the N jobs, I can effectively reduce the N*M total jobs to N*(M-1) jobs + 1 matrix job (in the first stage). With variable expansion this number can be brought d own to M matrix jobs.