Recommended settings for AWS layer 4 load balancer(NLB) won't work

Summary

There seems to be some issues with the example config here: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/v4.10.0/examples/aws/elb-layer4-loadbalancer.yaml

1. This config creates a classic load balancer, not NLB as stated.
2. If we enable proxy protocol using the annotation: service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" , it applies to all the elb backends, not just SSH, causing the HTTP traffic (TCP 80, 443) to fail with 400 bad request errors. Doc: AWS Loadbalancers

Workaround:

To serve both HTTP(s) and SSH through the classic ELB,

• Remove all annotations from nginx-ingress.controller.service. It creates all listeners with proxy protocol disabled.
• Create a proxy protocol policy just for SSH service at AWS following the instructions: Configure proxy protocol support for your Classic Load Balancer. Example commands:

$ aws elb create-load-balancer-policy --load-balancer-name <lb-name> --policy-name proxy-protocol --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true

$ aws elb set-load-balancer-policies-for-backend-server --load-balancer-name <lb-name> --instance-port <instance-port> --policy-names proxy-protocol

Please note that the <instance-port> above is not 22, it's the Nodeport(Instance port) mapped to the SSH service on the Loadbalancer. You'll get this from the Loadbalancer listener page.

Potential permanent fixes:

• Find the reason why enabling proxy protocol(service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" ) on port 80 and 443 breaks web services (404 bad request) and fix it from the application side.
• Wait until aws-load-balancer-proxy-protocol supports customisation of backend services. Then turn on proxy protocol for just SSH via annotations.
• Use separate service/ingress for SSH service and enable proxy protocol just for that through service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" .

Steps to reproduce

Install GitLab using Helm, using the following config. Only custom values are given.

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

global:
  shell:
    # Configure GitLab Shell's NGINX TCP to expect ProxyProtcol headers (and strip them)
    tcp:
      proxyProtocol: true
nginx-ingress:
  controller:
    service:
      annotations:
        # Configure NLB, Layer 4 with 'tcp'
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
        # Configure NLB to use ProxyProtocol
        service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" 
      config:
        proxy-real-ip-cidr: 10.2.0.0/16  # The VPC CIDR, to be trusted
        use-proxy-protocol: "true"       # Tell NGINX to Expect ProxyProtocol headers

I have also tried flipping TLS off and on for global.ingress with various settings such as cert-manager, own self-signed certificates etc. But, nothing changed the error behaviour. It appeared as if proxy protocol was creating issues with web services.

Current behavior

If we use the recommended settings for Layer 4 load balancing, web services won't work.

Expected behavior

Web services + SSH should work when Layer 4 settings are used.

Versions

• Chart: 4.10.3
• Platform:
  • Cloud: EKS
• Kubernetes: (kubectl version)
  • Client: v1.16.6-beta.0
  • Server: v1.18.9-eks-d1db3c
• Helm: (helm version)
  • Client: v3.1.2
  • Server:

Relevant logs

Was getting 400 bad request error on UI. Ingress logs showed 400 return code for PROXY TCP4 requests.