Line numbering in diff, perhaps related to Show unchanged lines, seems broken

Summary

When using show changed lines, the line numbering restarts midway through the file. After doing this, clicking on the line numbers behaves erratically as well.

Steps to reproduce / Screenshots

From gitlab-runner!1834 (diffs)

Edit: Actually, this might be deeper, as you can see the issue sometimes even before clicking: