[Bug] Failure in browser_ui/3_create/repository/add_list_delete_branches_spec.rb | Create Create, list, and delete branches via web lists branches correctly after CRUD operations
Summary
browser_ui/3_create/repository/add_list_delete_branches_spec.rb:62 E2E test started to fail on Praefect E2E QA jobs, example: https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/983222812
The test creates a project with 3 files. Then pushes a new file into a new second branch and pushes another new file into a new third branch. After that it tries to delete third branch and fails.
Branch fails to be deleted on Praefect with an error "rpc error: code = FailedPrecondition desc = Could not update refs/heads/third-branch. Please refresh and try again". If try to click on Delete button once more, it's still not being deleted.
Steps to reproduce
Option 1
Run gitlab-qa Test::Integration::Praefect EE --no-teardown -- qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb:62:
- it will setup Gitaly cluster using the latest GitLab nightly image, more information in https://docs.gitlab.com/ee/development/testing_guide/end_to_end/running_tests_that_require_special_setup.html#gitaly-cluster-tests
Option 2
If Gitaly cluster is already setup, the error could be replicated by doing the following:
- Create a project with master branch
- Push a file to a new branch 2
- Push a file to a new branch 3
- Try to remove the branch 3 or branch 2 via UI on
/branchespage
or running gitlab-qa against the environment gitlab-qa Test::Instance::Any EE http://gitlab.url -- --tag qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb:62
Actual Results
Branch is not deleted after clicking on "Delete" button.
Praefect logs:
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"error": "rpc error: code = FailedPrecondition desc = Could not update refs/heads/third-branch. Please refresh and try again.",
"grpc.code": "FailedPrecondition",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-web",
"grpc.meta.deadline_type": "regular",
"grpc.method": "UserDeleteBranch",
"grpc.request.deadline": "2021-01-25T14:10:09Z",
"grpc.request.fullMethod": "/gitaly.OperationService/UserDeleteBranch",
"grpc.service": "gitaly.OperationService",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 430.728,
"level": "warning",
"msg": "finished streaming call with code FailedPrecondition",
"peer.address": "172.19.0.7:56832",
"pid": 552,
"relative_path": "@hashed/6f/4b/6f4b6612125fb3a0daecd2799dfd6c9c299424fd920f9b308110a2c1fbd8f443.git",
"remote_ip": "172.19.0.8",
"sentry.skip": {},
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.553Z",
"username": "root",
"virtual_storage": "default"
}Full logs
Full logs on correlation ID
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"grpc.code": "OK",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-web",
"grpc.meta.deadline_type": "regular",
"grpc.method": "FindBranch",
"grpc.request.deadline": "2021-01-25T14:09:44Z",
"grpc.request.fullMethod": "/gitaly.RefService/FindBranch",
"grpc.service": "gitaly.RefService",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 16.437,
"level": "info",
"msg": "finished streaming call with code OK",
"peer.address": "172.19.0.7:56832",
"pid": 552,
"relative_path": "@hashed/6f/4b/6f4b6612125fb3a0daecd2799dfd6c9c299424fd920f9b308110a2c1fbd8f443.git",
"remote_ip": "172.19.0.8",
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.114Z",
"username": "root",
"virtual_storage": "default"
}
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"grpc.code": "OK",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 200.371,
"level": "info",
"msg": "finished unary call with code OK",
"peer.address": "172.19.0.4:34802",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.372Z"
}
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"grpc.code": "OK",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 0.137,
"level": "info",
"msg": "finished unary call with code OK",
"peer.address": "172.19.0.2:53980",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.372Z"
}
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"grpc.code": "OK",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 239.835,
"level": "info",
"msg": "finished unary call with code OK",
"peer.address": "172.19.0.3:36520",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.372Z"
}
{
"component": "transactions.Manager",
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"error": "transaction did not reach quorum: got 2/4 votes",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"hash": "66ad7b4d061e4cdc2ba5b7f1824a74d019ba33ff",
"level": "error",
"msg": "VoteTransaction: did not reach quorum",
"node": "gitaly2",
"peer.address": "172.19.0.3:36520",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.487Z",
"transaction_id": 15747431541673630000
}
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"grpc.code": "OK",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 3.598,
"level": "info",
"msg": "finished unary call with code OK",
"peer.address": "172.19.0.3:36520",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.487Z"
}
{
"component": "transactions.Manager",
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"error": "transaction did not reach quorum: got 2/4 votes",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"hash": "66ad7b4d061e4cdc2ba5b7f1824a74d019ba33ff",
"level": "error",
"msg": "VoteTransaction: did not reach quorum",
"node": "gitaly3",
"peer.address": "172.19.0.4:34802",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.488Z",
"transaction_id": 15747431541673630000
}
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"grpc.code": "OK",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 1.357,
"level": "info",
"msg": "finished unary call with code OK",
"peer.address": "172.19.0.4:34802",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.488Z"
}
{
"component": "transactions.Manager",
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"error": "transaction did not reach quorum: got 3/4 votes",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"hash": "87385cf42479d93c0e9f1e23ecce1a27d84caf62",
"level": "error",
"msg": "VoteTransaction: did not reach quorum",
"node": "gitaly1",
"peer.address": "172.19.0.2:53980",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.488Z",
"transaction_id": 15747431541673630000
}
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"grpc.code": "OK",
"grpc.meta.auth_version": "v2",
"grpc.meta.deadline_type": "unknown",
"grpc.method": "VoteTransaction",
"grpc.request.deadline": "2021-01-25T14:09:24Z",
"grpc.request.fullMethod": "/gitaly.RefTransaction/VoteTransaction",
"grpc.request.repo": null,
"grpc.service": "gitaly.RefTransaction",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 53.892,
"level": "info",
"msg": "finished unary call with code OK",
"peer.address": "172.19.0.2:53980",
"pid": 552,
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.489Z"
}
{
"committed": "0/3",
"component": "transactions.Manager",
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-web",
"grpc.meta.deadline_type": "regular",
"grpc.method": "UserDeleteBranch",
"grpc.request.deadline": "2021-01-25T14:10:09Z",
"grpc.request.fullMethod": "/gitaly.OperationService/UserDeleteBranch",
"grpc.service": "gitaly.OperationService",
"grpc.start_time": "2021-01-25T14:09:14Z",
"level": "info",
"msg": "transaction completed",
"peer.address": "172.19.0.7:56832",
"pid": 552,
"relative_path": "@hashed/6f/4b/6f4b6612125fb3a0daecd2799dfd6c9c299424fd920f9b308110a2c1fbd8f443.git",
"remote_ip": "172.19.0.8",
"sentry.skip": {},
"span.kind": "server",
"subtransactions": 2,
"system": "grpc",
"time": "2021-01-25T14:09:14.552Z",
"transaction_id": 15747431541673630000,
"username": "root",
"virtual_storage": "default"
}
{
"correlation_id": "01EWWY13PT8WT37KTJH39QJXZ2",
"error": "rpc error: code = FailedPrecondition desc = Could not update refs/heads/third-branch. Please refresh and try again.",
"grpc.code": "FailedPrecondition",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-web",
"grpc.meta.deadline_type": "regular",
"grpc.method": "UserDeleteBranch",
"grpc.request.deadline": "2021-01-25T14:10:09Z",
"grpc.request.fullMethod": "/gitaly.OperationService/UserDeleteBranch",
"grpc.service": "gitaly.OperationService",
"grpc.start_time": "2021-01-25T14:09:14Z",
"grpc.time_ms": 430.728,
"level": "warning",
"msg": "finished streaming call with code FailedPrecondition",
"peer.address": "172.19.0.7:56832",
"pid": 552,
"relative_path": "@hashed/6f/4b/6f4b6612125fb3a0daecd2799dfd6c9c299424fd920f9b308110a2c1fbd8f443.git",
"remote_ip": "172.19.0.8",
"sentry.skip": {},
"span.kind": "server",
"system": "grpc",
"time": "2021-01-25T14:09:14.553Z",
"username": "root",
"virtual_storage": "default"
}Old description
Full description
Create Create, list, and delete branches via web lists branches correctly after CRUD operations
File path
./qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb
Stack trace
Failures:
1) Create Create, list, and delete branches via web lists branches correctly after CRUD operations
Failure/Error: expect(branches_page).to have_no_branch(third_branch)
QA::Support::Repeater::WaitExceededError:
Wait condition not met after 60 seconds
# ./qa/support/repeater.rb:44:in `repeat_until'First happened in https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/983222812.
Edited by Nailia Iskhakova