GItlab restore from Gitlab Tool fails to restore repositories due to connection reset with Gitaly

EDIT: After some further digging, the amount of traffic pushed during the restore from the Toolbox caused the Toolbox proxy sidecar to receive OOM and reset the connection. This was resolved by setting the proxy memory limits to 512Mi instead of the default 256Mi. Feel free to close if this is not an issue that bigbang can fix and rather dependent on customer configuration.

When restoring from backup, Gitlab-toolbox recieves connection reset with Gitaly, resulting in failure to restore repositories. Was able to confirm configuration issue with Istio by removing injection from the namespace and running the backup to success. Still working through a solution.

Command ran on Gitlab Toolbox container: backup-utility --s3tool awscli --restore -t <timestamp of tar in s3>

Backup Errors:

{"command":"restore","error":"manager: remove repository: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.2.52.195:8075: connect: connection refused\"","gl_project_path":"safco/ansible-roles/pre-teardown","level":"error","msg":"restore failed","relative_path":"@hashed/16/dc/16dc368a89b428b2485484313ba67a3912ca03f2b2b42429174a4f8b3dc84e44.git","storage_name":"default","time":"2022-09-24T16:25:03.619Z"}
{"command":"restore","error":"manager: remove repository: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.2.52.195:8075: connect: connection refused\"","gl_project_path":"safco/ansible-roles/pre-teardown","level":"error","msg":"restore failed","relative_path":"@hashed/16/dc/16dc368a89b428b2485484313ba67a3912ca03f2b2b42429174a4f8b3dc84e44.design.git","storage_name":"default","time":"2022-09-24T16:25:03.619Z"}

Gitaly sidecar error:

[2022-09-24T16:31:49.753Z] "POST /gitaly.RepositoryService/FetchBundle HTTP/2" 200 - via_upstream - "-" 106385681 5 9923 7828 "-" "grpc-go/1.48.0" "8f159403-ffe1-90e9-a313-13d7074dbea7" "gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075" "10.2.52.195:8075" inbound|8075|| 127.0.0.6:56295 10.2.52.195:8075 10.2.36.46:56122 outbound_.8075_._.gitlab-gitaly.gitlab.svc.cluster.local default traceID=75ce52da12a3c00537312589ccffe59f
[2022-09-24T16:31:54.849Z] "POST /gitaly.RepositoryService/FetchBundle HTTP/2" 0 DC downstream_remote_disconnect - "-" 51033906 0 5357 - "-" "grpc-go/1.48.0" "3e641f2c-85b5-981f-a107-69c4b4fb99ea" "gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075" "10.2.52.195:8075" inbound|8075|| 127.0.0.6:56295 10.2.52.195:8075 10.2.36.46:56122 outbound_.8075_._.gitlab-gitaly.gitlab.svc.cluster.local default traceID=5dcd0a68a215ac30d43cb9029ea75

Graph from Kiali while running the restore