Gitlab Mailroom fails to read incoming email
Bug
Gitlab Mailroom pod is in CrashLoopBackOff because it is unable to pull and arbitrate incoming emails. We are using microsoft_graph for the inbox.
Description
This happens on startup of the mailroom pod. The pod immediately crashes. The same mailroom configuration was working on older versions of GitLab and Big Bang (v1.48.0)
Appears to be an outdated redis-namespace dependency that is incompatible with Ruby 3 in the UBI mailroom image (see previous related ticket: #201 (closed)).
This is the pod logs on startup:
Begin parsing .erb templates from /etc
[2023-11-30 20:19:41] INFO WEBrick 1.7.0
[2023-11-30 20:19:41] INFO ruby 3.0.6 (2023-03-30) [x86_64-linux]
{"severity":"INFO","time":"2023-11-30T20:19:41.175+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Setting up watcher"}
[2023-11-30 20:19:41] INFO WEBrick::HTTPServer#start: pid=1 port=8080
{"severity":"INFO","time":"2023-11-30T20:19:41.269+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Retrieving OAuth2 token..."}
{"severity":"INFO","time":"2023-11-30T20:19:41.579+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Processing started"}
{"severity":"INFO","time":"2023-11-30T20:19:42.128+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"uid":"<redacted>","action":"asking arbiter to deliver","arbitrator":"MailRoom::Arbitration::Redis"}
#<Thread:0x00007fba09c667c0 /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox_watcher.rb:35 run> terminated with exception (report_on_exception is true):
/usr/lib64/ruby/gems/3.0.0/gems/redis-5.0.7/lib/redis/commands/strings.rb:83:in `set': wrong number of arguments (given 3, expected 2) (ArgumentError)
from /usr/lib64/ruby/gems/3.0.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:442:in `call_with_namespace'
from /usr/lib64/ruby/gems/3.0.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:328:in `method_missing'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/arbitration/redis.rb:34:in `deliver?'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox.rb:108:in `deliver?'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `block in unread_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `select'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `unread_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:153:in `block (2 levels) in new_message_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:152:in `loop'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:152:in `block in new_message_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `each'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `each_with_object'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `messages_for_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:144:in `new_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:122:in `process_mailbox'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:26:in `wait'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox_watcher.rb:37:in `block in run'
[2023-11-30 20:19:42] INFO going to shutdown ...
[2023-11-30 20:19:42] INFO WEBrick::HTTPServer#start done.
{"severity":"INFO","time":"2023-11-30T20:19:42.151+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Quitting connection..."}
{"severity":"INFO","time":"2023-11-30T20:19:42.151+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Terminating watching thread..."}
/usr/lib64/ruby/gems/3.0.0/gems/redis-5.0.7/lib/redis/commands/strings.rb:83:in `set': wrong number of arguments (given 3, expected 2) (ArgumentError)
from /usr/lib64/ruby/gems/3.0.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:442:in `call_with_namespace'
from /usr/lib64/ruby/gems/3.0.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:328:in `method_missing'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/arbitration/redis.rb:34:in `deliver?'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox.rb:108:in `deliver?'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `block in unread_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `select'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `unread_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:153:in `block (2 levels) in new_message_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:152:in `loop'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:152:in `block in new_message_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `each'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `each_with_object'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `messages_for_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:144:in `new_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:122:in `process_mailbox'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:26:in `wait'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox_watcher.rb:37:in `block in run'And this is the mailbox configuration. Again, this was previously working before the Ruby 3 upgrade in GitLab.
:mailboxes:
-
:email: <redacted>
:name: inbox
:delete_after_delivery: false
:expunge_deleted: false
:logger:
:log_path: "/dev/stdout"
:inbox_method: microsoft_graph
:inbox_options:
:tenant_id: "<redacted>"
:client_id: "<redacted>"
:client_secret: <%= File.read("/etc/gitlab/mailroom/client_id_incoming_email").strip.to_json %>
:poll_interval: 60
:azure_ad_endpoint: https://login.microsoftonline.us
:graph_endpoint: https://graph.microsoft.us
:delivery_method: postback
:delivery_options:
:delivery_url: 'http://gitlab-webservice-default.gitlab.svc:8181/api/v4/internal/mail_room/incoming_email'
:content_type: "text/plain"
:jwt_auth_header: "Gitlab-Mailroom-Api-Request"
:jwt_issuer: "gitlab-mailroom"
:jwt_algorithm: "HS256"
:jwt_secret_path: "/etc/gitlab/mailroom/incoming_email_webhook_secret"
:arbitration_method: redis
:arbitration_options:
:redis_url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379
:namespace: mail_room:gitlabBigBang Version
What version of BigBang were you running?
2.15.1