To add to this I further analysed the syslog and found that within 8 hours after the first crash there were three restarts after signal 11 (SEGV). After the third restart the fourth server crash was with signal 6 (ABRT) after DB runs out of lock entries:

Apr 13 18:23:03 [hostname] citserver[711154]: citserver[711154]: bdb: BDB2055 Lock table is out of available lock entries

Apr 13 18:23:03 [hostname] citserver[711154]: citserver[711154]: bdb: bdb_store(09): error 12: Cannot allocate memory

Apr 13 18:23:03 [hostname] citserver[711154]: bdb: BDB2055 Lock table is out of available lock entries

Apr 13 18:23:03 [hostname] citserver[711154]: bdb: bdb_store(09): error 12: Cannot allocate memory

Apr 13 18:23:03 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=6/ABRT 

From there the server restarted and survived for about another 3 minutes accepting connections. Then restarted 231 times generating around 6.500 log files within 19 minutes.

Looks like one of the SEGV somehow corrupt or block the DB. The lock table issue is just the fallout. Is there a log where we can see what causes signal 11 (SEGV)?

So Apr 13 2025 17:20:30 UTC von p.agsten Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055

 

Fr Apr 11 2025 13:22:11 UTC von IGnatius T Foobar Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055

these in the folder. Can this be related to database corruption or I/O
delays?

Yes, once the system has been running for a while it will clean up all log files that have been fully committed to the database. Usually this is rock solid so I'm wondering if you had a resource problem of some sort (disk full, etc)

Disk is not a problem, less than 50%. Did restored the DB again from backup. Server was running for a couple of hours after. Then this morning again sudden stopand down. The server was restarted automatically but the restarts then happen every couple of minutes. Restart count was eventually at 235. Up to restart 230 or so there was only one log.XXX file in the data folder, but then accumulated rapidly. Below is the last log files that caused the first automatic restart with error code 11. Interesting part is the message not found just before the crash, can this cause the issue? Or is this some sort of attack? Does any of this make sense?

Apr 13 10:55:14 [hostname] citserver[701535]: citserver[701535]: context: session 7508 (SMTP-MTA) ended.

Apr 13 10:55:14 [hostname] citserver[701535]: context: session 7508 (SMTP-MTA) ended.

Apr 13 10:55:16 [hostname] citserver[701535]: citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:21 [hostname] citserver[701535]: citserver[701535]: context: session (SMTP-MTA) started from 80.94.95.228 (80.94.95.228) uid=-1

Apr 13 10:55:21 [hostname] citserver[701535]: context: session (SMTP-MTA) started from 80.94.95.228 (80.94.95.228) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: context: session (SMTP-MTA) started from 167.94.146.61 (167.94.146.61) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: context: session (SMTP-MTA) started from 167.94.146.61 (167.94.146.61) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: crypto: TLS using TLS_CHACHA20_POLY1305_SHA256 on TLSv1.3 (256 of 256 bits)

Apr 13 10:55:22 [hostname] citserver[701535]: crypto: TLS using TLS_CHACHA20_POLY1305_SHA256 on TLSv1.3 (256 of 256 bits)

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: crypto: ending TLS on this session

Apr 13 10:55:22 [hostname] citserver[701535]: crypto: ending TLS on this session

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: SMTP: client disconnected: ending session.

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: context: session 7510 (SMTP-MTA) ended.

Apr 13 10:55:22 [hostname] citserver[701535]: SMTP: client disconnected: ending session.

Apr 13 10:55:22 [hostname] citserver[701535]: context: session 7510 (SMTP-MTA) ended.

Apr 13 10:55:26 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=11/SEGV

Apr 13 10:55:26 [hostname] systemd[1]: citadel.service: Failed with result 'signal'.

Apr 13 10:55:27 [hostname] systemd[1]: citadel.service: Scheduled restart job, restart counter is at 1.

Apr 13 10:55:27 [hostname] systemd[1]: Stopped Citadel Server.

Apr 13 10:55:27 [hostname] systemd[1]: Started Citadel Server.

Apr 13 10:55:27 [hostname] citserver:

Apr 13 10:55:27 [hostname] citserver:

Apr 13 10:55:27 [hostname] citserver:*** Citadel server engine ***

Apr 13 10:55:27 [hostname] citserver:Version 998 (build 24041) ***

I had a similar problem this late afternoon. I had to recover the data files from a backup due a full disk. I reported a similar problem some time ago.

Sun Apr 13 2025 17:54:25 UTC from p.agsten Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

To add to this I further analysed the syslog and found that within 8 hours after the first crash there were three restarts after signal 11 (SEGV). After the third restart the fourth server crash was with signal 6 (ABRT) after DB runs out of lock entries:

Apr 13 18:23:03 [hostname] citserver[711154]: citserver[711154]: bdb: BDB2055 Lock table is out of available lock entries

Apr 13 18:23:03 [hostname] citserver[711154]: citserver[711154]: bdb: bdb_store(09): error 12: Cannot allocate memory

Apr 13 18:23:03 [hostname] citserver[711154]: bdb: BDB2055 Lock table is out of available lock entries

Apr 13 18:23:03 [hostname] citserver[711154]: bdb: bdb_store(09): error 12: Cannot allocate memory

Apr 13 18:23:03 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=6/ABRT 

From there the server restarted and survived for about another 3 minutes accepting connections. Then restarted 231 times generating around 6.500 log files within 19 minutes.

Looks like one of the SEGV somehow corrupt or block the DB. The lock table issue is just the fallout. Is there a log where we can see what causes signal 11 (SEGV)?

So Apr 13 2025 17:20:30 UTC von p.agsten Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055

 

Fr Apr 11 2025 13:22:11 UTC von IGnatius T Foobar Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055

these in the folder. Can this be related to database corruption or I/O
delays?

Yes, once the system has been running for a while it will clean up all log files that have been fully committed to the database. Usually this is rock solid so I'm wondering if you had a resource problem of some sort (disk full, etc)

Disk is not a problem, less than 50%. Did restored the DB again from backup. Server was running for a couple of hours after. Then this morning again sudden stopand down. The server was restarted automatically but the restarts then happen every couple of minutes. Restart count was eventually at 235. Up to restart 230 or so there was only one log.XXX file in the data folder, but then accumulated rapidly. Below is the last log files that caused the first automatic restart with error code 11. Interesting part is the message not found just before the crash, can this cause the issue? Or is this some sort of attack? Does any of this make sense?

Apr 13 10:55:14 [hostname] citserver[701535]: citserver[701535]: context: session 7508 (SMTP-MTA) ended.

Apr 13 10:55:14 [hostname] citserver[701535]: context: session 7508 (SMTP-MTA) ended.

Apr 13 10:55:16 [hostname] citserver[701535]: citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:21 [hostname] citserver[701535]: citserver[701535]: context: session (SMTP-MTA) started from 80.94.95.228 (80.94.95.228) uid=-1

Apr 13 10:55:21 [hostname] citserver[701535]: context: session (SMTP-MTA) started from 80.94.95.228 (80.94.95.228) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: context: session (SMTP-MTA) started from 167.94.146.61 (167.94.146.61) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: context: session (SMTP-MTA) started from 167.94.146.61 (167.94.146.61) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: crypto: TLS using TLS_CHACHA20_POLY1305_SHA256 on TLSv1.3 (256 of 256 bits)

Apr 13 10:55:22 [hostname] citserver[701535]: crypto: TLS using TLS_CHACHA20_POLY1305_SHA256 on TLSv1.3 (256 of 256 bits)

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: crypto: ending TLS on this session

Apr 13 10:55:22 [hostname] citserver[701535]: crypto: ending TLS on this session

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: SMTP: client disconnected: ending session.

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: context: session 7510 (SMTP-MTA) ended.

Apr 13 10:55:22 [hostname] citserver[701535]: SMTP: client disconnected: ending session.

Apr 13 10:55:22 [hostname] citserver[701535]: context: session 7510 (SMTP-MTA) ended.

Apr 13 10:55:26 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=11/SEGV

Apr 13 10:55:26 [hostname] systemd[1]: citadel.service: Failed with result 'signal'.

Apr 13 10:55:27 [hostname] systemd[1]: citadel.service: Scheduled restart job, restart counter is at 1.

Apr 13 10:55:27 [hostname] systemd[1]: Stopped Citadel Server.

Apr 13 10:55:27 [hostname] systemd[1]: Started Citadel Server.

Apr 13 10:55:27 [hostname] citserver:

Apr 13 10:55:27 [hostname] citserver:

Apr 13 10:55:27 [hostname] citserver:*** Citadel server engine ***

Apr 13 10:55:27 [hostname] citserver:Version 998 (build 24041) ***

 

Mon Apr 14 2025 01:39:12 UTC from IGnatius T Foobar Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

I had a similar problem this late afternoon. I had to recover the data
files from a backup due a full disk. I reported a similar problem some
time ago.

You know what, I'm going to add something to the server to make it refuse to accept new messages when the free disk space is under 100 MB. It sure seems better than seeing people have problems *after* they run out of disk.

Dear All,

What I think that will help a lot from my experience dealing with this is doing the following:

It is doing Auto-Purging from 15 minutes in 15 minutes instead only one time a day.

Thanks,

Luís Gonçalves

Mon Apr 14 2025 06:57:22 UTC from luisgo Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

 

Mon Apr 14 2025 01:39:12 UTC from IGnatius T Foobar Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

I had a similar problem this late afternoon. I had to recover the data
files from a backup due a full disk. I reported a similar problem some
time ago.

You know what, I'm going to add something to the server to make it refuse to accept new messages when the free disk space is under 100 MB. It sure seems better than seeing people have problems *after* they run out of disk.

Dear All,

What I think that will help a lot from my experience dealing with this is doing the following:

It is doing Auto-Purging from 15 minutes in 15 minutes instead only one time a day.

Hour to run database auto-purge

Thanks,

Luís Gonçalves

Doing that without accepting transactions (stopped server).

Dear All,

I tried to disable without success. Is it normal?

ManageSieve Port (-1 to disable)

Thanks,

Luís Gonçalves

Mo Apr 14 2025 01:39:12 UTC von IGnatius T Foobar Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055

I had a similar problem this late afternoon. I had to recover the data
files from a backup due a full disk. I reported a similar problem some
time ago.

You know what, I'm going to add something to the server to make it refuse to accept new messages when the free disk space is under 100 MB. It sure seems better than seeing people have problems *after* they run out of disk.

That could indeed be helpful for scenarios where disk space is an issue. However, just to re-confirm in my particular case disk is no issue at all. I have around 470GiB free on the file system where the database resides. So question is still how I can understand what causes the SIGSEGV crash on the citadel server.