Language:
switch to room list switch to menu My folders
Go to page: First ... 22 23 24 25 [26]
[#] Sun Apr 13 2025 17:54:25 UTC from p.agsten

Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

[Reply] [ReplyQuoted] [Headers] [Print]

To add to this I further analysed the syslog and found that within 8 hours after the first crash there were three restarts after signal 11 (SEGV). After the third restart the fourth server crash was with signal 6 (ABRT) after DB runs out of lock entries:

Apr 13 18:23:03 [hostname] citserver[711154]: citserver[711154]: bdb: BDB2055 Lock table is out of available lock entries

Apr 13 18:23:03 [hostname] citserver[711154]: citserver[711154]: bdb: bdb_store(09): error 12: Cannot allocate memory

Apr 13 18:23:03 [hostname] citserver[711154]: bdb: BDB2055 Lock table is out of available lock entries

Apr 13 18:23:03 [hostname] citserver[711154]: bdb: bdb_store(09): error 12: Cannot allocate memory

Apr 13 18:23:03 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=6/ABRT 

From there the server restarted and survived for about another 3 minutes accepting connections. Then restarted 231 times generating around 6.500 log files within 19 minutes.

Looks like one of the SEGV somehow corrupt or block the DB. The lock table issue is just the fallout. Is there a log where we can see what causes signal 11 (SEGV)?

So Apr 13 2025 17:20:30 UTC von p.agsten Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055

 

Fr Apr 11 2025 13:22:11 UTC von IGnatius T Foobar Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055
these in the folder. Can this be related to database corruption or I/O
delays?

Yes, once the system has been running for a while it will clean up all log files that have been fully committed to the database. Usually this is rock solid so I'm wondering if you had a resource problem of some sort (disk full, etc)

Disk is not a problem, less than 50%. Did restored the DB again from backup. Server was running for a couple of hours after. Then this morning again sudden stopand down. The server was restarted automatically but the restarts then happen every couple of minutes. Restart count was eventually at 235. Up to restart 230 or so there was only one log.XXX file in the data folder, but then accumulated rapidly. Below is the last log files that caused the first automatic restart with error code 11. Interesting part is the message not found just before the crash, can this cause the issue? Or is this some sort of attack? Does any of this make sense?

Apr 13 10:55:14 [hostname] citserver[701535]: citserver[701535]: context: session 7508 (SMTP-MTA) ended.

Apr 13 10:55:14 [hostname] citserver[701535]: context: session 7508 (SMTP-MTA) ended.

Apr 13 10:55:16 [hostname] citserver[701535]: citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:21 [hostname] citserver[701535]: citserver[701535]: context: session (SMTP-MTA) started from 80.94.95.228 (80.94.95.228) uid=-1

Apr 13 10:55:21 [hostname] citserver[701535]: context: session (SMTP-MTA) started from 80.94.95.228 (80.94.95.228) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: context: session (SMTP-MTA) started from 167.94.146.61 (167.94.146.61) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: context: session (SMTP-MTA) started from 167.94.146.61 (167.94.146.61) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: crypto: TLS using TLS_CHACHA20_POLY1305_SHA256 on TLSv1.3 (256 of 256 bits)

Apr 13 10:55:22 [hostname] citserver[701535]: crypto: TLS using TLS_CHACHA20_POLY1305_SHA256 on TLSv1.3 (256 of 256 bits)

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: crypto: ending TLS on this session

Apr 13 10:55:22 [hostname] citserver[701535]: crypto: ending TLS on this session

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: SMTP: client disconnected: ending session.

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: context: session 7510 (SMTP-MTA) ended.

Apr 13 10:55:22 [hostname] citserver[701535]: SMTP: client disconnected: ending session.

Apr 13 10:55:22 [hostname] citserver[701535]: context: session 7510 (SMTP-MTA) ended.

Apr 13 10:55:26 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=11/SEGV

Apr 13 10:55:26 [hostname] systemd[1]: citadel.service: Failed with result 'signal'.

Apr 13 10:55:27 [hostname] systemd[1]: citadel.service: Scheduled restart job, restart counter is at 1.

Apr 13 10:55:27 [hostname] systemd[1]: Stopped Citadel Server.

Apr 13 10:55:27 [hostname] systemd[1]: Started Citadel Server.

Apr 13 10:55:27 [hostname] citserver:

Apr 13 10:55:27 [hostname] citserver:

Apr 13 10:55:27 [hostname] citserver:*** Citadel server engine ***

Apr 13 10:55:27 [hostname] citserver:Version 998 (build 24041) ***



 



[#] Sun Apr 13 2025 21:35:15 UTC from luisgo

Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

[Reply] [ReplyQuoted] [Headers] [Print]

I had a similar problem this late afternoon. I had to recover the data files from a backup due a full disk. I reported a similar problem some time ago.

 

Sun Apr 13 2025 17:54:25 UTC from p.agsten Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

To add to this I further analysed the syslog and found that within 8 hours after the first crash there were three restarts after signal 11 (SEGV). After the third restart the fourth server crash was with signal 6 (ABRT) after DB runs out of lock entries:

Apr 13 18:23:03 [hostname] citserver[711154]: citserver[711154]: bdb: BDB2055 Lock table is out of available lock entries

Apr 13 18:23:03 [hostname] citserver[711154]: citserver[711154]: bdb: bdb_store(09): error 12: Cannot allocate memory

Apr 13 18:23:03 [hostname] citserver[711154]: bdb: BDB2055 Lock table is out of available lock entries

Apr 13 18:23:03 [hostname] citserver[711154]: bdb: bdb_store(09): error 12: Cannot allocate memory

Apr 13 18:23:03 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=6/ABRT 

From there the server restarted and survived for about another 3 minutes accepting connections. Then restarted 231 times generating around 6.500 log files within 19 minutes.

Looks like one of the SEGV somehow corrupt or block the DB. The lock table issue is just the fallout. Is there a log where we can see what causes signal 11 (SEGV)?

So Apr 13 2025 17:20:30 UTC von p.agsten Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055

 

Fr Apr 11 2025 13:22:11 UTC von IGnatius T Foobar Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055
these in the folder. Can this be related to database corruption or I/O
delays?

Yes, once the system has been running for a while it will clean up all log files that have been fully committed to the database. Usually this is rock solid so I'm wondering if you had a resource problem of some sort (disk full, etc)

Disk is not a problem, less than 50%. Did restored the DB again from backup. Server was running for a couple of hours after. Then this morning again sudden stopand down. The server was restarted automatically but the restarts then happen every couple of minutes. Restart count was eventually at 235. Up to restart 230 or so there was only one log.XXX file in the data folder, but then accumulated rapidly. Below is the last log files that caused the first automatic restart with error code 11. Interesting part is the message not found just before the crash, can this cause the issue? Or is this some sort of attack? Does any of this make sense?

Apr 13 10:55:14 [hostname] citserver[701535]: citserver[701535]: context: session 7508 (SMTP-MTA) ended.

Apr 13 10:55:14 [hostname] citserver[701535]: context: session 7508 (SMTP-MTA) ended.

Apr 13 10:55:16 [hostname] citserver[701535]: citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:16 [hostname] citserver[701535]: msgbase: message #93631 was not found

Apr 13 10:55:21 [hostname] citserver[701535]: citserver[701535]: context: session (SMTP-MTA) started from 80.94.95.228 (80.94.95.228) uid=-1

Apr 13 10:55:21 [hostname] citserver[701535]: context: session (SMTP-MTA) started from 80.94.95.228 (80.94.95.228) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: context: session (SMTP-MTA) started from 167.94.146.61 (167.94.146.61) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: context: session (SMTP-MTA) started from 167.94.146.61 (167.94.146.61) uid=-1

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: crypto: TLS using TLS_CHACHA20_POLY1305_SHA256 on TLSv1.3 (256 of 256 bits)

Apr 13 10:55:22 [hostname] citserver[701535]: crypto: TLS using TLS_CHACHA20_POLY1305_SHA256 on TLSv1.3 (256 of 256 bits)

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: crypto: ending TLS on this session

Apr 13 10:55:22 [hostname] citserver[701535]: crypto: ending TLS on this session

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: SMTP: client disconnected: ending session.

Apr 13 10:55:22 [hostname] citserver[701535]: citserver[701535]: context: session 7510 (SMTP-MTA) ended.

Apr 13 10:55:22 [hostname] citserver[701535]: SMTP: client disconnected: ending session.

Apr 13 10:55:22 [hostname] citserver[701535]: context: session 7510 (SMTP-MTA) ended.

Apr 13 10:55:26 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=11/SEGV

Apr 13 10:55:26 [hostname] systemd[1]: citadel.service: Failed with result 'signal'.

Apr 13 10:55:27 [hostname] systemd[1]: citadel.service: Scheduled restart job, restart counter is at 1.

Apr 13 10:55:27 [hostname] systemd[1]: Stopped Citadel Server.

Apr 13 10:55:27 [hostname] systemd[1]: Started Citadel Server.

Apr 13 10:55:27 [hostname] citserver:

Apr 13 10:55:27 [hostname] citserver:

Apr 13 10:55:27 [hostname] citserver:*** Citadel server engine ***

Apr 13 10:55:27 [hostname] citserver:Version 998 (build 24041) ***



 



 



[#] Mon Apr 14 2025 01:22:09 UTC from IGnatius T Foobar

Subject: Re: STARTTLS isn't supported

[Reply] [ReplyQuoted] [Headers] [Print]

Well, I guess then we discovered a problem with the manual at
https://citadel.org/sslcertificates.html

Thanks for pointing that out. It has been updated.

I am starting to wonder whether we should just put a copy of certbot inside the container and let it run.

[#] Mon Apr 14 2025 01:23:24 UTC from IGnatius T Foobar

Subject: Re: STARTTLS isn't supported

[Reply] [ReplyQuoted] [Headers] [Print]

So any script you put in to the "post" folder will be executed after
certbot renews a cert.
That would be the perfect place to add a copy script.

Actually the "deploy" directory is the correct place to put it.

And ... duh ... it looks like I did exactly that when I containerized. I was obviously not paying attention to the documentation at the time.

[#] Mon Apr 14 2025 01:31:11 UTC from IGnatius T Foobar

Subject: Re: _BASEROOM_

[Reply] [ReplyQuoted] [Headers] [Print]

\ --name=citadel citadeldotorg/citadel --http-port=8080
--https-port=8443 -g "/dotgoto?room=Welcome"

It looks correct ... the only thing I can think of is maybe the question mark is getting escaped by the shell since you're using double quotes instead of single quotes?

For reference, here is my launch command:

docker run \
-d \
--restart=unless-stopped \
--network host \
--volume=/usr/local/citadel:/citadel-data \
--volume=/usr/local/webcit/.well-known:/usr/local/webcit/.well-known \
--name=citadel \
citadeldotorg/citadel -g '/dotgoto?room=Welcome to UNCENSORED!'

I've got a question mark, spaces, and an exclamation point in there.

[#] Mon Apr 14 2025 01:39:12 UTC from IGnatius T Foobar

Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

[Reply] [ReplyQuoted] [Headers] [Print]

I had a similar problem this late afternoon. I had to recover the data
files from a backup due a full disk. I reported a similar problem some
time ago.

You know what, I'm going to add something to the server to make it refuse to accept new messages when the free disk space is under 100 MB. It sure seems better than seeing people have problems *after* they run out of disk.

[#] Mon Apr 14 2025 06:57:22 UTC from luisgo

Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

[Reply] [ReplyQuoted] [Headers] [Print]

 

Mon Apr 14 2025 01:39:12 UTC from IGnatius T Foobar Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055
I had a similar problem this late afternoon. I had to recover the data
files from a backup due a full disk. I reported a similar problem some
time ago.

You know what, I'm going to add something to the server to make it refuse to accept new messages when the free disk space is under 100 MB. It sure seems better than seeing people have problems *after* they run out of disk.

Dear All,

What I think that will help a lot from my experience dealing with this is doing the following:

It is doing Auto-Purging from 15 minutes in 15 minutes instead only one time a day.

Hour to run database auto-purge

Thanks,

Luís Gonçalves



[#] Mon Apr 14 2025 06:59:51 UTC from luisgo

Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

[Reply] [ReplyQuoted] [Headers] [Print]

 

Mon Apr 14 2025 06:57:22 UTC from luisgo Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

 

Mon Apr 14 2025 01:39:12 UTC from IGnatius T Foobar Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055
I had a similar problem this late afternoon. I had to recover the data
files from a backup due a full disk. I reported a similar problem some
time ago.

You know what, I'm going to add something to the server to make it refuse to accept new messages when the free disk space is under 100 MB. It sure seems better than seeing people have problems *after* they run out of disk.

Dear All,

What I think that will help a lot from my experience dealing with this is doing the following:

It is doing Auto-Purging from 15 minutes in 15 minutes instead only one time a day.

Hour to run database auto-purge

Thanks,

Luís Gonçalves



Doing that without accepting transactions (stopped server).



[#] Mon Apr 14 2025 07:14:02 UTC from luisgo

Subject: Disable ManageSieve.

[Reply] [ReplyQuoted] [Headers] [Print]

Dear All,

I tried to disable without success. Is it normal?

ManageSieve Port (-1 to disable)

Thanks,

Luís Gonçalves



[#] Mon Apr 14 2025 17:35:19 UTC from p.agsten

Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

[Reply] [ReplyQuoted] [Headers] [Print]

 

Mo Apr 14 2025 01:39:12 UTC von IGnatius T Foobar Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055
I had a similar problem this late afternoon. I had to recover the data
files from a backup due a full disk. I reported a similar problem some
time ago.

You know what, I'm going to add something to the server to make it refuse to accept new messages when the free disk space is under 100 MB. It sure seems better than seeing people have problems *after* they run out of disk.

That could indeed be helpful for scenarios where disk space is an issue. However, just to re-confirm in my particular case disk is no issue at all. I have around 470GiB free on the file system where the database resides. So question is still how I can understand what causes the SIGSEGV crash on the citadel server. 



[#] Mon Apr 14 2025 18:24:05 UTC from IGnatius T Foobar

Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

[Reply] [ReplyQuoted] [Headers] [Print]

What I think that will help a lot from my experience dealing with this
is doing the following:

It is doing Auto-Purging from 15 minutes in 15 minutes instead only one
time a day.

If your database is growing so fast that you have to purge it every 15 minutes, you have a runaway writer, expire policy is not the answer.

[#] Mon Apr 14 2025 18:24:35 UTC from IGnatius T Foobar

Subject: Re: Disable ManageSieve.

[Reply] [ReplyQuoted] [Headers] [Print]

I tried to disable without success. Is it normal?

ManageSieve Port (-1 to disable)

The managesieve service was discontinued a long time ago. Setting or unsetting its port number has no effect.

[#] Mon Apr 14 2025 18:25:54 UTC from IGnatius T Foobar

Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055

[Reply] [ReplyQuoted] [Headers] [Print]

However, just to re-confirm in my particular case disk is no issue at
all. I have around 470GiB free on the file system where the database
resides. So question is still how I can understand what causes the
SIGSEGV crash on the citadel server. 

If you have a core dump you can do a stack trace against it. If not, you can run citserver in the debugger. Like this:

systemctl stop citserver
cd /usr/local/citadel
gdb ./citserver
gdb> run -x9
[ wait for it to crash ]
gdb> thread apply all bt

Then post the output here.

Go to page: First ... 22 23 24 25 [26]