Please wait...
0 files
Subject: Citadel Server sudden stop and restart fails with error DBD2055
Hello,
out of the blue yesterday night Citadel server (DB version 998) stopped. Was unable to restart and got message DBD2055 in the log. File system shows lots of db log files which was unusual so I assumed some sort of DB corruption. Restored from last backup the day before (after which the server was running fine for at least 16 hours), i.e. this db was good.
After starting the server it was running for some minutes and I could see connections from MTAs as well as users but then again sudden stop and same symptom in the logs:
Apr 10 10:40:25 [hostname] citserver[659026]: Existing database version on disk is 998
Apr 10 10:40:25 [hostname] citserver[659026]: extensions: service DICT_TCP has been manually disabled, skipping
Apr 10 10:40:25 [hostname] citserver[659026]: extensions: TCP port 0.0.0.0:5222: (XMPP) registered.
Apr 10 10:40:25 [hostname] citserver[659026]: main: changing uid to 116
Apr 10 10:40:26 [hostname] citserver[659026]: citserver[659026]: bdb: BDB2055 Lock table is out of available lock entries
Apr 10 10:40:26 [hostname] citserver[659026]: citserver[659026]: bdb: bdb_fetch(9): error 12: Cannot allocate memory
Apr 10 10:40:26 [hostname] citserver[659026]: bdb: BDB2055 Lock table is out of available lock entries
Apr 10 10:40:26 [hostname] citserver[659026]: bdb: bdb_fetch(9): error 12: Cannot allocate memory
Apr 10 10:40:26 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=6/ABRT
Hi,
I faced suspicious behaviour happening on SMTP. There is a some servers that connecting to SMTP and try to login with different account names. This is obvious account scanning. Do you know a way to prevent such attacks?
I manually add some of those IPs on firewall, but this is not a way. Especially that after few minutes of silence new one occurs. IPs are from around the world so blocking specifing country is not a solution.
Kind regards,
Wojtek
I have install, reinstalled, reinstalled, reinstalled, reinstalled, etc. but each time Citadel comes up the same ->
Citadel Server
Just that but the login does not work...
Can anyone suggest what is wrong?
Zemran
Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055
out of the blue yesterday night Citadel server (DB version 998) stopped.
Was unable to restart and got message DBD2055 in the log. File system
The solution described here may fix your issue:
[ https://www.citadel.org/out_of_lock_entries.html ]
I faced suspicious behaviour happening on SMTP. There is a some servers
that connecting to SMTP and try to login with different account names.
This is obvious account scanning. Do you know a way to prevent such
attacks?
Unless you have access to nuclear weapons, get used to it. That is life as usual on the Internet. Every server (including this one) are getting targeted 24/7/365 with dictionary attacks.
I don't know how to do that. I am quite ignorant when it comes
to C coding. And I can't figure out why that would be when I am
running your docker code without modification.
Can you please walk me though that?
Yeah, I thought about that and realized you couldn't possibly be missing libraries if you're running the container. Also, I checked and apparently had forgotten that at some point we made OpenSSL a strict dependency. So forget about that suggestion.
I'm looking at the code for CtdlStartTLS() which is the common function that gets called for the STARTTLS or equivalent commands on every protocol. (And I did check out your server, it's misbehaving in the same way on every protocol handled by citserver, e it SMTP or POP or IMAP or the citadel client protocol or whatever.) The only way that message can be produced is if the SSL context for the session has not been initialized. So ... hang tight and I'll add more logging to see if we can figure out why it is doing this only on your system.
In the mean time you could run the container in the foreground with the "-x9" full debugging flag and maybe look for log messages hinting that OpenSSL is failing to initialize itself.
In the mean time you could run the container in the foreground with
the "-x9" full debugging flag and maybe look for log messages hinting
that OpenSSL is failing to initialize itself.
After further consideration ... please do exactly that.
Look for log messages beginning with "crypto:" during the initial startup of the server. There *will* be an error describing what went wrong. My guess is that somehow WebCit is picking up your certificate and Citadel Server isn't ... which is weird considering they're both looking in the same place.
After further consideration ... please do exactly that.
In fact, I think I know the problem, but I want to see your log messages first.
The ONLY thing Citadel Server does differently from WebCit when it comes to initializing TLS is that it's requesting a specific set of ciphers instead of "DEFAULT". I'm thinking that if the default set is good enough for HTTPS it's good enough for all the other protocols. And I've committed the change to git but I'm not going to publish it quite yet because I don't want to get side effects.
Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055
Fr Apr 11 2025 01:56:00 UTC von IGnatius T Foobar Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055out of the blue yesterday night Citadel server (DB version 998) stopped.
Was unable to restart and got message DBD2055 in the log. File system
The solution described here may fix your issue:
[ https://www.citadel.org/out_of_lock_entries.html ]
Excellent, thank you for your support as always.
Database is not that big (I think), around 11GiB. What I observe recently though are a huge amount of log.XXX files being created with 10MiB each - at the moment there are ~350 of them. I would have though these will be cleaned up as in the past I remember having usually one of these in the folder. Can this be related to database corruption or I/O delays?
Regards
Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055
these in the folder. Can this be related to database corruption or I/O
delays?
Yes, once the system has been running for a while it will clean up all log files that have been fully committed to the database. Usually this is rock solid so I'm wondering if you had a resource problem of some sort (disk full, etc)
In the mean time you could run the container in the foreground with
the "-x9" full debugging flag and maybe look for log messages hinting
that OpenSSL is failing to initialize itself.
After further consideration ... please do exactly that.
Look for log messages beginning with "crypto:" during the initial startup of the server. There *will* be an error describing what went wrong. My guess is that somehow WebCit is picking up your certificate and Citadel Server isn't ... which is weird considering they're both looking in the same place.
You are right but the files are there, more on that below
NO neither one of them are picking up the certs. port 80 and 443 are running behind a webserver / proxy. Remember ( --http-port=8080 --https-port=8443 )
So if you go to port 8443 you get this:
Secure Connection Failed
An error occurred during a connection to srv2.tamer.pw:8443. Cannot communicate securely with peer: no common encryption algorithm(s).
Here in GREEN is what I did, and the result in black.
# docker run -i --rm --network host --volume=/usr/local/citadel:/citadel-data citadeldotorg/citadel --http-port=8080 --https-port=8443 -x9 >>citadel.out 2>>citadel.err
# cat citadel.err | grep crypto citserver[7]: crypto: generating RSA key pair webcit[9]: crypto: [re]installing key "/citadel-data/keys/citadel.key" and certificate "/citadel-data/keys/citadel.cer" citserver[7]: crypto: generating a self-signed certificate citserver[7]: crypto: cannot read the private key citserver[7]: crypto: using certificate chain keys/citadel.cer citserver[7]: crypto: SSL_CTX_use_certificate_chain_file failed: No such file or directory citserver[7]: crypto: SSL failed: ../../context.has not been initialized # ls -alh /usr/local/citadel/keys total 8.0K drwx------ 2 root root 4.0K Apr 6 19:35 ./ drwxr-xr-x 6 root root 4.0K Apr 11 19:54 ../ lrwxrwxrwx 1 root root 49 Apr 6 19:35 citadel.cer -> /etc/letsencrypt/live/srv2.tamer.pw/fullchain.pem lrwxrwxrwx 1 root root 47 Apr 6 19:34 citadel.key -> /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem # cat /usr/local/citadel/keys/citadel.cer -----BEGIN CERTIFICATE----- MIIDqzCCAzCgAwIBAgISBub/j7QwilgbIDgv8Lo8lPfeMAoGCCqGSM49BAMDMDIx
So you are right, but the keys are there.
Further more these are the same keys the web browser is using on port 443, meaning the certs are fine. And they are linked to the correct location/folder.
What am I missing?
Here in GREEN is what I did, and the result in black.
# docker run -i --rm --network host --volume=/usr/local/citadel:/citadel-data citadeldotorg/citadel --http-port=8080 --https-port=8443 -x9 >>citadel.out 2>>citadel.err
# cat citadel.err | grep crypto citserver[7]: crypto: generating RSA key pair webcit[9]: crypto: [re]installing key "/citadel-data/keys/citadel.key" and certificate "/citadel-data/keys/citadel.cer" citserver[7]: crypto: generating a self-signed certificate citserver[7]: crypto: cannot read the private key citserver[7]: crypto: using certificate chain keys/citadel.cer citserver[7]: crypto: SSL_CTX_use_certificate_chain_file failed: No such file or directory citserver[7]: crypto: SSL failed: ../../context.has not been initialized # ls -alh /usr/local/citadel/keys total 8.0K drwx------ 2 root root 4.0K Apr 6 19:35 ./ drwxr-xr-x 6 root root 4.0K Apr 11 19:54 ../ lrwxrwxrwx 1 root root 49 Apr 6 19:35 citadel.cer -> /etc/letsencrypt/live/srv2.tamer.pw/fullchain.pem lrwxrwxrwx 1 root root 47 Apr 6 19:34 citadel.key -> /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem # cat /usr/local/citadel/keys/citadel.cer -----BEGIN CERTIFICATE----- MIIDqzCCAzCgAwIBAgISBub/j7QwilgbIDgv8Lo8lPfeMAoGCCqGSM49BAMDMDIxSo you are right, but the keys are there.
Further more these are the same keys the web browser is using on port 443, meaning the certs are fine. And they are linked to the correct location/folder.What am I missing?
Hm, I think I am on to something. I went in to the container and did following
# ls -al /citadel-data/keys total 8 drwx------ 2 root root 4096 Apr 6 19:35 . drwxr-xr-x 6 root root 4096 Apr 11 20:12 .. lrwxrwxrwx 1 root root 49 Apr 6 19:35 citadel.cer -> /etc/letsencrypt/live/srv2.tamer.pw/fullchain.pem lrwxrwxrwx 1 root root 47 Apr 6 19:34 citadel.key -> /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem # cat /citadel-data/keys/citadel.key cat: /citadel-data/keys/citadel.key: No such file or directory # cat /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem cat: /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem: No such file or directory
citadel can't follow the symlink. symlink is out of the containers reach.
But how does that work on your system?
Ok, so let's say you've got Citadel container running in Docker, using a command like this one:
docker run [other options] --volume=/usr/local/citadel:/citadel-data [other options]
So that means "/usr/local/citadel" on the host is mounted to "/citadel-data" in the container. That means you need to put your private key in /usr/local/citadel/keys/citadel.key , and you need to put your certificate (full chain) in /usr/local/citadel/keys/citadel.cer .
And I'm going to take a guess as to where you might have gone wrong, because I've seen this before: citadel.key and citadel.cer CANNOT BE SYMLINKS to something elsewhere in the host's filesystem, because the container can't follow those symlinks out of the mapped space. So the key and certificate need to be copied there, not linked, if /usr/local/citadel/keys is not their home.
Let me know, how close did I get? :)
citadel can't follow the symlink. symlink is out of the
containers reach.
TWO MINDS, ONE GREAT THOUGHT! :)
citadel can't follow the symlink. symlink is out of the containers reach.
But how does that work on your system?
So I removed the symlink and copied (hard linked) the files in to the keys folder.
Everything works fine now. How is this working on your end?
Hard linking is not a good idea,
Actually I could write a sh script for the Letsencript bot as a "post install, execute" script. That would work.
But lets here from you. How does that work on your end?
On my system, /usr/local/citadel/keys is the native location of the files, so there is no copying or linking involved. Your setup is a little more complex. We can work through some ideas if you want to. Otherwise the best course of action might be to just have a script that detects when there is a new certificate and copies it over.
Just thought of another idea: if copying the keys doesn't sit well with you, you could also map the directory where your keys live to /citadel-data/keys/ in the container. I think that might work.
On my system, /usr/local/citadel/keys is the native location of the files, so there is no copying or linking involved. Your setup is a little more complex. We can work through some ideas if you want to. Otherwise the best course of action might be to just have a script that detects when there is a new certificate and copies it over.
Well, I guess then we discovered a problem with the manual at https://citadel.org/sslcertificates.html
The manual symlinks the keys.
ln -sfv /etc/letsencrypt/live/${HOSTNAME}/privkey.pem /usr/local/citadel/keys/citadel.key
ln -sfv /etc/letsencrypt/live/${HOSTNAME}/fullchain.pem /usr/local/citadel/keys/citadel.cer
On my system, /usr/local/citadel/keys is the native location of the files, so there is no copying or linking involved. Your setup is a little more complex. We can work through some ideas if you want to. Otherwise the best course of action might be to just have a script that detects when there is a new certificate and copies it over.
Well that was what I was talking about. You don't have to detect anything certbot has ''"renewal-hooks" dir with 3 subdirs "deploy", "post", "pre"
So any script you put in to the "post" folder will be executed after certbot renews a cert.
That would be the perfect place to add a copy script.
I think you need to fix the manual tough.
Because anybody who follows that manual will have the same problem. Just most people will not persist in trying to fix it, and they will just walk away from the project.
Here is what it looks like on my server:
# pwd /etc/letsencrypt/renewal-hooks root@srv2 /e/l/renewal-hooks# ls -alh . total 20K drwxr-xr-x 5 root root 4.0K Mar 30 19:57 ./ drwxr-xr-x 7 root root 4.0K Apr 2 15:15 ../ drwxr-xr-x 2 root root 4.0K Mar 30 19:57 deploy/ drwxr-xr-x 2 root root 4.0K Apr 1 17:09 post/ drwxr-xr-x 2 root root 4.0K Apr 1 17:08 pre/ root@srv2 /e/l/renewal-hooks# ls -alh * deploy: total 8.0K drwxr-xr-x 2 root root 4.0K Mar 30 19:57 ./ drwxr-xr-x 5 root root 4.0K Mar 30 19:57 ../ post: total 12K drwxr-xr-x 2 root root 4.0K Apr 1 17:09 ./ drwxr-xr-x 5 root root 4.0K Mar 30 19:57 ../ -rwxr-xr-x 1 root root 31 Apr 1 17:09 citadel.sh* pre: total 12K drwxr-xr-x 2 root root 4.0K Apr 1 17:08 ./ drwxr-xr-x 5 root root 4.0K Mar 30 19:57 ../ -rwxr-xr-x 1 root root 30 Apr 1 17:08 citadel.sh* root@srv2 /e/l/renewal-hooks#
Here is what my hooks look like. I just added the cp hooks. The start stop I had added earlier.
root@srv2 /e/l/renewal-hooks# cat pre/citadel.sh #!/bin/sh docker stop citadel root@srv2 /e/l/renewal-hooks# cat post/citadel.sh #!/bin/sh cp /etc/letsencrypt/live/srv2.tamer.pw/fullchain.pem /usr/local/citadel/keys/citadel.cer cp /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem /usr/local/citadel/keys/citadel.key wait docker start citadel