Please wait...
0 files
Check out https://www.citadel.org/docker.html and scroll to the bottom to see. Specifically, you can now do "--http-port" and/or "--https-port" to change the port numbers used by WebCit for HTTP and/or HTTPS.
That ought to make "--network=host" mode work better on your system, right?
By the way, I tried the "-a" option to "docker run" and it worked fine when combined with "--network=host" so I'm not sure what's different on yours.
Also, why "-a" and not "-i"? Doesn't "-i" do the same thing?
Anyway, if there's anything else we can do to make the container run better on your system, let me know. It really isn't that difficult to add options.
without going into details: how do backup Citadel DB without stopping
server? ctdldump require (accourding to docs) to stop citadel server. Is
there any other way to do backup?
You can back up the Citadel database directly [ https://www.citadel.org/what_is_the_best_way_to_backup_my_citadel_installation.html ] as long as you make sure the cdb.* files are backed up first, before the log.* files.
The dump format is not really intended for backups. It's intended for migrating between different architectures.
But let me tell you how I do it :)
I've got my Citadel stored on a filesystem that can do snapshots. For me, that's BTRFS, but you can use any filesystem that can do point-in-time snapshots.
So it's simple, really: take a snapshot of the volume (or subvolume) that has Citadel on it, then rsync that snapshot to wherever you want to save it.
I happen to go the extra mile and rotate my snapshots over the course of a week, but you get the idea: the snapshot is guaranteed by the filesystem to be point-in-time consistent, and Citadel Server of any version starting with 993 has absolutely rock solid recoverability as long as you've got all the recent logs still on disk.
Hi,
thanks a lot. Seems that I have to migrate to btrfs of lvm.
Thanks for fast reply.
All right, my friend, it was bothering me that the container image was missing a few simple options that might make it work better for you, so I added them.
Check out https://www.citadel.org/docker.html and scroll to the bottom to see. Specifically, you can now do "--http-port" and/or "--https-port" to change the port numbers used by WebCit for HTTP and/or HTTPS.
That ought to make "--network=host" mode work better on your system, right?
By the way, I tried the "-a" option to "docker run" and it worked fine when combined with "--network=host" so I'm not sure what's different on yours.
Also, why "-a" and not "-i"? Doesn't "-i" do the same thing?
Anyway, if there's anything else we can do to make the container run better on your system, let me know. It really isn't that difficult to add options.
First of all, thank you for putting up with me, and this issue. I am sure it will turn up all to be my own fault.
Second I was wrong it has nothing to do with --network-host
I did upgrade to the new docker version of yours. Webcit has the same ssl problem I am running webcit on port 8443 so you can check that here: https://srv2.tamer.pw:8443/ vs the same ssl keys running behind lightltp web server https://srv2.tamer.pw
Making sure I got the right keys linked in to docker:
root@srv2 /u/l/c/keys# la
total 0
lrwxrwxrwx 1 root root 49 Apr 6 19:35 citadel.cer -> /etc/letsencrypt/live/srv2.tamer.pw/fullchain.pem
lrwxrwxrwx 1 root root 47 Apr 6 19:34 citadel.key -> /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem
I also checked other ports again as you suggested such as imap on 143, pop3 on 993, and smtp on 587
So the only thing left to check is your suggestion to see if citadel is linked to openssl.
I don't know how to do that. I am quite ignorant when it comes to C coding. And I can't figure out why that would be when I am running your docker code without modification.
Can you please walk me though that?
Thanks again. We will figure this out.
Subject: Citadel Server sudden stop and restart fails with error DBD2055
Hello,
out of the blue yesterday night Citadel server (DB version 998) stopped. Was unable to restart and got message DBD2055 in the log. File system shows lots of db log files which was unusual so I assumed some sort of DB corruption. Restored from last backup the day before (after which the server was running fine for at least 16 hours), i.e. this db was good.
After starting the server it was running for some minutes and I could see connections from MTAs as well as users but then again sudden stop and same symptom in the logs:
Apr 10 10:40:25 [hostname] citserver[659026]: Existing database version on disk is 998
Apr 10 10:40:25 [hostname] citserver[659026]: extensions: service DICT_TCP has been manually disabled, skipping
Apr 10 10:40:25 [hostname] citserver[659026]: extensions: TCP port 0.0.0.0:5222: (XMPP) registered.
Apr 10 10:40:25 [hostname] citserver[659026]: main: changing uid to 116
Apr 10 10:40:26 [hostname] citserver[659026]: citserver[659026]: bdb: BDB2055 Lock table is out of available lock entries
Apr 10 10:40:26 [hostname] citserver[659026]: citserver[659026]: bdb: bdb_fetch(9): error 12: Cannot allocate memory
Apr 10 10:40:26 [hostname] citserver[659026]: bdb: BDB2055 Lock table is out of available lock entries
Apr 10 10:40:26 [hostname] citserver[659026]: bdb: bdb_fetch(9): error 12: Cannot allocate memory
Apr 10 10:40:26 [hostname] systemd[1]: citadel.service: Main process exited, code=killed, status=6/ABRT
Hi,
I faced suspicious behaviour happening on SMTP. There is a some servers that connecting to SMTP and try to login with different account names. This is obvious account scanning. Do you know a way to prevent such attacks?
I manually add some of those IPs on firewall, but this is not a way. Especially that after few minutes of silence new one occurs. IPs are from around the world so blocking specifing country is not a solution.
Kind regards,
Wojtek
I have install, reinstalled, reinstalled, reinstalled, reinstalled, etc. but each time Citadel comes up the same ->
Citadel Server
Just that but the login does not work...
Can anyone suggest what is wrong?
Zemran
Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055
out of the blue yesterday night Citadel server (DB version 998) stopped.
Was unable to restart and got message DBD2055 in the log. File system
The solution described here may fix your issue:
[ https://www.citadel.org/out_of_lock_entries.html ]
I faced suspicious behaviour happening on SMTP. There is a some servers
that connecting to SMTP and try to login with different account names.
This is obvious account scanning. Do you know a way to prevent such
attacks?
Unless you have access to nuclear weapons, get used to it. That is life as usual on the Internet. Every server (including this one) are getting targeted 24/7/365 with dictionary attacks.
I don't know how to do that. I am quite ignorant when it comes
to C coding. And I can't figure out why that would be when I am
running your docker code without modification.
Can you please walk me though that?
Yeah, I thought about that and realized you couldn't possibly be missing libraries if you're running the container. Also, I checked and apparently had forgotten that at some point we made OpenSSL a strict dependency. So forget about that suggestion.
I'm looking at the code for CtdlStartTLS() which is the common function that gets called for the STARTTLS or equivalent commands on every protocol. (And I did check out your server, it's misbehaving in the same way on every protocol handled by citserver, e it SMTP or POP or IMAP or the citadel client protocol or whatever.) The only way that message can be produced is if the SSL context for the session has not been initialized. So ... hang tight and I'll add more logging to see if we can figure out why it is doing this only on your system.
In the mean time you could run the container in the foreground with the "-x9" full debugging flag and maybe look for log messages hinting that OpenSSL is failing to initialize itself.
In the mean time you could run the container in the foreground with
the "-x9" full debugging flag and maybe look for log messages hinting
that OpenSSL is failing to initialize itself.
After further consideration ... please do exactly that.
Look for log messages beginning with "crypto:" during the initial startup of the server. There *will* be an error describing what went wrong. My guess is that somehow WebCit is picking up your certificate and Citadel Server isn't ... which is weird considering they're both looking in the same place.
After further consideration ... please do exactly that.
In fact, I think I know the problem, but I want to see your log messages first.
The ONLY thing Citadel Server does differently from WebCit when it comes to initializing TLS is that it's requesting a specific set of ciphers instead of "DEFAULT". I'm thinking that if the default set is good enough for HTTPS it's good enough for all the other protocols. And I've committed the change to git but I'm not going to publish it quite yet because I don't want to get side effects.
Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055
Fr Apr 11 2025 01:56:00 UTC von IGnatius T Foobar Betreff: Re: Citadel Server sudden stop and restart fails with error DBD2055out of the blue yesterday night Citadel server (DB version 998) stopped.
Was unable to restart and got message DBD2055 in the log. File system
The solution described here may fix your issue:
[ https://www.citadel.org/out_of_lock_entries.html ]
Excellent, thank you for your support as always.
Database is not that big (I think), around 11GiB. What I observe recently though are a huge amount of log.XXX files being created with 10MiB each - at the moment there are ~350 of them. I would have though these will be cleaned up as in the past I remember having usually one of these in the folder. Can this be related to database corruption or I/O delays?
Regards
Subject: Re: Citadel Server sudden stop and restart fails with error DBD2055
these in the folder. Can this be related to database corruption or I/O
delays?
Yes, once the system has been running for a while it will clean up all log files that have been fully committed to the database. Usually this is rock solid so I'm wondering if you had a resource problem of some sort (disk full, etc)
In the mean time you could run the container in the foreground with
the "-x9" full debugging flag and maybe look for log messages hinting
that OpenSSL is failing to initialize itself.
After further consideration ... please do exactly that.
Look for log messages beginning with "crypto:" during the initial startup of the server. There *will* be an error describing what went wrong. My guess is that somehow WebCit is picking up your certificate and Citadel Server isn't ... which is weird considering they're both looking in the same place.
You are right but the files are there, more on that below
NO neither one of them are picking up the certs. port 80 and 443 are running behind a webserver / proxy. Remember ( --http-port=8080 --https-port=8443 )
So if you go to port 8443 you get this:
Secure Connection Failed
An error occurred during a connection to srv2.tamer.pw:8443. Cannot communicate securely with peer: no common encryption algorithm(s).
Here in GREEN is what I did, and the result in black.
# docker run -i --rm --network host --volume=/usr/local/citadel:/citadel-data citadeldotorg/citadel --http-port=8080 --https-port=8443 -x9 >>citadel.out 2>>citadel.err
# cat citadel.err | grep crypto citserver[7]: crypto: generating RSA key pair webcit[9]: crypto: [re]installing key "/citadel-data/keys/citadel.key" and certificate "/citadel-data/keys/citadel.cer" citserver[7]: crypto: generating a self-signed certificate citserver[7]: crypto: cannot read the private key citserver[7]: crypto: using certificate chain keys/citadel.cer citserver[7]: crypto: SSL_CTX_use_certificate_chain_file failed: No such file or directory citserver[7]: crypto: SSL failed: ../../context.has not been initialized # ls -alh /usr/local/citadel/keys total 8.0K drwx------ 2 root root 4.0K Apr 6 19:35 ./ drwxr-xr-x 6 root root 4.0K Apr 11 19:54 ../ lrwxrwxrwx 1 root root 49 Apr 6 19:35 citadel.cer -> /etc/letsencrypt/live/srv2.tamer.pw/fullchain.pem lrwxrwxrwx 1 root root 47 Apr 6 19:34 citadel.key -> /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem # cat /usr/local/citadel/keys/citadel.cer -----BEGIN CERTIFICATE----- MIIDqzCCAzCgAwIBAgISBub/j7QwilgbIDgv8Lo8lPfeMAoGCCqGSM49BAMDMDIx
So you are right, but the keys are there.
Further more these are the same keys the web browser is using on port 443, meaning the certs are fine. And they are linked to the correct location/folder.
What am I missing?
Here in GREEN is what I did, and the result in black.
# docker run -i --rm --network host --volume=/usr/local/citadel:/citadel-data citadeldotorg/citadel --http-port=8080 --https-port=8443 -x9 >>citadel.out 2>>citadel.err
# cat citadel.err | grep crypto citserver[7]: crypto: generating RSA key pair webcit[9]: crypto: [re]installing key "/citadel-data/keys/citadel.key" and certificate "/citadel-data/keys/citadel.cer" citserver[7]: crypto: generating a self-signed certificate citserver[7]: crypto: cannot read the private key citserver[7]: crypto: using certificate chain keys/citadel.cer citserver[7]: crypto: SSL_CTX_use_certificate_chain_file failed: No such file or directory citserver[7]: crypto: SSL failed: ../../context.has not been initialized # ls -alh /usr/local/citadel/keys total 8.0K drwx------ 2 root root 4.0K Apr 6 19:35 ./ drwxr-xr-x 6 root root 4.0K Apr 11 19:54 ../ lrwxrwxrwx 1 root root 49 Apr 6 19:35 citadel.cer -> /etc/letsencrypt/live/srv2.tamer.pw/fullchain.pem lrwxrwxrwx 1 root root 47 Apr 6 19:34 citadel.key -> /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem # cat /usr/local/citadel/keys/citadel.cer -----BEGIN CERTIFICATE----- MIIDqzCCAzCgAwIBAgISBub/j7QwilgbIDgv8Lo8lPfeMAoGCCqGSM49BAMDMDIxSo you are right, but the keys are there.
Further more these are the same keys the web browser is using on port 443, meaning the certs are fine. And they are linked to the correct location/folder.What am I missing?
Hm, I think I am on to something. I went in to the container and did following
# ls -al /citadel-data/keys total 8 drwx------ 2 root root 4096 Apr 6 19:35 . drwxr-xr-x 6 root root 4096 Apr 11 20:12 .. lrwxrwxrwx 1 root root 49 Apr 6 19:35 citadel.cer -> /etc/letsencrypt/live/srv2.tamer.pw/fullchain.pem lrwxrwxrwx 1 root root 47 Apr 6 19:34 citadel.key -> /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem # cat /citadel-data/keys/citadel.key cat: /citadel-data/keys/citadel.key: No such file or directory # cat /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem cat: /etc/letsencrypt/live/srv2.tamer.pw/privkey.pem: No such file or directory
citadel can't follow the symlink. symlink is out of the containers reach.
But how does that work on your system?
Ok, so let's say you've got Citadel container running in Docker, using a command like this one:
docker run [other options] --volume=/usr/local/citadel:/citadel-data [other options]
So that means "/usr/local/citadel" on the host is mounted to "/citadel-data" in the container. That means you need to put your private key in /usr/local/citadel/keys/citadel.key , and you need to put your certificate (full chain) in /usr/local/citadel/keys/citadel.cer .
And I'm going to take a guess as to where you might have gone wrong, because I've seen this before: citadel.key and citadel.cer CANNOT BE SYMLINKS to something elsewhere in the host's filesystem, because the container can't follow those symlinks out of the mapped space. So the key and certificate need to be copied there, not linked, if /usr/local/citadel/keys is not their home.
Let me know, how close did I get? :)
citadel can't follow the symlink. symlink is out of the
containers reach.
TWO MINDS, ONE GREAT THOUGHT! :)
citadel can't follow the symlink. symlink is out of the containers reach.
But how does that work on your system?
So I removed the symlink and copied (hard linked) the files in to the keys folder.
Everything works fine now. How is this working on your end?
Hard linking is not a good idea,
Actually I could write a sh script for the Letsencript bot as a "post install, execute" script. That would work.
But lets here from you. How does that work on your end?
On my system, /usr/local/citadel/keys is the native location of the files, so there is no copying or linking involved. Your setup is a little more complex. We can work through some ideas if you want to. Otherwise the best course of action might be to just have a script that detects when there is a new certificate and copies it over.