Fri Jul 31 2026 18:06:32 EDT from IGnatius T Foobar Subject: Re: Connections hanging from server side rules

Here is the problem.  When the process is forwarding the message it is running into a lock issue and can't continue.  It leaves the message hanging and it never times out.  Eventually the ports fill up (even with the threads set to 1024) and the server eventually stops taking calls.

With such a fantastic and well-researched bug report, how can I not address it right away?   :)

I was able to reproduce your problem, and a fix has been developed.   Easy Install has the new version, and the Docker containers are being built now.

Make sure you get version 17855350 or newer.

Thanks!  I'll give it a try this evening.

Hello,

since build v17855350 Citadel setup hangs when selecting ports, but when I disconnect the SSH session new version is up an running:

I am using build script for updating to new version [curl https://easyinstall.citadel.org/install | bash]

Citadel is running on Debain 12 (Bookworm)

Hello,

I am using easy install script for updating

Setup hangs when selecting ports:

but after disconnecting ssh session new version is up an running:

System: Debian Bookworm

Hi @Ignarious, please help with the below issue I have with citadel

Running: Citadel Server 17817242 with WebCit 17817242

Environment:

- Citadel Server running inside official Docker container.

Background / Incident:

1. Server crashed due to 100% disk space exhaustion.

2. Berkeley DB transaction logs (log.0000000XXX) were removed during troubleshooting.

Current State:

- WebCit displays historical emails as blank rows dated "Jan 01 1970" with message length 0.

- Tasks, Calendar entries, and Contacts appear empty in WebCit.

- cdb.08 (4GB) is fully intact and contains all raw MIME messages, attachments, VTODO, VEVENT, and VCARD objects.

- cdb.00 contains room/subfolder definitions.

- New incoming emails arrive normally and render with full subject/date headers.

Diagnostic Conclusion:

The room definitions in cdb.00 and raw payloads in cdb.08 are intact, but cdb.09 lost its header/index entries for historical message IDs during the ungraceful shutdown and subsequent log removal.

Question:

Is there an internal citserver command, BDB utility, or repair routine to force Citadel to re-scan cdb.08 and rebuild the missing cdb.09 header index table in-place, without doing an external export and re-import?

thanks

Craig.  (ps this was generated by Gemini as it was helping me troubleshoot.. not very well I must add)

Craig,

First, a quick correction: the breakdown generated by Gemini got Citadel's database structure backwards. cdb.08 is the user table and cdb.09 is system configuration. The actual message content lives in cdb.02, and the room message indexes live in cdb.03. 

The reason you are seeing empty rows dated "Jan 01 1970" is because the message pointers in cdb.03 are pointing to message IDs in cdb.02 that were corrupted or lost when the active transaction logs (log.*) were manually deleted during the crash. In Berkeley DB, deleting active log files destroys the database's ability to maintain transactional integrity or replay pending writes.

To answer your question directly: No. There is no internal citserver repair command or in-place scan utility to reconstruct message tables without the transaction logs.

Your recovery paths at this point:

1. Restore from a known good backup made prior to the disk exhaustion event.

2. If no backup exists, standard BDB repair tools (like db_dump and db_load) can sometimes salvage salvageable pages from the individual tables, but any transactions that were in flight when the logs were deleted are gone, and referential integrity between rooms and messages cannot be automatically recreated in-place.

If you have a recent backup, restoring it is the cleanest and fastest way back to a healthy state.

Also I'm a little surprised that you had a server crash from lack of disk space alone.  Citadel Server stops accepting new messages when there is less than 100 MB free on disk.  Are you sure the server didn't crash after you deleted log.* ?

Hi, Can you please double check this as my cdb08 file is 4GB in size, the rest of the files are much smaller in comparison -

-rw------- 1 root root      479232 Aug 24 05:45 cdb.0a

-rw------- 1 root root        8192 Aug 24 09:50 cdb.0b

-rw------- 1 root root       53248 Aug 24 09:50 cdb.05

-rw------- 1 root root       12288 Aug 24 10:02 cdb.0d

-rw------- 1 root root     5033984 Aug 24 10:02 cdb.09

-rw------- 1 root root  4244172800 Aug 24 10:02 cdb.08

-rw------- 1 root root      536576 Aug 24 10:02 cdb.04

-rw------- 1 root root       16384 Aug 24 10:02 cdb.03

-rw------- 1 root root       40960 Aug 24 10:02 cdb.02

-rw------- 1 root root       20480 Aug 24 10:02 cdb.01

-rw------- 1 root root      479232 Aug 24 10:02 cdb.00

Fri Aug 21 2026 10:35:47 EDT from IGnatius T Foobar Subject: Re: Recovering form Citadel DB crash

Craig,

First, a quick correction: the breakdown generated by Gemini got Citadel's database structure backwards. cdb.08 is the user table and cdb.09 is system configuration. The actual message content lives in cdb.02, and the room message indexes live in cdb.03. 

The reason you are seeing empty rows dated "Jan 01 1970" is because the message pointers in cdb.03 are pointing to message IDs in cdb.02 that were corrupted or lost when the active transaction logs (log.*) were manually deleted during the crash. In Berkeley DB, deleting active log files destroys the database's ability to maintain transactional integrity or replay pending writes.

To answer your question directly: No. There is no internal citserver repair command or in-place scan utility to reconstruct message tables without the transaction logs.

Your recovery paths at this point:

1. Restore from a known good backup made prior to the disk exhaustion event.

2. If no backup exists, standard BDB repair tools (like db_dump and db_load) can sometimes salvage salvageable pages from the individual tables, but any transactions that were in flight when the logs were deleted are gone, and referential integrity between rooms and messages cannot be automatically recreated in-place.

If you have a recent backup, restoring it is the cleanest and fastest way back to a healthy state.

Hi, I dont have a recent backup.. that's the issue.. I stopped using google drive for backups few months back due to the price increase.. I was in process of looking for alternative.

Tue Aug 25 2026 18:24:38 EDT from IGnatius T Foobar Subject: Re: Recovering form Citadel DB crash

Hi, Can you please double check this as my cdb08 file is 4GB in size,
the rest of the files are much smaller in comparison -

The files are different sizes because they contain different tables. But they can't be recovered on their own. If you deleted the journal files (log.*) and maps (__db.00*) you've deleted the parts of the database that make it recoverable.

Please just restore your most recent backup. It is your best and only option right now.

If you don't have a backup, then I'm afraid you're out of luck.  When you deleted the write-ahead logs you deleted the very thing that makes the database recoverable.  Berkeley DB is exceptionally stable but obviously it can't re-create data that isn't there.

If there's something in those files that you really need, you might be able to use your system's "strings" utility to dump them into a text file, but the chances of recovering them back into Citadel is essentially zero.

I'm trying to be gentle here because you're clearly a loyal Citadel user and I appreciate that immensely, but your database is definitely unrecoverable and you have to start over.

actually, Gemini has been helping with Berkely and Citadel and we have managed to extract the emails from the database tables and import them back in with IMAP.   It's taking a while but a least I have most of what I need back. I also managed to scrape the tasks and contacts etc.

Wed Aug 26 2026 14:59:01 EDT from IGnatius T Foobar Subject: Re: Recovering form Citadel DB crash

If you don't have a backup, then I'm afraid you're out of luck.  When you deleted the write-ahead logs you deleted the very thing that makes the database recoverable.  Berkeley DB is exceptionally stable but obviously it can't re-create data that isn't there.

If there's something in those files that you really need, you might be able to use your system's "strings" utility to dump them into a text file, but the chances of recovering them back into Citadel is essentially zero.

I'm trying to be gentle here because you're clearly a loyal Citadel user and I appreciate that immensely, but your database is definitely unrecoverable and you have to start over.

cool

Thu Aug 27 2026 19:19:24 EDT from IGnatius T Foobar Subject: Re: Recovering form Citadel DB crash

<osborne> By the way, the new UI is just days away from feature complete, and I can't wait to show it to you! </osborne>