Hi, I dont have a recent backup.. that's the issue.. I stopped using google drive for backups few months back due to the price increase.. I was in process of looking for alternative.

Tue Aug 25 2026 18:24:38 EDT from IGnatius T Foobar Subject: Re: Recovering form Citadel DB crash

Hi, Can you please double check this as my cdb08 file is 4GB in size,
the rest of the files are much smaller in comparison -

The files are different sizes because they contain different tables. But they can't be recovered on their own. If you deleted the journal files (log.*) and maps (__db.00*) you've deleted the parts of the database that make it recoverable.

Please just restore your most recent backup. It is your best and only option right now.

Hi, Can you please double check this as my cdb08 file is 4GB in size, the rest of the files are much smaller in comparison -

-rw------- 1 root root      479232 Aug 24 05:45 cdb.0a

-rw------- 1 root root        8192 Aug 24 09:50 cdb.0b

-rw------- 1 root root       53248 Aug 24 09:50 cdb.05

-rw------- 1 root root       12288 Aug 24 10:02 cdb.0d

-rw------- 1 root root     5033984 Aug 24 10:02 cdb.09

-rw------- 1 root root  4244172800 Aug 24 10:02 cdb.08

-rw------- 1 root root      536576 Aug 24 10:02 cdb.04

-rw------- 1 root root       16384 Aug 24 10:02 cdb.03

-rw------- 1 root root       40960 Aug 24 10:02 cdb.02

-rw------- 1 root root       20480 Aug 24 10:02 cdb.01

-rw------- 1 root root      479232 Aug 24 10:02 cdb.00

Fri Aug 21 2026 10:35:47 EDT from IGnatius T Foobar Subject: Re: Recovering form Citadel DB crash

Craig,

First, a quick correction: the breakdown generated by Gemini got Citadel's database structure backwards. cdb.08 is the user table and cdb.09 is system configuration. The actual message content lives in cdb.02, and the room message indexes live in cdb.03. 

The reason you are seeing empty rows dated "Jan 01 1970" is because the message pointers in cdb.03 are pointing to message IDs in cdb.02 that were corrupted or lost when the active transaction logs (log.*) were manually deleted during the crash. In Berkeley DB, deleting active log files destroys the database's ability to maintain transactional integrity or replay pending writes.

To answer your question directly: No. There is no internal citserver repair command or in-place scan utility to reconstruct message tables without the transaction logs.

Your recovery paths at this point:

1. Restore from a known good backup made prior to the disk exhaustion event.

2. If no backup exists, standard BDB repair tools (like db_dump and db_load) can sometimes salvage salvageable pages from the individual tables, but any transactions that were in flight when the logs were deleted are gone, and referential integrity between rooms and messages cannot be automatically recreated in-place.

If you have a recent backup, restoring it is the cleanest and fastest way back to a healthy state.

Also I'm a little surprised that you had a server crash from lack of disk space alone.  Citadel Server stops accepting new messages when there is less than 100 MB free on disk.  Are you sure the server didn't crash after you deleted log.* ?

Craig,

First, a quick correction: the breakdown generated by Gemini got Citadel's database structure backwards. cdb.08 is the user table and cdb.09 is system configuration. The actual message content lives in cdb.02, and the room message indexes live in cdb.03. 

The reason you are seeing empty rows dated "Jan 01 1970" is because the message pointers in cdb.03 are pointing to message IDs in cdb.02 that were corrupted or lost when the active transaction logs (log.*) were manually deleted during the crash. In Berkeley DB, deleting active log files destroys the database's ability to maintain transactional integrity or replay pending writes.

To answer your question directly: No. There is no internal citserver repair command or in-place scan utility to reconstruct message tables without the transaction logs.

Your recovery paths at this point:

1. Restore from a known good backup made prior to the disk exhaustion event.

2. If no backup exists, standard BDB repair tools (like db_dump and db_load) can sometimes salvage salvageable pages from the individual tables, but any transactions that were in flight when the logs were deleted are gone, and referential integrity between rooms and messages cannot be automatically recreated in-place.

If you have a recent backup, restoring it is the cleanest and fastest way back to a healthy state.

Hi @Ignarious, please help with the below issue I have with citadel

Running: Citadel Server 17817242 with WebCit 17817242

Environment:

- Citadel Server running inside official Docker container.

Background / Incident:

1. Server crashed due to 100% disk space exhaustion.

2. Berkeley DB transaction logs (log.0000000XXX) were removed during troubleshooting.

Current State:

- WebCit displays historical emails as blank rows dated "Jan 01 1970" with message length 0.

- Tasks, Calendar entries, and Contacts appear empty in WebCit.

- cdb.08 (4GB) is fully intact and contains all raw MIME messages, attachments, VTODO, VEVENT, and VCARD objects.

- cdb.00 contains room/subfolder definitions.

- New incoming emails arrive normally and render with full subject/date headers.

Diagnostic Conclusion:

The room definitions in cdb.00 and raw payloads in cdb.08 are intact, but cdb.09 lost its header/index entries for historical message IDs during the ungraceful shutdown and subsequent log removal.

Question:

Is there an internal citserver command, BDB utility, or repair routine to force Citadel to re-scan cdb.08 and rebuild the missing cdb.09 header index table in-place, without doing an external export and re-import?

thanks

Craig.  (ps this was generated by Gemini as it was helping me troubleshoot.. not very well I must add)

Hello,

I am using easy install script for updating

Setup hangs when selecting ports:

System: Debian Bookworm

Hello,

since build v17855350 Citadel setup hangs when selecting ports, but when I disconnect the SSH session new version is up an running:

curl https://easyinstall.citadel.org/install | bash]

Citadel is running on Debain 12 (Bookworm)

2099569965@Uncensored

Fri Jul 31 2026 18:06:32 EDT from IGnatius T Foobar Subject: Re: Connections hanging from server side rules

Here is the problem.  When the process is forwarding the message it is running into a lock issue and can't continue.  It leaves the message hanging and it never times out.  Eventually the ports fill up (even with the threads set to 1024) and the server eventually stops taking calls.

With such a fantastic and well-researched bug report, how can I not address it right away?   :)

I was able to reproduce your problem, and a fix has been developed.   Easy Install has the new version, and the Docker containers are being built now.

Make sure you get version 17855350 or newer.

Thanks!  I'll give it a try this evening.

Here is the problem.  When the process is forwarding the message it is running into a lock issue and can't continue.  It leaves the message hanging and it never times out.  Eventually the ports fill up (even with the threads set to 1024) and the server eventually stops taking calls.

With such a fantastic and well-researched bug report, how can I not address it right away?   :)

I was able to reproduce your problem, and a fix has been developed.   Easy Install has the new version, and the Docker containers are being built now.

Make sure you get version 17855350 or newer.

Hello,

I installed the following on my Fedora 43 server.

Running: Citadel Server 17846585 with WebCit 17846585
Server Build: 17846585

The DKIM addition worked great.  Thanks!

I did however find an issue that I'm hoping can be corrected.  Here is what is happening:

I like to separate my spam from my good email so my phone doesn't get overloaded.  In the old version I had a rule that looked for the Spam-Status flag and if it was 'Yes' it would forward the email to a spam account and then delete the message.  This way the email went to the spam account and cleared up the main account.  It also allowed me to go back to that spam account in case email went there by accident.

Here is the problem.  When the process is forwarding the message it is running into a lock issue and can't continue.  It leaves the message hanging and it never times out.  Eventually the ports fill up (even with the threads set to 1024) and the server eventually stops taking calls.

Here is part of a conversation with Gemini that was achieved by running a debugger and tracing the futex to the memory location.

Migrated my Citadel setup from internal SMTP delivery to native LMTP

Hi everyone,

after running Citadel in production together with Postfix and Rspamd for quite some time, I recently migrated the internal mail delivery path from SMTP (localhost:2025) to native LMTP using Citadel's Unix domain sockets.

The setup is running stable in production now and works very well.

Current architecture:

Internet | Postfix | Rspamd | LMTP Unix Socket | Citadel 

The migration uses:

• Postfix as SMTP edge gateway

• Rspamd for filtering and signing

• Citadel as backend groupware and mail store

• LMTP over Unix sockets for local delivery

I documented the migration process, configuration changes and testing steps here:

Postfix LMTP Migration Documentation

The repository itself is here:

modern-citadel-mail-platform

Maybe this helps someone building a modern Citadel-based mail platform with Postfix and Rspamd.

And thanks to the Citadel developers for still keeping classic Unix-style groupware alive 🙂

So Mai 17 2026 22:32:02 EDT von IGnatius T Foobar Betreff: Re: Thoughts about modern Citadel

That's really neat.  The current implementation of Citadel was specifically designed so that it can run as a self-sufficient standalone service or be part of a larger software ecosystem.  I'm thrilled to see that you're doing the latter.  

I'm curious how you've coupled Postfix and Citadel.  Are you using the LMTP socket?

Some system specs... Maybe this isn’t the right thread for it, but if you’re interested, let’s talk about it ;)

At the moment I'm actually using Postfix as SMTP frontend together with Rspamd, and then forwarding internally to Citadel via SMTP on port 2025.

So the current architecture is roughly:

Internet -> Postfix -> Rspamd -> Citadel SMTP

I intentionally kept the setup modular and relatively classic Unix-oriented instead of tightly coupling everything together.

LMTP is something I'm currently evaluating, but the existing SMTP-based integration has been extremely stable and easy to maintain so far.

So Mai 17 2026 22:32:02 EDT von IGnatius T Foobar Betreff: Re: Thoughts about modern Citadel

That's really neat.  The current implementation of Citadel was specifically designed so that it can run as a self-sufficient standalone service or be part of a larger software ecosystem.  I'm thrilled to see that you're doing the latter.  

I'm curious how you've coupled Postfix and Citadel.  Are you using the LMTP socket?

At the moment I'm actually not using LMTP between Postfix and Citadel.

The current setup uses Postfix mainly as SMTP frontend together with the traditional virtual/local transport model while Citadel remains the central communication and storage platform behind it.

I intentionally kept the architecture relatively classic and modular instead of tightly coupling everything together.

The overall goal was long-term maintainability, transparency and low operational complexity while still implementing modern mail security standards around the platform.

So Mai 17 2026 22:32:02 EDT von IGnatius T Foobar Betreff: Re: Thoughts about modern Citadel

That's really neat.  The current implementation of Citadel was specifically designed so that it can run as a self-sufficient standalone service or be part of a larger software ecosystem.  I'm thrilled to see that you're doing the latter.  

I'm curious how you've coupled Postfix and Citadel.  Are you using the LMTP socket?

That's really neat.  The current implementation of Citadel was specifically designed so that it can run as a self-sufficient standalone service or be part of a larger software ecosystem.  I'm thrilled to see that you're doing the latter.  

I'm curious how you've coupled Postfix and Citadel.  Are you using the LMTP socket?

If someome is interessted 

Modern self-hosted Citadel communication platform

Greetings

This works really good for me. I have different purge rules for different folders. The purge will done with

My Citadel Version is 1022 now for a long time. 

If you want know more about my settings, let me know. 

Greeintngs

Mike

So Mai 17 2026 04:03:52 EDT von luisgo Betreff: Re: Size of Data Files.

I think you are providing a semi-featured Citadel. The purge is not working. It supposed to purge the messages from 1st of February and it is not doing.

 

 

Mon May 11 2026 10:56:43 EDT from Motobike Subject: Re: Size of Data Files.

You can define a purge rule for every folder. Also a rule to move it to archive. 

 

So Mai 10 2026 04:53:26 EDT von luisgo Betreff: Re: Size of Data Files.

I want to shrink by purging the older emails previous deleted manually. I has the Auto-Purge setting.

I suppose that I can not do it configuring "Default message expire policy for private mailboxes". Or can I? I suppose that is to delete and purge active messages.

Seldom times appear notifications of purged messages at Aide and if so only one purged message or EEDI (??)

Mon May 04 2026 19:25:09 EDT from IGnatius T Foobar Subject: Re: Size of Data Files.

 

Are there some way to shrink the size of the main citadel data file without compromizing the citadel stability?

Do you have the auto-purger configured to delete "old" messages?

You can do this at the site-wide level, or on an individual room (such as your trash folder).

If new data is arriving at approximately the same rate old data is expiring, then yes, eventually you will reach an equilibrium where the database files stabilize in size.

Do look in the Aide room for messages confirming this.  The helpful Citadel Aide will tell you how many objects are being deleted during the nightly purger run.

 

 

I think you are providing a semi-featured Citadel. The purge is not working. It supposed to purge the messages from 1st of February and it is not doing.

Mon May 11 2026 10:56:43 EDT from Motobike Subject: Re: Size of Data Files.

With a Citadel Network user appears.

Tue May 12 2026 16:13:45 EDT from Motobike Subject: Re: Size of Data Files.

with his own folders / rooms he can. 

depends on rights of the folders. but your own folders like inbox, archive and so on, you can. 

Di Mai 12 2026 15:29:18 EDT von luisgo Betreff: Re: Size of Data Files.

or on an individual room

A user without Administrator level can not do this. Am I right? Why so?

Mon May 04 2026 19:25:09 EDT from IGnatius T Foobar Subject: Re: Size of Data Files.

 

Are there some way to shrink the size of the main citadel data file without compromizing the citadel stability?

Do you have the auto-purger configured to delete "old" messages?

You can do this at the site-wide level, or on an individual room (such as your trash folder).

If new data is arriving at approximately the same rate old data is expiring, then yes, eventually you will reach an equilibrium where the database files stabilize in size.

Do look in the Aide room for messages confirming this.  The helpful Citadel Aide will tell you how many objects are being deleted during the nightly purger run.

 

or on an individual room

A user without Administrator level can not do this. Am I right? Why so?

Mon May 04 2026 19:25:09 EDT from IGnatius T Foobar Subject: Re: Size of Data Files.

 

Are there some way to shrink the size of the main citadel data file without compromizing the citadel stability?

Do you have the auto-purger configured to delete "old" messages?

You can do this at the site-wide level, or on an individual room (such as your trash folder).

If new data is arriving at approximately the same rate old data is expiring, then yes, eventually you will reach an equilibrium where the database files stabilize in size.

Do look in the Aide room for messages confirming this.  The helpful Citadel Aide will tell you how many objects are being deleted during the nightly purger run.

You can define a purge rule for every folder. Also a rule to move it to archive. 

So Mai 10 2026 04:53:26 EDT von luisgo Betreff: Re: Size of Data Files.

I want to shrink by purging the older emails previous deleted manually. I has the Auto-Purge setting.

I suppose that I can not do it configuring "Default message expire policy for private mailboxes". Or can I? I suppose that is to delete and purge active messages.

Seldom times appear notifications of purged messages at Aide and if so only one purged message or EEDI (??)

Mon May 04 2026 19:25:09 EDT from IGnatius T Foobar Subject: Re: Size of Data Files.

 

Are there some way to shrink the size of the main citadel data file without compromizing the citadel stability?

Do you have the auto-purger configured to delete "old" messages?

You can do this at the site-wide level, or on an individual room (such as your trash folder).

If new data is arriving at approximately the same rate old data is expiring, then yes, eventually you will reach an equilibrium where the database files stabilize in size.

Do look in the Aide room for messages confirming this.  The helpful Citadel Aide will tell you how many objects are being deleted during the nightly purger run.

 

I want to shrink by purging the older emails previous deleted manually. I has the Auto-Purge setting.

I suppose that I can not do it configuring "Default message expire policy for private mailboxes". Or can I? I suppose that is to delete and purge active messages.

Seldom times appear notifications of purged messages at Aide and if so only one purged message or EEDI (??)

Mon May 04 2026 19:25:09 EDT from IGnatius T Foobar Subject: Re: Size of Data Files.

 

Are there some way to shrink the size of the main citadel data file without compromizing the citadel stability?

Do you have the auto-purger configured to delete "old" messages?

You can do this at the site-wide level, or on an individual room (such as your trash folder).

If new data is arriving at approximately the same rate old data is expiring, then yes, eventually you will reach an equilibrium where the database files stabilize in size.

Do look in the Aide room for messages confirming this.  The helpful Citadel Aide will tell you how many objects are being deleted during the nightly purger run.

Are there some way to shrink the size of the main citadel data file without compromizing the citadel stability?

Do you have the auto-purger configured to delete "old" messages?

You can do this at the site-wide level, or on an individual room (such as your trash folder).

If new data is arriving at approximately the same rate old data is expiring, then yes, eventually you will reach an equilibrium where the database files stabilize in size.

Do look in the Aide room for messages confirming this.  The helpful Citadel Aide will tell you how many objects are being deleted during the nightly purger run.

Found it!

You have to map /etc/localtime in the container to /etc/localtime on the host.  You can do it with a command like this:

docker run \
   -d \
   -e TZ=America/New_York \
   -v /etc/localtime:/etc/localtime:ro \
   --restart=unless-stopped \
   --network host \
   --volume=/usr/local/citadel:/citadel-data \
   --volume=/usr/local/webcit/.well-known:/usr/local/webcit/.well-known \
   --name=citadel \
   citadeldotorg/citadel

 The key is that "-v" command up there.  You were correct in setting "-e" to your timezone.  Add the "-v" and it ought to work.

Tested, right here, just now.

Mon May 04 2026 11:08:02 PM UTC from IGnatius T Foobar Subject: Re: How to modify timezone for users/server to set correct time on emails

 

I stand corrected.  Look at that, it's showing the time in UTC right here on Uncensored, which is definitely running in US Eastern time zone.   Let me check the build and see what's up with that.

Thanks for the report.

I have checked the result of $date in the docker container (docker exec -it citadel /bin/bash) which is correct, and have also tried to change the timezone environment variable in the docker run command with -e TZ=GMT and -e TZ=Africa/Johannesburg, which does reflect the time change in the docker container bash date command, however this doesn't make a difference to the timestamp of emails sent/received, so I'm assuming I can set this somewhere on Citadel but am hoping for a nudge in the right direction as to where since I'm unable to find anything in the web interface logged in as admin or as the mailbox user. 

Citadel clients (including WebCit) render the date and time based on the time zone the client is set to.  Since you're probably using WebCit Classic that's definitely the web server's time zone setting.

For sending out mail, the timezone of the host is relevant, the zone the host is set to before you run the Docker container.

Dear All,

My settings:

Centos 9, 2 threads, 3GBytes Memory, 4GBytes Swap Memory, 15GBytes free Disk. Citadel with 2 email users.

As I said in a previous post my citadel data files does a rotation every 3-4 months. I have emails of only of the last 3 to 4 months. Every month I delete the older month. Usually after 3-4 months the citadel main data file reduces drastically in size after increasing for 3-4 months. Now I notice that citadel seems more stable (no core dumps). However I do not see the data file redution. By seeing the "Size on Disk" (with Properties) in Thunderbird I can see that I have 1.5 GBytes of emails but the main citadel data file have 4 GBytes.

I use a lot the ctdldump and ctdlload (offline commands) as I saw that improves the stability of Citadel. I use/do usually when I trash a lot of emails (in size).

Are there some way to shrink the size of the main citadel data file without compromizing the citadel stability?

Thank you,

Luís Gonçalves.

Hello fellow Citadel supporters

I hope you are well.

I have deployed Citadel using the docker container method. It seems all e-mails timestamps are offset by 2 hours ahead. I am wondering if there is a way and what the best method is to set a date/time or timezone for the Citadel server or per user.

I have checked the result of $date in the docker container (docker exec -it citadel /bin/bash) which is correct, and have also tried to change the timezone environment variable in the docker run command with -e TZ=GMT and -e TZ=Africa/Johannesburg, which does reflect the time change in the docker container bash date command, however this doesn't make a difference to the timestamp of emails sent/received, so I'm assuming I can set this somewhere on Citadel but am hoping for a nudge in the right direction as to where since I'm unable to find anything in the web interface logged in as admin or as the mailbox user. 

Thank you

Kind Regards

Andre

It was sent with Thunderbird.

Mon Apr 20 2026 20:44:34 UTC from luisgo Subject: Possible bug.

Dear Suporters,

Try to send to this email (it does not exists. It is distorted from a real one). Use all (name+email)

Albertí Quintá, Xavier <xalberti@indla.fr>

It will returned but see the Contacts folder. It appears 2 new contacts. One wrong and one good. But with the original email (not distorted) appear only the wrong one.

Let me please know if it is a bug or if it is only in my server.  

Thank you,

Luís Gonçalves.

Dear Suporters,

Try to send to this email (it does not exists. It is distorted from a real one). Use all (name+email)

Albertí Quintá, Xavier <xalberti@indla.fr>

It will returned but see the Contacts folder. It appears 2 new contacts. One wrong and one good. But with the original email (not distorted) appear only the wrong one.

Let me please know if it is a bug or if it is only in my server.  

Thank you,

Luís Gonçalves.

Nothing is being done with Internet mail.   Our Citadel install is all within Citadel itself.  The email addresses are being used as an isolated intranet setup.   The issue with contacts is the fact the data entered in the fields of the phone numbers is not saved after clicking save.  The fields stay blank which means WebCit is not interacting correctly with the database.  'And, I want to know how that can be fixed to work appropriately.   The same with the internal Citadel email with server fully qualified domain name.   The field stays blank no matter what is entered.

Some good news here!  I was able to install the Linux citadel-client and able to use the .Admin command set.  Didn't fix any of the other issues but at least this aspect is basically working.

One more thing!  AI said to telnet to port 504 and edit the database that way.  I was able to login and quit.  I was able to GOTO room.   'But any .Admin command returned 503 not recognized despite the fact I was literally signed in as Admin.  

Good evening!   I work as a volunteer for a non profit, under privileged youth organization in southwestern Pennsylvania.   We have a camp coming up and we're planning on using Citadel as a much needed method of collaboration.   Installed Citadel in Debian 13 via the easy install script.  Everything went fine, no errors.   Created users and realized no matter what I type for a phone number or primary email address in contacts nothing saves.  It stays blank.  I Googled possible problems and AI said the install must be corrupted so manually remove the install and rerun setup, which I did!  Still the same issue.   I have to add email in the alias field in contacts for it to show up in the address book.  Still phone numbers don't save.  BUT that's ok!  I just put phone numbers in PO Box field instead and the data is there. Saves fine in that field.   Next, finally got the users setup and the group address book able show email addresses to send emails to instead of just the username which wo ]]>http://uncensored.citadel.org/readfwd?go=Citadel%20Support?start_reading_at=2099559799Wed, 08 Apr 2026 21:29:15 +05002099559799@Uncensored

Hi Mike,

Happy Easter holiday to you too!

Because of holiday, the troubleshooting process had been paused and shall resume again.

The ESXi host had no mac binding arrangements -- I had other VMs running both adapters which can successfully use DHCP and static external IP separately

May try traceroute approach later.

Sun Apr 05 2026 07:26:39 AM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Alan,

first, happy easter to you and your family. 

Is there any MAC boundary to the old Maschine inside ESx ? 

If telnet works on internal lan and not from outside, so must be a routing fail from the firewall or antoher service. Also check for persisten custom routes. 

You also can do a traceroute from inside and from outside. Maybe this gives you some more informations. 

 

Greetings

Mike

 

 

Hi Alan,

first, happy easter to you and your family. 

Is there any MAC boundary to the old Maschine inside ESx ? 

If telnet works on internal lan and not from outside, so must be a routing fail from the firewall or antoher service. Also check for persisten custom routes. 

You also can do a traceroute from inside and from outside. Maybe this gives you some more informations. 

Greetings

Mike

Mi Apr 01 2026 02:09:39 UTC von alanlau Betreff: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Thanks Mike for help.

1. DNS - noted

2. Installation -- inside our own premise on VMWare host.  Host with 2 LAN card, one for LAN and another for WAN.  WAN adapter had external IP independent from LAN.

3. Since new server not working on existing Domain name and IP, it is now running simulatenously together with current server but with different LAN and WAN IP for troubleshooting

4. Telnet -- already test within own LAN as well as externally (from my home and at another office) -- both can successfully telnet port 25 of new server, on both WAN IP (24 & 28 when trying to switch).

 

Existing server setup: Citadel on Ubuntu 18.04 VM on VMWare, WAN IP xxx.xxx.xxx.28.  Server had been running since 2018 and having maintenence issue therefore prepared to be replaced.

New server now prepared: Citadel on Debian 13 VM on VMWare, WAN IP xxx.xxx.xxx.24.  Completely new install and not migration.  Both server on SAME vmware host with same network adapter, both LAN and WAN.

 

Problem: after shutdown existing server, and replace the new server to IP ending with 28, it does not receive external mail nor passing MXTOOLBOX server test.  However when placing back to IP ending with 24, MXTOOLBOX test (using IP) works.  Returning the existing server back online will have everything working fine.

 

Tue Mar 31 2026 10:04:57 AM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Alan,

 

i think, i need some more details about your setup. 

The DNS entrys doenst need to change. 

Your Citadel installation is on premise / selfhosted inside your own network?

If the new server have a different IP Address, maybe it wouldnt work. Do you checked your Firewall ports and port routing ? 

If you Citadel is inside your network, can you telnet from a different computer to your Citadel Server ?

 

For more details about your setup you can mail me if you want. Maybe there  informations not for everyones eyes, but the solution is good for all. 

 

Greetings 

Mike

michael@g00r00.com

 

 

Forget to mention both Ubutnu and Debian had no Firewall enabled.  

External IPs also running without Firewall equipment/UTM until everything had solved.

Thanks Mike for help.

1. DNS - noted

2. Installation -- inside our own premise on VMWare host.  Host with 2 LAN card, one for LAN and another for WAN.  WAN adapter had external IP independent from LAN.

3. Since new server not working on existing Domain name and IP, it is now running simulatenously together with current server but with different LAN and WAN IP for troubleshooting

4. Telnet -- already test within own LAN as well as externally (from my home and at another office) -- both can successfully telnet port 25 of new server, on both WAN IP (24 & 28 when trying to switch).

Existing server setup: Citadel on Ubuntu 18.04 VM on VMWare, WAN IP xxx.xxx.xxx.28.  Server had been running since 2018 and having maintenence issue therefore prepared to be replaced.

New server now prepared: Citadel on Debian 13 VM on VMWare, WAN IP xxx.xxx.xxx.24.  Completely new install and not migration.  Both server on SAME vmware host with same network adapter, both LAN and WAN.

Problem: after shutdown existing server, and replace the new server to IP ending with 28, it does not receive external mail nor passing MXTOOLBOX server test.  However when placing back to IP ending with 24, MXTOOLBOX test (using IP) works.  Returning the existing server back online will have everything working fine.

Tue Mar 31 2026 10:04:57 AM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Alan,

 

i think, i need some more details about your setup. 

The DNS entrys doenst need to change. 

Your Citadel installation is on premise / selfhosted inside your own network?

If the new server have a different IP Address, maybe it wouldnt work. Do you checked your Firewall ports and port routing ? 

If you Citadel is inside your network, can you telnet from a different computer to your Citadel Server ?

 

For more details about your setup you can mail me if you want. Maybe there  informations not for everyones eyes, but the solution is good for all. 

 

Greetings 

Mike

michael@g00r00.com

 

 

Hi Alan,

i think, i need some more details about your setup. 

The DNS entrys doenst need to change. 

Your Citadel installation is on premise / selfhosted inside your own network?

If the new server have a different IP Address, maybe it wouldnt work. Do you checked your Firewall ports and port routing ? 

If you Citadel is inside your network, can you telnet from a different computer to your Citadel Server ?

For more details about your setup you can mail me if you want. Maybe there  informations not for everyones eyes, but the solution is good for all. 

Greetings 

Mike

michael@g00r00.com

Di Mär 31 2026 09:09:38 UTC von alanlau Betreff: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Mike,

My server is running on VMs.  Since my new server is not working last week, I had reinstate the old server VM back to use the original Domain and IP, and I had the new server VM continue running but use another IP.

If running on another IP, it response to MXTOOLBOX mail server test.

If put the new server back to my original IP, it doesn't.

Had used your command to check the common ports  (24/456/587 etc) are all listening by Citadel only.

Will it be any relations to DNS setting that binded to the original old server? Such as MAC address binding?  

 

Mon Mar 30 2026 06:37:27 AM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Alan,

please try to figure out, whats going on.

 

Check Services on Ports:

ss -tulpn

Also you can swtich port inside Citadel to antoher MTA Port and check, if you can Telnet it. Do you import Setting from an old Citadel installation ? 

I checked lates Debian and Ubuntu Version with Citadel and had no problems. 

 

Sorry the previous message typo - ports 25/465/587 

Hi Mike,

My server is running on VMs.  Since my new server is not working last week, I had reinstate the old server VM back to use the original Domain and IP, and I had the new server VM continue running but use another IP.

If running on another IP, it response to MXTOOLBOX mail server test.

If put the new server back to my original IP, it doesn't.

Had used your command to check the common ports  (24/456/587 etc) are all listening by Citadel only.

Will it be any relations to DNS setting that binded to the original old server? Such as MAC address binding?  

Mon Mar 30 2026 06:37:27 AM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Alan,

please try to figure out, whats going on.

 

Check Services on Ports:

ss -tulpn

Also you can swtich port inside Citadel to antoher MTA Port and check, if you can Telnet it. Do you import Setting from an old Citadel installation ? 

I checked lates Debian and Ubuntu Version with Citadel and had no problems. 

 

Hi Alan,

please try to figure out, whats going on.

Check Services on Ports:

ss -tulpn

Also you can swtich port inside Citadel to antoher MTA Port and check, if you can Telnet it. Do you import Setting from an old Citadel installation ? 

I checked lates Debian and Ubuntu Version with Citadel and had no problems. 

My Settings:

Mo Mär 30 2026 01:25:28 UTC von alanlau Betreff: Re: New Citadel Installation not receiving mail nor response to external SMTP check

I just writing here to report that my previous new Debian installation for Citadel wasn't working as expected, with same result that unable to receive external message nor response to MXTOOLBOX server test.

When I first preparing it I used another IP for both internal and external network and it works fine.  However when I change back to my original email domain's IP address it just not responding.

Even though it's not responding to MXTOOLBOX server test, it does response to telnet port 25.  

I had no choice but put the old server back and everything fine again.  I am still investigating the reason....

Great ;)

Problem solved. But i will test it on a new Ubuntu installation and figure out, what the problem is. 

Hve a nice day

Greetings 

Mike

Mi Mär 25 2026 01:32:16 UTC von alanlau Betreff: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Mike,

Successfully tested on Debian 13 and works like a charm.

Therefore I concluded it MUST be ubuntu's problem, or at least its on OS setting I'm not awared of.

I had prepared a native Debian 13 and using native installation by Easyinstall script.

Tue Mar 24 2026 05:38:54 PM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi, its not that different to Ubuntu. Its the same base. 

So i think you dont have any problems to use it. How you install citadel on Ubuntu ? As a Docker Container or nativ ? you need to chown right on Ubuntu. 

The problem is not the OS itself. 

 

But if you can telnet citdael on port 25, it might be not a OS problem. i think its a administrativ right problem for citadel or the service.

you checked the status for citadel service ?

 

systemctl status citadel.service

 

Greetings Mike

 

D

Hi Mike,

Successfully tested on Debian 13 and works like a charm.

Therefore I concluded it MUST be ubuntu's problem, or at least its on OS setting I'm not awared of.

I had prepared a native Debian 13 and using native installation by Easyinstall script.

Tue Mar 24 2026 05:38:54 PM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi, its not that different to Ubuntu. Its the same base. 

So i think you dont have any problems to use it. How you install citadel on Ubuntu ? As a Docker Container or nativ ? you need to chown right on Ubuntu. 

The problem is not the OS itself. 

 

But if you can telnet citdael on port 25, it might be not a OS problem. i think its a administrativ right problem for citadel or the service.

you checked the status for citadel service ?

 

systemctl status citadel.service

 

Greetings Mike

 

D

Hi, its not that different to Ubuntu. Its the same base. 

So i think you dont have any problems to use it. How you install citadel on Ubuntu ? As a Docker Container or nativ ? you need to chown right on Ubuntu. 

The problem is not the OS itself. 

But if you can telnet citdael on port 25, it might be not a OS problem. i think its a administrativ right problem for citadel or the service.

you checked the status for citadel service ?

systemctl status citadel.service

Greetings Mike

Di Mär 24 2026 01:22:45 UTC von alanlau Betreff: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Thanks for suggestion, and it confirmed one of my suspicion: its the OS broke it instead of Citadel  ;>

Since I'm not too familier on Debian I may need some time to test this configuration and will come up again once I am able to test it.

 

Mon Mar 23 2026 08:10:30 AM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Yeah,

i know the Problem. I got such problem with Ubuntu too. I think, the UFW is turned off. I fixed the incoming MTA and use Postfix -> rspamD -> SMTP Citadel 2025. For more compatibility, try a normal Debian installation for a Citadel only installation. 

Oherwise check what the logs say..

tail -f /var/log/syslog 

and trace it to a file to analyse whats going on. 

 

Mike

 

Mo Mär 23 2026 02:32:44 UTC von alanlau Betreff: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Mike,

I used to run the mentioned MX test by MXTOOLBOX and having the result of server not responding.

All DNS records were not changed as with the existing old Citadel server.  Everything works fine on old one, but just not the case if I directly replace a new Citadel with it.

I install and tested the new server on both Ubuntu 22.04 and 24.04, as I wonder it is the OS lead to not responding instead of Citadel.

Any idea on it?

 

 

Thanks for suggestion, and it confirmed one of my suspicion: its the OS broke it instead of Citadel  ;>

Since I'm not too familier on Debian I may need some time to test this configuration and will come up again once I am able to test it.

Mon Mar 23 2026 08:10:30 AM UTC from Motobike Subject: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Yeah,

i know the Problem. I got such problem with Ubuntu too. I think, the UFW is turned off. I fixed the incoming MTA and use Postfix -> rspamD -> SMTP Citadel 2025. For more compatibility, try a normal Debian installation for a Citadel only installation. 

Oherwise check what the logs say..

tail -f /var/log/syslog 

and trace it to a file to analyse whats going on. 

 

Mike

 

Mo Mär 23 2026 02:32:44 UTC von alanlau Betreff: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Mike,

I used to run the mentioned MX test by MXTOOLBOX and having the result of server not responding.

All DNS records were not changed as with the existing old Citadel server.  Everything works fine on old one, but just not the case if I directly replace a new Citadel with it.

I install and tested the new server on both Ubuntu 22.04 and 24.04, as I wonder it is the OS lead to not responding instead of Citadel.

Any idea on it?

 

Yeah,

i know the Problem. I got such problem with Ubuntu too. I think, the UFW is turned off. I fixed the incoming MTA and use Postfix -> rspamD -> SMTP Citadel 2025. For more compatibility, try a normal Debian installation for a Citadel only installation. 

Oherwise check what the logs say..

tail -f /var/log/syslog 

and trace it to a file to analyse whats going on. 

Mike

Mo Mär 23 2026 02:32:44 UTC von alanlau Betreff: Re: New Citadel Installation not receiving mail nor response to external SMTP check

Hi Mike,

I used to run the mentioned MX test by MXTOOLBOX and having the result of server not responding.

All DNS records were not changed as with the existing old Citadel server.  Everything works fine on old one, but just not the case if I directly replace a new Citadel with it.

I install and tested the new server on both Ubuntu 22.04 and 24.04, as I wonder it is the OS lead to not responding instead of Citadel.

Any idea on it?

Hi Mike,

I used to run the mentioned MX test by MXTOOLBOX and having the result of server not responding.

All DNS records were not changed as with the existing old Citadel server.  Everything works fine on old one, but just not the case if I directly replace a new Citadel with it.

I install and tested the new server on both Ubuntu 22.04 and 24.04, as I wonder it is the OS lead to not responding instead of Citadel.

Any idea on it?

Hi, 

you checked the DNS for a A Record and a AAAA Record ?

Some ISP will use AAAA only.

If you can telnet the port 25, it looks fine. Also you can check with mxtoolbox.

Greetings

Mike

Di Mär 17 2026 02:43:59 UTC von alanlau Betreff: New Citadel Installation not receiving mail nor response to external SMTP check

Dear team,

We had been running Citadel since 2008 and now preparing a new Citadel server to replace the old one because of the age and capacity.

We setup and configurared a new VM for Citadel, copied most of the running settings to it, and it works well internally, however it just not responsing to external messages.

That is, it successfully send messages, but not receiving any from outside.

Confirm all necessary ports are opened.

Running MX tool SMTP check on internet shows the server is not responding.

However if we telnet the server IP address, it responses.  

May I ask whether there are missing configuration we needed?

Alan Lau

 

Tue Mar 17 2026 08:51:47 AM UTC from alanlau Subject: Re: Received not sent

Harry are you asking me about your question?

As long as you described, it looks like your account in Citadel was created with "Permission to send internet mail" option checked.

If you are not your Citadel administrator, you will need to found him to open that option for you.

Just my 2 cents

Alan

Sorry I mean "not checked" instead of "checked"

Thanks Alan for the help.

Harry

Tue Mar 17 2026 08:53:54 UTC from alanlau Subject: Re: Received not sent

 

Tue Mar 17 2026 08:51:47 AM UTC from alanlau Subject: Re: Received not sent

Harry are you asking me about your question?

As long as you described, it looks like your account in Citadel was created with "Permission to send internet mail" option checked.

If you are not your Citadel administrator, you will need to found him to open that option for you.

Just my 2 cents

Alan

Sorry I mean "not checked" instead of "checked"

Tue Mar 17 2026 08:51:47 AM UTC from alanlau Subject: Re: Received not sent

Harry are you asking me about your question?

As long as you described, it looks like your account in Citadel was created with "Permission to send internet mail" option checked.

If you are not your Citadel administrator, you will need to found him to open that option for you.

Just my 2 cents

Alan

Sorry I mean "not checked" instead of "checked"

Harry are you asking me about your question?

As long as you described, it looks like your account in Citadel was created with "Permission to send internet mail" option checked.

If you are not your Citadel administrator, you will need to found him to open that option for you.

Just my 2 cents

Alan

Dear team,

We had been running Citadel since 2008 and now preparing a new Citadel server to replace the old one because of the age and capacity.

We setup and configurared a new VM for Citadel, copied most of the running settings to it, and it works well internally, however it just not responsing to external messages.

That is, it successfully send messages, but not receiving any from outside.

Confirm all necessary ports are opened.

Running MX tool SMTP check on internet shows the server is not responding.

However if we telnet the server IP address, it responses.  

May I ask whether there are missing configuration we needed?

Alan Lau

Hi Alan,

Harry received an email from the internet but did not have permission to send a reply. Who can give me permission? What port numbes are is used for Citadel emails in and out?

Thanks for your help.

Harry

Dear team,

We had been running Citadel since 2008 and now preparing a new Citadel server to replace the old one because of the age and capacity.

We setup and configurared a new VM for Citadel, copied most of the running settings to it, and it works well internally, however it just not responsing to external messages.

That is, it successfully send messages, but not receiving any from outside.

Confirm all necessary ports are opened.

Running MX tool SMTP check on internet shows the server is not responding.

However if we telnet the server IP address, it responses.  

May I ask whether there are missing configuration we needed?

Alan Lau

Hi,

User harry could not reply to an email and was shown this message: 

You do not have permission to send Internet mail.

Please let me send email.

Cheers,

Harry

My solution is this:

[Unit]

Description=Citadel WebCit HTTP Redirect

After=citadel.target

[Service]

ExecStart=/usr/local/webcit/webcit \

    -p80 \

    -R https://%H \

    uds /usr/local/citadel

KillMode=process

Restart=always

RestartSec=3

[Install]

WantedBy=multi-user.target

Di Jan 27 2026 20:04:40 UTC von Motobike Betreff: Re: First Room

Hi Ignatius,

do you habe a quick hint for me, how to redirect the webcit http service to https ?

webcit -x1 -p 80 -g ? 

Also i have question to your http to telnet webservice. What Tool are you using ?

 

Citadel is now fully stable running for 1 year. It was the best dession for all my needs :)

o/

Mike

Do Mai 01 2025 09:26:51 UTC von Motobike Betreff: Re: First Room

Hi Ignatius,

but where will it be defined ? First start citadel, it will defined in a config that starts webcit with parameters X. 

 

Do Feb 13 2025 23:38:47 UTC von IGnatius T Foobar Betreff: Re: First Room

What you mean with [other options] ?

WebCit is normally called with options to specify the port number it's listening on, whether to enable HTTPS, and the directory where Citadel Server can be found.

For example, here are the commands we are using to start WebCit here:

webcit -x1 -s -p 443 -g /dotgoto?room='Welcome to UNCENSORED!' uds /citadel-data
webcit -x1 -p 80 -g /dotgoto?room='Welcome to UNCENSORED!' uds /citadel-data

(Most sites will have two, one for HTTP and one for HTTPS, but if you have Citadel sitting behind a proxy or if you have some other webserver redirecting HTTP to HTTPS you might only have one.)

 

 

Hi Ignatius,

do you habe a quick hint for me, how to redirect the webcit http service to https ?

webcit -x1 -p 80 -g ? 

Also i have question to your http to telnet webservice. What Tool are you using ?

Citadel is now fully stable running for 1 year. It was the best dession for all my needs :)

o/

Mike

Do Mai 01 2025 09:26:51 UTC von Motobike Betreff: Re: First Room

Hi Ignatius,

but where will it be defined ? First start citadel, it will defined in a config that starts webcit with parameters X. 

 

Do Feb 13 2025 23:38:47 UTC von IGnatius T Foobar Betreff: Re: First Room

What you mean with [other options] ?

WebCit is normally called with options to specify the port number it's listening on, whether to enable HTTPS, and the directory where Citadel Server can be found.

For example, here are the commands we are using to start WebCit here:

webcit -x1 -s -p 443 -g /dotgoto?room='Welcome to UNCENSORED!' uds /citadel-data
webcit -x1 -p 80 -g /dotgoto?room='Welcome to UNCENSORED!' uds /citadel-data

(Most sites will have two, one for HTTP and one for HTTPS, but if you have Citadel sitting behind a proxy or if you have some other webserver redirecting HTTP to HTTPS you might only have one.)

 

For things like this, 'reaching out' TOR is brain dead easy.  Just fire up the TOR browser. Done.  ( sure there are other ways, but why futz with it? )   For incoming, its not really hard either, but does limit your audience greatly if its the ONLY route in since fewer people will know about you.

And some of us, well, we have our reasons for doing it.

Sat Jan 24 2026 06:19:22 UTC from ParanoidDelusions

I'm insolent. I can use the onion... I understand it - but I want them to know - and I want to know they know, but don't really care, as long as I stay in my lane. It is a lot of extra effort to use TOR, just to make them MORE interested in what you may or may not be doing. 

I am a "security through opacity" kind of guy. If they can see it - without TOR and VPNs - they don't care. If I start making it difficult to track and watch - then they might pay more attention and something they would otherwise miss - might be caught. 

 

I'm insolent. I can use the onion... I understand it - but I want them to know - and I want to know they know, but don't really care, as long as I stay in my lane. It is a lot of extra effort to use TOR, just to make them MORE interested in what you may or may not be doing. 

I am a "security through opacity" kind of guy. If they can see it - without TOR and VPNs - they don't care. If I start making it difficult to track and watch - then they might pay more attention and something they would otherwise miss - might be caught. 

Without it, its not like things are unusable,  just it would speed access up a little and prevent me from having to re-post when it gets eaten. Just me, not worth effort really, but it might increase traffic too. I'm sure  i am not the only person hiding inside the onion... 

Thu Jan 22 2026 23:40:03 UTC from IGnatius T Foobar

If I put Uncensored on a .onion proxy, would that be beneficial to you?

Regular VPNs work fine.  There are a lot of cheap ones, and some free ones too.  Might be enough for you to hide behind? ( not slamming.. just donno how far you have to hide..)

Not that it matters, and i know you cant support incoming via a TOR out-port since its out of your control, but just to add after some testing and paying better attention:

Logging in seems to work 99% of the time. But sometimes i get a 'not found' which i guess does happen sometimes.  Posting, when it does not work, i finally saw the message today relating to not posting an empty post.  Been taking a copy of the post first, so i can easily redo easily if it bombs. Seems like on the 3rd try it works. 

I know its my problem not yours, but still be neat if you added a TOR proxy to go along with the I2P proxy. ( since i have never had any luck getting to the outside via I2P, tho i read its possible, somehow )

If you are looking to run a citadel server but only have Windows I provide Ubuntu hosting starting at $7/month with first month free to try it out... Get more info at https://citadelhosting.retro-os.live.

Marisa

I would like to know, if the software has a windows version?

You could run it in WSL, or in Docker on Windows.

But there's no native Windows version.

I would like to know, if the software has a windows version?

You could run it in WSL, or in Docker on Windows.

But there's no native Windows version.

Did you share the URL? Is Other BBSs> a thing still? 

Don't assume it's a BBS.  The vast majority of Citadel instances are not.

I would like to know, if the software has a windows version?

Did you share the URL? Is Other BBSs> a thing still? 

Sat Jan 03 2026 17:18:59 UTC from Kurisu

Killer - worked a treat.

Thank ya.

So happy to be running a citadel instance again.

Fri Jan 02 2026 17:05:41 UTC from IGnatius T Foobar

 

Dear All,

Exists or not any way to send emails for a certain domain forcing IPv4 and not using IPv6 as normal?

For any other domains remains unchanged.

Thank you,

Luís Gonçalves. 

Killer - worked a treat.

Thank ya.

So happy to be running a citadel instance again.

Fri Jan 02 2026 17:05:41 UTC from IGnatius T Foobar

You want the "-g" flag, which allows you to pre-fill a location of your choice to use as the landing page.

For example, here we use:

webcit [other flags] -g '/dotgoto?room=Welcome to UNCENSORED!' [other arguments]

This works when you're calling WebCit directly from the host, as you would when running the Easy Install distribution, but it also works when you're running the Docker container: the "-g" flag and its argument are passed through from the caller to both WebCit instances (which is how I run it here).

Minor question, what was the flag to have webcit-http and webcit-https display a particular room (as opposed to defaulting to lobby) like Uncensored does with its wiki page? Had that going fine on a previous Citadel instance which has since been retired and I'd like to do that again for this new one but can't find the information anymore.

Thanks!

Right, and has the SPA been built in such a way as to make the back button do something that makes sense? :D

Wed Dec 31 2025 06:26:39 UTC from IGnatius T Foobar

Will the "Back" button on the browser in the next WebCit
actually work properly, or just be psycho like it is now? :-D

WebCit-NG is going to be a whole different ballgame. Since it's a "one page web app" the semantics are different.

I doubt WebCit-classic is going to get any more love in that department.

Will the "Back" button on the browser in the next WebCit actually work properly, or just be psycho like it is now? :-D

For me its not anger, its facepalm.

And sorry, its the wrong room.

Sun Dec 28 2025 00:55:07 UTC from IGnatius T Foobar

I struggle with Christian compassion. A lot. Charlie Kirk would
be disappointed. 

You're not alone. Not by far. It's one of those things we struggle with -- as believers we want to show Christ-like love and compassion, while as humans we want to choke the living shit out of some asshole who desperately deserves it.

Yeah. I struggle with anger sometimes. We aren't all as far along as Charlie was. But we all receive the same measure of grace if we ask for it. And that's pretty darn cool.

Most of us do, its not the 1980s anymore.   ( really in any industry, not just IT )

I have moved to the attitude of "whatever, let it burn" and go grab a bag of marshmallows.

Fri Dec 26 2025 06:45:41 UTC from ParanoidDelusions

You're nicer than me. I deal with highly educated, industry specific idiots who f'k things up every day and it makes us all pay more. 

You're nicer than me. I deal with highly educated, industry specific idiots who f'k things up every day and it makes us all pay more. I feel like we should be mean those people. 

I struggle with Christian compassion. A lot. Charlie Kirk would be disappointed. 

Thu Dec 25 2025 14:11:44 UTC from Nurb432

It happens sometimes. Newbies don't know. Often times IG will move it. 

 

Right room. How can developer people be so stupid? I can't code - but I'm not dumb enough to dump this in the lobby. 

Maybe it is me. 

It happens sometimes. Newbies don't know. Often times IG will move it. 

Thu Dec 25 2025 07:40:44 UTC from ParanoidDelusions

Well, I feel this wasn't the right room for THAT. 

Glad it didn't somehow crash Uncensored. 

Anyhow - Merry Christmas, and goodnight. ;) 

 

Well, I feel this wasn't the right room for THAT. 

Glad it didn't somehow crash Uncensored. 

Anyhow - Merry Christmas, and goodnight. ;) 

Hello Citadel Team,

I'm reporting a reproducible segmentation fault (SIGSEGV) observed in production, which appears to be caused by a concurrency / thread-safety issue in Citadel under normal usage conditions. I have collected multiple core dumps showing consistent call patterns and triggers.

Deployment: Docker

OS: Linux x86_64 (glibc-based)

Protocols in use:

* IMAPS

* SMTP (inbound + outbound)

* SMTP queue delivery

Sieve: Enabled (email forwarding rules)

No custom patches. No abnormal traffic patterns.

Observed Behaviour

Citadel crashes with SIGSEGV during normal operation. The crashes began after ~26 weeks of stable, uninterrupted operation, without any infrastructure or configuration changes. The only relevant behavioural change before the first crash was the introduction of new Sieve forwarding rules, which increased outbound SMTP activity.

From Core Dumps

The crashes consistently occur when the following operations overlap in time:

Thread A – IMAP FETCH (user reading mail)

imap_fetch_body()

imap_do_fetch_msg()

imap_do_fetch()

imap_command_loop()

Thread B – SMTP DATA (receiving mail)

smtp_data()

smtp_command_loop()

CtdlReadMessageBodyBuf()

StrBufTCP_read_buffered_line_fast()

Thread C – SMTP Forwarding / Queue Delivery

smtp_attempt_delivery()

smtp_do_queue()

smtp_do_queue_full()

curl_easy_perform()

curl_multi_poll()

Additional Concurrent Activity

* TLS handshakes (CtdlStartTLS, imaps_greeting, pop3s_greeting)

* Session cleanup and hooks (PerformSessionHooks, do_housekeeping)

* In some crashes, XMPP hooks are also active

Analysis / Suspected Root Cause

Based on repeated core dump analysis:

• These execution paths share internal buffers/message structures

• Some of these paths appear not to be fully thread-safe

• Increased outbound SMTP concurrency (introduced by Sieve forwarding) significantly raises the likelihood of these paths overlapping

• This results in memory corruption, eventually causing a SIGSEGV (often observed shortly after read(), select(), or buffer handling)

Important notes:

• The Sieve rules themselves are valid and correctly defined

• The issue is not caused by user error or misconfiguration

• Forwarding acts as a trigger, not the root defect

Reproduction Pattern (High-Level)

While not 100% deterministic, the crash frequency increases significantly when:

1. Sieve forwarding is enabled for active mailboxes

2. Inbound SMTP traffic is present

3. Users fetch messages (IMAP FETCH) concurrently

4. SMTP queue delivery runs in parallel (especially to external domains)

Reducing worker thread concurrency significantly reduces or eliminates crashes, which further suggests a concurrency-related issue.

Expected Behaviour

Citadel should be able to safely handle concurrent:

• IMAP FETCH

• SMTP DATA

• SMTP forwarding (Sieve)

• SMTP queue delivery

without memory corruption or segmentation faults.

Questions for the Development Team

1. Is this a known concurrency issue?

2. Has this behaviour been addressed in newer Citadel versions?

3. Are there recommended patches, configuration flags, or build options to mitigate this safely?

4. Would additional debugging data (full core dumps, symbols, logs) be helpful?

Thank you for your time and for maintaining Citadel.

root@tpc-co-cit02:/# gdb /usr/local/citadel/citserver /citadel-data/server1-backup/core.7 

GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git

Copyright (C) 2021 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<https://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/local/citadel/citserver...

(No debugging symbols found in /usr/local/citadel/citserver)

[New LWP 69]

[New LWP 18]

[New LWP 116]

[New LWP 113]

[New LWP 7]

[New LWP 105]

[New LWP 15]

[New LWP 92]

[New LWP 120]

[New LWP 10]

[New LWP 67]

[New LWP 597]

[New LWP 11]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `citserver -x9 -h /citadel-data'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x0000561e86ca5f3f in imap_fetch_body ()

[Current thread is 1 (Thread 0x7f280bfff700 (LWP 69))]

(gdb) set pagination off

(gdb) thread apply all bt full

Thread 13 (Thread 0x7f28225c7700 (LWP 11)):

#0  0x00007f2825d27a0f in __GI___poll (fds=0x7f28225bf870, nfds=2, timeout=1000) at ../sysdeps/unix/sysv/linux/poll.c:29

        __arg2 = <optimized out>

        _a3 = 1000

        _a1 = 139810351872112

        resultvar = <optimized out>

        __arg3 = <optimized out>

        __arg1 = <optimized out>

        _a2 = 2

        resultvar = <optimized out>

        sc_cancel_oldtype = <optimized out>

        sc_ret = <optimized out>

#1  0x00007f2825e75bca in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4

No symbol table info available.

#2  0x00007f2825e6abcc in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4

No symbol table info available.

#3  0x00007f2825e6ad56 in curl_multi_poll () from /usr/lib/x86_64-linux-gnu/libcurl.so.4

No symbol table info available.

#4  0x00007f2825e439cf in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl.so.4

No symbol table info available.

#5  0x0000561e86cc32c5 in smtp_attempt_delivery ()

No symbol table info available.

#6  0x0000561e86cc38f2 in smtp_process_one_msg ()

No symbol table info available.

#7  0x0000561e86cc3f33 in smtp_do_queue ()

No symbol table info available.

#8  0x0000561e86cc3fbf in smtp_do_queue_full ()

No symbol table info available.

#9  0x0000561e86c87e13 in PerformSessionHooks ()

No symbol table info available.

#10 0x0000561e86c73dd5 in do_housekeeping ()

No symbol table info available.

#11 0x0000561e86c8a952 in worker_thread ()

No symbol table info available.

#12 0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810351904512, -5926168278360471484, 140734826778382, 140734826778383, 139810351902080, 8396800, 6021948088529373252, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#13 0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#14 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 12 (Thread 0x7f2808ff9700 (LWP 597)):

#0  0x00007f2825d29ec3 in __GI___select (nfds=59, readfds=0x7f2808ff8bf0, writefds=0x0, exceptfds=0x0, timeout=0x7f2808ff8be0) at ../sysdeps/unix/sysv/linux/select.c:41

        __arg3 = 0

        _a5 = 139809926384608

        _a2 = 139809926384624

        __arg4 = 0

        __arg1 = 59

        _a3 = 0

        resultvar = <optimized out>

        __arg5 = 139809926384608

        __arg2 = 139809926384624

        _a4 = 0

        _a1 = 59

        resultvar = <optimized out>

        sc_cancel_oldtype = -514

        sc_ret = <optimized out>

#1  0x0000561e86c8a447 in worker_thread ()

No symbol table info available.

#2  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809926387456, -5926168278360471484, 140734826778382, 140734826778383, 139809926385024, 139809977318912, 6021898356029311044, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#3  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#4  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 11 (Thread 0x7f2820dbc700 (LWP 67)):

#0  __libc_read (nbytes=8191, buf=0x7f2810130b10, fd=33) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=33, buf=0x7f2810130b10, nbytes=8191) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f2825fffed6 in StrBufTCP_read_buffered_line_fast (Line=0x7f28100fe740, IOBuf=0x7f28000dfb30, Pos=0x7f28000a36e0, fd=0x7f28000a3728, timeout=5, selectresolution=<optimized out>, Error=0x7f2820dba9a0) at lib/stringbuf.c:4317

        pche = <optimized out>

        pos = <optimized out>

        pLF = 0x0

        len = <optimized out>

        rlen = <optimized out>

        retlen = 19

        nSuccessLess = 0

        rfds = {fds_bits = {0, 524288, 126, 32783, 524336, 8192, 35390530551809, 541165879534, 128, 579820586075, 1024, 16, 131072, 4096, 524288, 978326477955633920}}

        pch = <optimized out>

        fdflags = <optimized out>

        IsNonBlock = 0

        tv = {tv_sec = 32767, tv_usec = 7}

#3  0x0000561e86c89cbb in CtdlClientGetLine ()

No symbol table info available.

#4  0x0000561e86c81944 in CtdlReadMessageBodyBuf ()

No symbol table info available.

#5  0x0000561e86cc1847 in smtp_data ()

No symbol table info available.

#6  0x0000561e86cc2227 in smtp_command_loop ()

No symbol table info available.

#7  0x0000561e86c8a8a7 in worker_thread ()

No symbol table info available.

#8  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810326693632, -5926168278360471484, 140734826778382, 140734826778383, 139810326691200, 8396800, 6021951441288218692, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#9  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#10 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 10 (Thread 0x7f2822dc8700 (LWP 10)):

#0  0x00007f2825d29ec3 in __GI___select (nfds=59, readfds=0x7f2822dc7bf0, writefds=0x0, exceptfds=0x0, timeout=0x7f2822dc7be0) at ../sysdeps/unix/sysv/linux/select.c:41

        __arg3 = 0

        _a5 = 139810360294368

        _a2 = 139810360294384

        __arg4 = 0

        __arg1 = 59

        _a3 = 0

        resultvar = <optimized out>

        __arg5 = 139810360294368

        __arg2 = 139810360294384

        _a4 = 0

        _a1 = 59

        resultvar = <optimized out>

        sc_cancel_oldtype = -514

        sc_ret = <optimized out>

#1  0x0000561e86c8a447 in worker_thread ()

No symbol table info available.

#2  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810360297216, -5926168278360471484, 140734826778382, 140734826778383, 139810360294784, 8396800, 6021946989554616388, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#3  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#4  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 9 (Thread 0x7f28097fa700 (LWP 120)):

#0  0x00007f2825d29ec3 in __GI___select (nfds=59, readfds=0x7f28097f9bf0, writefds=0x0, exceptfds=0x0, timeout=0x7f28097f9be0) at ../sysdeps/unix/sysv/linux/select.c:41

        __arg3 = 0

        _a5 = 139809934777312

        _a2 = 139809934777328

        __arg4 = 0

        __arg1 = 59

        _a3 = 0

        resultvar = <optimized out>

        __arg5 = 139809934777312

        __arg2 = 139809934777328

        _a4 = 0

        _a1 = 59

        resultvar = <optimized out>

        sc_cancel_oldtype = -514

        sc_ret = <optimized out>

#1  0x0000561e86c8a447 in worker_thread ()

No symbol table info available.

#2  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809934780160, -5926168278360471484, 140734826778382, 140734826778383, 139809934777728, 139810111629120, 6021901655101065284, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#3  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#4  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 8 (Thread 0x7f280b7fe700 (LWP 92)):

#0  0x00007f2825d29ec3 in __GI___select (nfds=59, readfds=0x7f280b7fdbf0, writefds=0x0, exceptfds=0x0, timeout=0x7f280b7fdbe0) at ../sysdeps/unix/sysv/linux/select.c:41

        __arg3 = 0

        _a5 = 139809968348128

        _a2 = 139809968348144

        __arg4 = 0

        __arg1 = 59

        _a3 = 0

        resultvar = <optimized out>

        __arg5 = 139809968348128

        __arg2 = 139809968348144

        _a4 = 0

        _a1 = 59

        resultvar = <optimized out>

        sc_cancel_oldtype = -514

        sc_ret = <optimized out>

#1  0x0000561e86c8a447 in worker_thread ()

No symbol table info available.

#2  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809968350976, -5926168278360471484, 140734826778382, 140734826778383, 139809968348544, 139810110988112, 6021897259202037828, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#3  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#4  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 7 (Thread 0x7f2821dbe700 (LWP 15)):

#0  __libc_read (nbytes=63, buf=0x7f27e4122af0, fd=57) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=57, buf=0x7f27e4122af0, nbytes=63) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f2825fffed6 in StrBufTCP_read_buffered_line_fast (Line=0x7f2818078d50, IOBuf=0x7f27ec0fc8b0, Pos=0x7f27e4079ec0, fd=0x7f27e4079f08, timeout=5, selectresolution=<optimized out>, Error=0x7f2821dbc9a0) at lib/stringbuf.c:4317

        pche = <optimized out>

        pos = <optimized out>

        pLF = 0x0

        len = <optimized out>

        rlen = <optimized out>

        retlen = 0

        nSuccessLess = 0

        rfds = {fds_bits = {1, 139810178072704, 511, 7, 0, 8192, 139809775419499, 527, 8240, 128, 755914244609, 532575944816, 2, 511101108331, 16, 0}}

        pch = <optimized out>

        fdflags = <optimized out>

        IsNonBlock = 0

        tv = {tv_sec = 139810343471552, tv_usec = 139810409532778}

#3  0x0000561e86c89cbb in CtdlClientGetLine ()

No symbol table info available.

#4  0x0000561e86c81944 in CtdlReadMessageBodyBuf ()

No symbol table info available.

#5  0x0000561e86cc1847 in smtp_data ()

No symbol table info available.

#6  0x0000561e86cc2227 in smtp_command_loop ()

No symbol table info available.

#7  0x0000561e86c8a8a7 in worker_thread ()

No symbol table info available.

#8  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810343479040, -5926168278360471484, 140734826778382, 140734826778383, 139810343476608, 139810111003408, 6021953643532699716, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#9  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#10 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 6 (Thread 0x7f280affd700 (LWP 105)):

#0  __libc_read (nbytes=8191, buf=0x7f280018baf0, fd=56) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=56, buf=0x7f280018baf0, nbytes=8191) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f2825fffed6 in StrBufTCP_read_buffered_line_fast (Line=0x7f2800053fc0, IOBuf=0x7f281c090d00, Pos=0x7f281c0f7440, fd=0x7f281c0f7488, timeout=5, selectresolution=<optimized out>, Error=0x7f280affb9a0) at lib/stringbuf.c:4317

        pche = <optimized out>

        pos = <optimized out>

        pLF = 0x0

        len = <optimized out>

        rlen = <optimized out>

        retlen = 20

        nSuccessLess = 0

        rfds = {fds_bits = {0, 262144, 125, 16399, 262192, 4096, 17798344491009, 139810409524331, 64, 545460847195, 512, 8, 131072, 4096, 262144, 978326477955633920}}

        pch = <optimized out>

        fdflags = <optimized out>

        IsNonBlock = 0

        tv = {tv_sec = 16383, tv_usec = 7}

#3  0x0000561e86c89cbb in CtdlClientGetLine ()

No symbol table info available.

#4  0x0000561e86c81944 in CtdlReadMessageBodyBuf ()

No symbol table info available.

#5  0x0000561e86cc1847 in smtp_data ()

No symbol table info available.

#6  0x0000561e86cc2227 in smtp_command_loop ()

No symbol table info available.

#7  0x0000561e86c8a8a7 in worker_thread ()

No symbol table info available.

#8  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809959958272, -5926168278360471484, 140734826778382, 140734826778383, 139809959955840, 139810044056768, 6021893960130283588, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#9  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#10 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 5 (Thread 0x7f2822f2d980 (LWP 7)):

#0  0x00007f2825cfa241 in __GI___clock_nanosleep (clock_id=0, flags=0, req=0x7fff615b8fd0, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48

        sc_cancel_oldtype = <optimized out>

        sc_ret = <optimized out>

        r = <optimized out>

#1  0x00007f2825cffa23 in __GI___nanosleep (requested_time=0x0, remaining=0x0) at nanosleep.c:27

        ret = <optimized out>

#2  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 4 (Thread 0x7f280a7fc700 (LWP 113)):

#0  __libc_read (nbytes=5, buf=0x7f27f403a183, fd=47) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=47, buf=0x7f27f403a183, nbytes=5) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f28260f35e5 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#3  0x00007f28260ee7fa in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#4  0x00007f28260ed6a4 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#5  0x00007f28260edc73 in BIO_read () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#6  0x00007f2826346771 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#7  0x00007f282634a982 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#8  0x00007f28263482c0 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#9  0x00007f282637b499 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#10 0x00007f2826370f75 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#11 0x0000561e86c9667f in CtdlStartTLS ()

No symbol table info available.

#12 0x0000561e86c8806f in CtdlModuleStartCryptoMsgs ()

No symbol table info available.

#13 0x0000561e86caf4d1 in imaps_greeting ()

No symbol table info available.

#14 0x0000561e86c8a88a in worker_thread ()

No symbol table info available.

#15 0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809951565568, -5926168278360471484, 140734826778382, 140734826778383, 139809951563136, 139810044037136, 6021895056957556804, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#16 0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#17 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 3 (Thread 0x7f2809ffb700 (LWP 116)):

#0  __libc_read (nbytes=5, buf=0x7f27ec0d8873, fd=60) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=60, buf=0x7f27ec0d8873, nbytes=5) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f28260f35e5 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#3  0x00007f28260ee7fa in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#4  0x00007f28260ed6a4 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#5  0x00007f28260edc73 in BIO_read () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#6  0x00007f2826346771 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#7  0x00007f282634a982 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#8  0x00007f28263482c0 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#9  0x00007f282637b499 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#10 0x00007f2826370f75 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#11 0x0000561e86c9667f in CtdlStartTLS ()

No symbol table info available.

#12 0x0000561e86c8806f in CtdlModuleStartCryptoMsgs ()

No symbol table info available.

#13 0x0000561e86cb9de7 in pop3s_greeting ()

No symbol table info available.

#14 0x0000561e86c8a88a in worker_thread ()

No symbol table info available.

#15 0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809943172864, -5926168278360471484, 140734826778382, 140734826778383, 139809943170432, 139810111553776, 6021900553978824772, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#16 0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#17 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 2 (Thread 0x7f28215bd700 (LWP 18)):

#0  0x0000561e86ccdd8a in xmpp_queue_event ()

No symbol table info available.

#1  0x0000561e86ccc711 in xmpp_logout_hook ()

No symbol table info available.

#2  0x0000561e86c87e13 in PerformSessionHooks ()

No symbol table info available.

#3  0x0000561e86c8bf2e in CtdlUserLogout ()

No symbol table info available.

#4  0x0000561e86c6fe29 in RemoveContext ()

No symbol table info available.

#5  0x0000561e86c70943 in dead_session_purge ()

No symbol table info available.

#6  0x0000561e86c8a94d in worker_thread ()

No symbol table info available.

#7  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810335086336, -5926168278360471484, 140734826778382, 140734826778383, 139810335083904, 8396800, 6021954742507456580, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#8  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#9  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 1 (Thread 0x7f280bfff700 (LWP 69)):

#0  0x0000561e86ca5f3f in imap_fetch_body ()

No symbol table info available.

#1  0x0000561e86ca6cbd in imap_do_fetch_msg ()

No symbol table info available.

#2  0x0000561e86ca6fda in imap_do_fetch ()

No symbol table info available.

#3  0x0000561e86ca7a77 in imap_fetch ()

No symbol table info available.

#4  0x0000561e86cb1c0b in imap_command_loop ()

No symbol table info available.

#5  0x0000561e86c8a8a7 in worker_thread ()

No symbol table info available.

#6  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809976743680, -5926168278360471484, 140734826778382, 140734826778383, 139809976741248, 8396800, 6021896158079797316, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#7  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#8  0x0000000000000000 in ?? ()

No symbol table info available.

(gdb) 

Hello Citadel Team,

I'm reporting a reproducible segmentation fault (SIGSEGV) observed in production, which appears to be caused by a concurrency / thread-safety issue in Citadel under normal usage conditions. I have collected multiple core dumps showing consistent call patterns and triggers.

Deployment: Docker

OS: Linux x86_64 (glibc-based)

Protocols in use:

* IMAPS

* SMTP (inbound + outbound)

* SMTP queue delivery

Sieve: Enabled (email forwarding rules)

No custom patches. No abnormal traffic patterns.

Observed Behaviour

Citadel crashes with SIGSEGV during normal operation. The crashes began after ~26 weeks of stable, uninterrupted operation, without any infrastructure or configuration changes. The only relevant behavioural change before the first crash was the introduction of new Sieve forwarding rules, which increased outbound SMTP activity.

From Core Dumps

The crashes consistently occur when the following operations overlap in time:

Thread A – IMAP FETCH (user reading mail)

imap_fetch_body()

imap_do_fetch_msg()

imap_do_fetch()

imap_command_loop()

Thread B – SMTP DATA (receiving mail)

smtp_data()

smtp_command_loop()

CtdlReadMessageBodyBuf()

StrBufTCP_read_buffered_line_fast()

Thread C – SMTP Forwarding / Queue Delivery

smtp_attempt_delivery()

smtp_do_queue()

smtp_do_queue_full()

curl_easy_perform()

curl_multi_poll()

Additional Concurrent Activity

* TLS handshakes (CtdlStartTLS, imaps_greeting, pop3s_greeting)

* Session cleanup and hooks (PerformSessionHooks, do_housekeeping)

* In some crashes, XMPP hooks are also active

Analysis / Suspected Root Cause

Based on repeated core dump analysis:

• These execution paths share internal buffers/message structures

• Some of these paths appear not to be fully thread-safe

• Increased outbound SMTP concurrency (introduced by Sieve forwarding) significantly raises the likelihood of these paths overlapping

• This results in memory corruption, eventually causing a SIGSEGV (often observed shortly after read(), select(), or buffer handling)

Important notes:

• The Sieve rules themselves are valid and correctly defined

• The issue is not caused by user error or misconfiguration

• Forwarding acts as a trigger, not the root defect

Reproduction Pattern (High-Level)

While not 100% deterministic, the crash frequency increases significantly when:

1. Sieve forwarding is enabled for active mailboxes

2. Inbound SMTP traffic is present

3. Users fetch messages (IMAP FETCH) concurrently

4. SMTP queue delivery runs in parallel (especially to external domains)

Reducing worker thread concurrency significantly reduces or eliminates crashes, which further suggests a concurrency-related issue.

Expected Behaviour

Citadel should be able to safely handle concurrent:

• IMAP FETCH

• SMTP DATA

• SMTP forwarding (Sieve)

• SMTP queue delivery

without memory corruption or segmentation faults.

Questions for the Development Team

1. Is this a known concurrency issue?

2. Has this behaviour been addressed in newer Citadel versions?

3. Are there recommended patches, configuration flags, or build options to mitigate this safely?

4. Would additional debugging data (full core dumps, symbols, logs) be helpful?

Thank you for your time and for maintaining Citadel.

root@tpc-co-cit02:/# gdb /usr/local/citadel/citserver /citadel-data/server1-backup/core.7 

GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git

Copyright (C) 2021 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<https://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/local/citadel/citserver...

(No debugging symbols found in /usr/local/citadel/citserver)

[New LWP 69]

[New LWP 18]

[New LWP 116]

[New LWP 113]

[New LWP 7]

[New LWP 105]

[New LWP 15]

[New LWP 92]

[New LWP 120]

[New LWP 10]

[New LWP 67]

[New LWP 597]

[New LWP 11]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `citserver -x9 -h /citadel-data'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x0000561e86ca5f3f in imap_fetch_body ()

[Current thread is 1 (Thread 0x7f280bfff700 (LWP 69))]

(gdb) set pagination off

(gdb) thread apply all bt full

Thread 13 (Thread 0x7f28225c7700 (LWP 11)):

#0  0x00007f2825d27a0f in __GI___poll (fds=0x7f28225bf870, nfds=2, timeout=1000) at ../sysdeps/unix/sysv/linux/poll.c:29

        __arg2 = <optimized out>

        _a3 = 1000

        _a1 = 139810351872112

        resultvar = <optimized out>

        __arg3 = <optimized out>

        __arg1 = <optimized out>

        _a2 = 2

        resultvar = <optimized out>

        sc_cancel_oldtype = <optimized out>

        sc_ret = <optimized out>

#1  0x00007f2825e75bca in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4

No symbol table info available.

#2  0x00007f2825e6abcc in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4

No symbol table info available.

#3  0x00007f2825e6ad56 in curl_multi_poll () from /usr/lib/x86_64-linux-gnu/libcurl.so.4

No symbol table info available.

#4  0x00007f2825e439cf in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl.so.4

No symbol table info available.

#5  0x0000561e86cc32c5 in smtp_attempt_delivery ()

No symbol table info available.

#6  0x0000561e86cc38f2 in smtp_process_one_msg ()

No symbol table info available.

#7  0x0000561e86cc3f33 in smtp_do_queue ()

No symbol table info available.

#8  0x0000561e86cc3fbf in smtp_do_queue_full ()

No symbol table info available.

#9  0x0000561e86c87e13 in PerformSessionHooks ()

No symbol table info available.

#10 0x0000561e86c73dd5 in do_housekeeping ()

No symbol table info available.

#11 0x0000561e86c8a952 in worker_thread ()

No symbol table info available.

#12 0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810351904512, -5926168278360471484, 140734826778382, 140734826778383, 139810351902080, 8396800, 6021948088529373252, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#13 0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#14 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 12 (Thread 0x7f2808ff9700 (LWP 597)):

#0  0x00007f2825d29ec3 in __GI___select (nfds=59, readfds=0x7f2808ff8bf0, writefds=0x0, exceptfds=0x0, timeout=0x7f2808ff8be0) at ../sysdeps/unix/sysv/linux/select.c:41

        __arg3 = 0

        _a5 = 139809926384608

        _a2 = 139809926384624

        __arg4 = 0

        __arg1 = 59

        _a3 = 0

        resultvar = <optimized out>

        __arg5 = 139809926384608

        __arg2 = 139809926384624

        _a4 = 0

        _a1 = 59

        resultvar = <optimized out>

        sc_cancel_oldtype = -514

        sc_ret = <optimized out>

#1  0x0000561e86c8a447 in worker_thread ()

No symbol table info available.

#2  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809926387456, -5926168278360471484, 140734826778382, 140734826778383, 139809926385024, 139809977318912, 6021898356029311044, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#3  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#4  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 11 (Thread 0x7f2820dbc700 (LWP 67)):

#0  __libc_read (nbytes=8191, buf=0x7f2810130b10, fd=33) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=33, buf=0x7f2810130b10, nbytes=8191) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f2825fffed6 in StrBufTCP_read_buffered_line_fast (Line=0x7f28100fe740, IOBuf=0x7f28000dfb30, Pos=0x7f28000a36e0, fd=0x7f28000a3728, timeout=5, selectresolution=<optimized out>, Error=0x7f2820dba9a0) at lib/stringbuf.c:4317

        pche = <optimized out>

        pos = <optimized out>

        pLF = 0x0

        len = <optimized out>

        rlen = <optimized out>

        retlen = 19

        nSuccessLess = 0

        rfds = {fds_bits = {0, 524288, 126, 32783, 524336, 8192, 35390530551809, 541165879534, 128, 579820586075, 1024, 16, 131072, 4096, 524288, 978326477955633920}}

        pch = <optimized out>

        fdflags = <optimized out>

        IsNonBlock = 0

        tv = {tv_sec = 32767, tv_usec = 7}

#3  0x0000561e86c89cbb in CtdlClientGetLine ()

No symbol table info available.

#4  0x0000561e86c81944 in CtdlReadMessageBodyBuf ()

No symbol table info available.

#5  0x0000561e86cc1847 in smtp_data ()

No symbol table info available.

#6  0x0000561e86cc2227 in smtp_command_loop ()

No symbol table info available.

#7  0x0000561e86c8a8a7 in worker_thread ()

No symbol table info available.

#8  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810326693632, -5926168278360471484, 140734826778382, 140734826778383, 139810326691200, 8396800, 6021951441288218692, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#9  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#10 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 10 (Thread 0x7f2822dc8700 (LWP 10)):

#0  0x00007f2825d29ec3 in __GI___select (nfds=59, readfds=0x7f2822dc7bf0, writefds=0x0, exceptfds=0x0, timeout=0x7f2822dc7be0) at ../sysdeps/unix/sysv/linux/select.c:41

        __arg3 = 0

        _a5 = 139810360294368

        _a2 = 139810360294384

        __arg4 = 0

        __arg1 = 59

        _a3 = 0

        resultvar = <optimized out>

        __arg5 = 139810360294368

        __arg2 = 139810360294384

        _a4 = 0

        _a1 = 59

        resultvar = <optimized out>

        sc_cancel_oldtype = -514

        sc_ret = <optimized out>

#1  0x0000561e86c8a447 in worker_thread ()

No symbol table info available.

#2  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810360297216, -5926168278360471484, 140734826778382, 140734826778383, 139810360294784, 8396800, 6021946989554616388, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#3  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#4  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 9 (Thread 0x7f28097fa700 (LWP 120)):

#0  0x00007f2825d29ec3 in __GI___select (nfds=59, readfds=0x7f28097f9bf0, writefds=0x0, exceptfds=0x0, timeout=0x7f28097f9be0) at ../sysdeps/unix/sysv/linux/select.c:41

        __arg3 = 0

        _a5 = 139809934777312

        _a2 = 139809934777328

        __arg4 = 0

        __arg1 = 59

        _a3 = 0

        resultvar = <optimized out>

        __arg5 = 139809934777312

        __arg2 = 139809934777328

        _a4 = 0

        _a1 = 59

        resultvar = <optimized out>

        sc_cancel_oldtype = -514

        sc_ret = <optimized out>

#1  0x0000561e86c8a447 in worker_thread ()

No symbol table info available.

#2  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809934780160, -5926168278360471484, 140734826778382, 140734826778383, 139809934777728, 139810111629120, 6021901655101065284, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#3  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#4  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 8 (Thread 0x7f280b7fe700 (LWP 92)):

#0  0x00007f2825d29ec3 in __GI___select (nfds=59, readfds=0x7f280b7fdbf0, writefds=0x0, exceptfds=0x0, timeout=0x7f280b7fdbe0) at ../sysdeps/unix/sysv/linux/select.c:41

        __arg3 = 0

        _a5 = 139809968348128

        _a2 = 139809968348144

        __arg4 = 0

        __arg1 = 59

        _a3 = 0

        resultvar = <optimized out>

        __arg5 = 139809968348128

        __arg2 = 139809968348144

        _a4 = 0

        _a1 = 59

        resultvar = <optimized out>

        sc_cancel_oldtype = -514

        sc_ret = <optimized out>

#1  0x0000561e86c8a447 in worker_thread ()

No symbol table info available.

#2  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809968350976, -5926168278360471484, 140734826778382, 140734826778383, 139809968348544, 139810110988112, 6021897259202037828, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#3  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#4  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 7 (Thread 0x7f2821dbe700 (LWP 15)):

#0  __libc_read (nbytes=63, buf=0x7f27e4122af0, fd=57) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=57, buf=0x7f27e4122af0, nbytes=63) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f2825fffed6 in StrBufTCP_read_buffered_line_fast (Line=0x7f2818078d50, IOBuf=0x7f27ec0fc8b0, Pos=0x7f27e4079ec0, fd=0x7f27e4079f08, timeout=5, selectresolution=<optimized out>, Error=0x7f2821dbc9a0) at lib/stringbuf.c:4317

        pche = <optimized out>

        pos = <optimized out>

        pLF = 0x0

        len = <optimized out>

        rlen = <optimized out>

        retlen = 0

        nSuccessLess = 0

        rfds = {fds_bits = {1, 139810178072704, 511, 7, 0, 8192, 139809775419499, 527, 8240, 128, 755914244609, 532575944816, 2, 511101108331, 16, 0}}

        pch = <optimized out>

        fdflags = <optimized out>

        IsNonBlock = 0

        tv = {tv_sec = 139810343471552, tv_usec = 139810409532778}

#3  0x0000561e86c89cbb in CtdlClientGetLine ()

No symbol table info available.

#4  0x0000561e86c81944 in CtdlReadMessageBodyBuf ()

No symbol table info available.

#5  0x0000561e86cc1847 in smtp_data ()

No symbol table info available.

#6  0x0000561e86cc2227 in smtp_command_loop ()

No symbol table info available.

#7  0x0000561e86c8a8a7 in worker_thread ()

No symbol table info available.

#8  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810343479040, -5926168278360471484, 140734826778382, 140734826778383, 139810343476608, 139810111003408, 6021953643532699716, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#9  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#10 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 6 (Thread 0x7f280affd700 (LWP 105)):

#0  __libc_read (nbytes=8191, buf=0x7f280018baf0, fd=56) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=56, buf=0x7f280018baf0, nbytes=8191) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f2825fffed6 in StrBufTCP_read_buffered_line_fast (Line=0x7f2800053fc0, IOBuf=0x7f281c090d00, Pos=0x7f281c0f7440, fd=0x7f281c0f7488, timeout=5, selectresolution=<optimized out>, Error=0x7f280affb9a0) at lib/stringbuf.c:4317

        pche = <optimized out>

        pos = <optimized out>

        pLF = 0x0

        len = <optimized out>

        rlen = <optimized out>

        retlen = 20

        nSuccessLess = 0

        rfds = {fds_bits = {0, 262144, 125, 16399, 262192, 4096, 17798344491009, 139810409524331, 64, 545460847195, 512, 8, 131072, 4096, 262144, 978326477955633920}}

        pch = <optimized out>

        fdflags = <optimized out>

        IsNonBlock = 0

        tv = {tv_sec = 16383, tv_usec = 7}

#3  0x0000561e86c89cbb in CtdlClientGetLine ()

No symbol table info available.

#4  0x0000561e86c81944 in CtdlReadMessageBodyBuf ()

No symbol table info available.

#5  0x0000561e86cc1847 in smtp_data ()

No symbol table info available.

#6  0x0000561e86cc2227 in smtp_command_loop ()

No symbol table info available.

#7  0x0000561e86c8a8a7 in worker_thread ()

No symbol table info available.

#8  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809959958272, -5926168278360471484, 140734826778382, 140734826778383, 139809959955840, 139810044056768, 6021893960130283588, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#9  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#10 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 5 (Thread 0x7f2822f2d980 (LWP 7)):

#0  0x00007f2825cfa241 in __GI___clock_nanosleep (clock_id=0, flags=0, req=0x7fff615b8fd0, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48

        sc_cancel_oldtype = <optimized out>

        sc_ret = <optimized out>

        r = <optimized out>

#1  0x00007f2825cffa23 in __GI___nanosleep (requested_time=0x0, remaining=0x0) at nanosleep.c:27

        ret = <optimized out>

#2  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 4 (Thread 0x7f280a7fc700 (LWP 113)):

#0  __libc_read (nbytes=5, buf=0x7f27f403a183, fd=47) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=47, buf=0x7f27f403a183, nbytes=5) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f28260f35e5 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#3  0x00007f28260ee7fa in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#4  0x00007f28260ed6a4 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#5  0x00007f28260edc73 in BIO_read () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#6  0x00007f2826346771 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#7  0x00007f282634a982 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#8  0x00007f28263482c0 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#9  0x00007f282637b499 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#10 0x00007f2826370f75 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#11 0x0000561e86c9667f in CtdlStartTLS ()

No symbol table info available.

#12 0x0000561e86c8806f in CtdlModuleStartCryptoMsgs ()

No symbol table info available.

#13 0x0000561e86caf4d1 in imaps_greeting ()

No symbol table info available.

#14 0x0000561e86c8a88a in worker_thread ()

No symbol table info available.

#15 0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809951565568, -5926168278360471484, 140734826778382, 140734826778383, 139809951563136, 139810044037136, 6021895056957556804, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#16 0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#17 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 3 (Thread 0x7f2809ffb700 (LWP 116)):

#0  __libc_read (nbytes=5, buf=0x7f27ec0d8873, fd=60) at ../sysdeps/unix/sysv/linux/read.c:26

        resultvar = 18446744073709551104

        sc_cancel_oldtype = 0

        __arg3 = <optimized out>

        _a2 = <optimized out>

        sc_ret = <optimized out>

        __value = <optimized out>

        __arg1 = <optimized out>

        _a3 = <optimized out>

        resultvar = <optimized out>

        resultvar = <optimized out>

        __arg2 = <optimized out>

        _a1 = <optimized out>

#1  __libc_read (fd=60, buf=0x7f27ec0d8873, nbytes=5) at ../sysdeps/unix/sysv/linux/read.c:24

No locals.

#2  0x00007f28260f35e5 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#3  0x00007f28260ee7fa in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#4  0x00007f28260ed6a4 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#5  0x00007f28260edc73 in BIO_read () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

No symbol table info available.

#6  0x00007f2826346771 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#7  0x00007f282634a982 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#8  0x00007f28263482c0 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#9  0x00007f282637b499 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#10 0x00007f2826370f75 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1

No symbol table info available.

#11 0x0000561e86c9667f in CtdlStartTLS ()

No symbol table info available.

#12 0x0000561e86c8806f in CtdlModuleStartCryptoMsgs ()

No symbol table info available.

#13 0x0000561e86cb9de7 in pop3s_greeting ()

No symbol table info available.

#14 0x0000561e86c8a88a in worker_thread ()

No symbol table info available.

#15 0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809943172864, -5926168278360471484, 140734826778382, 140734826778383, 139809943170432, 139810111553776, 6021900553978824772, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#16 0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#17 0x0000000000000000 in ?? ()

No symbol table info available.

Thread 2 (Thread 0x7f28215bd700 (LWP 18)):

#0  0x0000561e86ccdd8a in xmpp_queue_event ()

No symbol table info available.

#1  0x0000561e86ccc711 in xmpp_logout_hook ()

No symbol table info available.

#2  0x0000561e86c87e13 in PerformSessionHooks ()

No symbol table info available.

#3  0x0000561e86c8bf2e in CtdlUserLogout ()

No symbol table info available.

#4  0x0000561e86c6fe29 in RemoveContext ()

No symbol table info available.

#5  0x0000561e86c70943 in dead_session_purge ()

No symbol table info available.

#6  0x0000561e86c8a94d in worker_thread ()

No symbol table info available.

#7  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139810335086336, -5926168278360471484, 140734826778382, 140734826778383, 139810335083904, 8396800, 6021954742507456580, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#8  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#9  0x0000000000000000 in ?? ()

No symbol table info available.

Thread 1 (Thread 0x7f280bfff700 (LWP 69)):

#0  0x0000561e86ca5f3f in imap_fetch_body ()

No symbol table info available.

#1  0x0000561e86ca6cbd in imap_do_fetch_msg ()

No symbol table info available.

#2  0x0000561e86ca6fda in imap_do_fetch ()

No symbol table info available.

#3  0x0000561e86ca7a77 in imap_fetch ()

No symbol table info available.

#4  0x0000561e86cb1c0b in imap_command_loop ()

No symbol table info available.

#5  0x0000561e86c8a8a7 in worker_thread ()

No symbol table info available.

#6  0x00007f2825fd1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

        ret = <optimized out>

        pd = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139809976743680, -5926168278360471484, 140734826778382, 140734826778383, 139809976741248, 8396800, 6021896158079797316, 6021962115441018948}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = 0

#7  0x00007f2825d33acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:71

No locals.

#8  0x0000000000000000 in ?? ()

No symbol table info available.

(gdb) 

First of all I am very new to this, so my apologies in advance.

I've had a longtime project to setup an email server that I'm getting back to.  I was following an older video as guidance in which Citadel had an Administration tab right next to the Advanced tab.  I already have the domain and need to enter that information so that I can create the various email accounts using the Citadel Server.  Without the Advanced tab, I can't find where to enter the domain information.  With so much reference information I can't seem to find what I need, I have no doubt that it's there-somewhere.   Can someone point me in the right direction?

That's correct, actually.  Over the last couple of days we've had some pretty bad network flooding.  I think I've got most of it worked out so try again.

Sorry, IG not LG.. fat fingers. 

Also, i was able to download it a bit ago, yes slowly, but it did finally complete. Id offer to share it out on google or something, but if you are smart, you would not trust me anyway. 

SorSun Nov 23 2025 23:43:28 UTC from Nurb432 Subject: Re: Berkeley DB cannot be downloaded

I don't want to speak for LG as i'm not sure if hes hosting the files via the same network provider as the board, but there have been network issues here of late here on the board, and if my guess is right, it could/would effect that too.

Keep retrying..

I don't want to speak for LG as i'm not sure if hes hosting the files via the same network provider as the board, but there have been network issues here of late here on the board, and if my guess is right, it could/would effect that too.

Keep retrying..

(I hope this is the Citadel Support room)

Hello,

I'm trying to install Citadel on my Rpi3 using your easyinstall script. The script does not complete. During installation it needs a certain Berkeley database file. This download is extremely slow and/or the file cannot be downloaded. When I go to https://easyinstall.citadel.org/db-18.1.40.tar.gz myself using a browser the download also fails.

Any help is appreciated.

Regards Bart

A piece of the script output is:

 Downloading Berkeley DB...
--2025-11-23 14:52:31--  https://easyinstall.citadel.org/db-18.1.40.tar.gz
Resolving easyinstall.citadel.org (easyinstall.citadel.org)... 72.0.224.90
Connecting to easyinstall.citadel.org (easyinstall.citadel.org)|72.0.224.90|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30745815 (29M) [application/octet-stream]
Saving to: 'db-18.1.40.tar.gz'

db-18.1.40.tar.gz    2%[                 ] 879.74K  --.-KB/s    eta 58m 51sc

Many thanks can now send emails to google accounts without rejection

Thanks again

easydmarc appears to be a scam site that wants to sell you a service, and deliberately shits on your domain to make you buy it.

Instead, try https://dkimvalidator.com

No signup.  It gives you an address to which you send an email.  You send that address an email, then go back to the site and it tells you if that message was properly DKIM signed.

I have deleted the domain and a new DKIM record was generated but  using easydmarc  to test it gives the following result:-

We couldn't find a DKIM record associated with your domain. You need to generate DKIM record to protect your domain against email phishing attacks.

Ok, so I just ran a quick test and confirmed: you can add a bogus domain (example.xxx or whatever) and then go right ahead and delete it, and it'll post the keys for all of your existing domains as well.

After installing Citadel no DKIM was generated.

Is there a way for me to install  a public and private DKIM key in Citadel sever

Again, there is no way to install a user-generated DKIM key into Citadel.  Citadel generates the private key for each domain and stores it in the database.  When it does so, the corresponding public key is posted to the "Aide" room.

If you are missing your key, you can delete and re-create your local domains on this screen:

[ Administration --> Domain Names and Internet Mail Configuration --> Local host aliases ]

When you do so, new DKIM keys will be generated and posted, and you can use those.

After installing Citadel no DKIM was generated.

Is there a way for me to install  a public and private DKIM key in Citadel sever

I made a dump and a load of the database and it start working but I do not have certain about if it is the reason.

Now a question. The contacts not being updated with new email addresses, is it meaning that the emails were not sent?

For information, the emails appeared in the "Sent Itens".

Wed Nov 19 2025 17:15:35 UTC from luisgo Subject: Possible problem with Citadel.

Dear All,

Until now when I sent an email with a new email address that email address was registed in the "Contacts" folder.

It is not happening anymore. I suppose that is happening since last Centos 9 update (with glibc (??) update, one of the Citadel dependencies was updated).

Notice that Thunderbird had also a recent update.

Is it happening with someone else?

Thank you,

Luís Gonçalves. 

 

Dear All,

Until now when I sent an email with a new email address that email address was registed in the "Contacts" folder.

It is not happening anymore. I suppose that is happening since last Centos 9 update (with glibc (??) update, one of the Citadel dependencies was updated).

Notice that Thunderbird had also a recent update.

Is it happening with someone else?

Thank you,

Luís Gonçalves. 

https://citadel.org/sslcertificates.html

Tue Nov 18 2025 14:48:49 UTC from luisgo Subject: Re: DKIM record

/usr/local/citadel/keys/

Tue Nov 18 2025 09:35:10 UTC from johno Subject: DKIM record

I am having a problem getting the DKIM record to be recognised. If i create new public and private keys where can i find the files in a citadel system to install these keys.

 

/usr/local/citadel/keys/

Tue Nov 18 2025 09:35:10 UTC from johno Subject: DKIM record

I am having a problem getting the DKIM record to be recognised. If i create new public and private keys where can i find the files in a citadel system to install these keys.

I am having a problem getting the DKIM record to be recognised. If i create new public and private keys where can i find the files in a citadel system to install these keys.

I forgot.

3GBytes of Citadel Email data and only 2 email users. 

Fri Nov 14 2025 15:29:14 UTC from luisgo Subject: Re: Concerning tha last two posts in Support Room.

I have 3GBytes of RAM (all). 20GBytes Free disk. It is a KVM Centos 9 container. Basically it only runs Apache and Citadel. The Web page is not heavy.

Wed Nov 12 2025 16:07:32 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

So ... all the weird issues you've been having were just constrained system resources? It's interesting, because Citadel isn't very resource hungry.
But this tells us perhaps that we would be well served by detecting that kind of situation and saying something about it.

We already refuse to accept new messages when there is less than 100 MB free on the disk. Perhaps we need to look at memory too.

 

I have 3GBytes of RAM (all). 20GBytes Free disk. It is a KVM Centos 9 container. Basically it only runs Apache and Citadel. The Web page is not heavy.

Wed Nov 12 2025 16:07:32 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

So ... all the weird issues you've been having were just constrained system resources? It's interesting, because Citadel isn't very resource hungry.
But this tells us perhaps that we would be well served by detecting that kind of situation and saying something about it.

We already refuse to accept new messages when there is less than 100 MB free on the disk. Perhaps we need to look at memory too.

Dear All,

Concerning this problem I improved the realibility of Citadel doing the following.

I noticed that the available memory (with "top") was not too much (some tens of Megabytes) and it had a lot of Buffer/Cache Memory. And the Swap has about 200 MBytes or more.

I suspected that when Citadel needed memory was not freed fast enough. Then I remembered that in Apache, I has configured disk cache with some time out to free it.

I disallowed the disk cache in Apache and the time out. I noticed that now I have more free memory (hundreds od Megabytes) and the swap is almost not used. Some times the free memory goes to more than a GByte showing that memory is freed.

Thank you,

Luís Gonçalves.   

Wed Oct 01 2025 06:46:58 UTC from luisgo Subject: Re: Concerning tha last two posts in Support Room.

Specify please what you want to solve the problem. Do you want to have access to my server? Stack trace? How I obtain it?

 

Sat Sep 20 2025 18:08:00 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

Sorry again.

This night happened again. More coredumps. I updated the citadel. It installed and compiled a version but it remained the same version number. Did that attended my problem?

If not a send the new coredumps.

https://luisgo.pro/down/coredumps3.zip

I hope that you solved or are solving this problem.

Let see that if or not happen again.

Unfortunately, your core dumps are not useful to anyone because they are specific to your citserver binary.  I suppose if you include your citserver binary it might be useful, otherwise a stack trace is better.

In your case, we're just seeing so many weird things on your system that don't happen anywhere else.  It's difficult to pin down why.  I would strongly suggest that you give some consideration to moving to the containerized version of the Citadel system.  It is lovingly built for you in a carefully controlled environment.   Perhaps switch to the container version, and if necessary, do a ctdldump/ctdlload on your existing data.

And of course take lots of backups before doing anything.

 

Dear All,

This is happening to me. 

I think that is happening since the last upgrade (some days ago) when I send a message it has the date one hour after I sent.

I only noticed now. I upgraded at 6 Nov, 17H39 Central Europe Time. I can verify in some emails in the "Sent Items" after that (6 Nov) are registered one hour after than when sent. 

Happens with all the emails clients.

Are this happening to someone else?

Thank you,

Luís Gonçalves.

SUMMARY

-------

Citadel Server (Version 1020, build 25289) crashes with SIGSEGV when processing 

emails containing malformed HTML with missing opening angle brackets. The server 

enters an unrecoverable crash loop on startup, requiring manual database 

intervention to restore service.

SEVERITY: CRITICAL

------------------

• Impact: Complete server unavailability (crash loop)

• Reproducibility: 100% reproducible

• Recovery: Requires manual database manipulation

• Root Cause: NULL pointer dereference in libcitadel.so.4

ENVIRONMENT

-----------

Citadel Version: 1020 (build 25289)

libcitadel: libcitadel.so.4 (unnumbered)

OS: Ubuntu 24 (Linux x86_64)

Crash Location: lib/html_to_ascii.c:161

PROBLEM DESCRIPTION

-------------------

When Citadel receives email containing HTML where the opening '<' character is 

missing from a tag, the html_to_ascii() function crashes with a NULL pointer 

dereference. This occurs during full-text indexing at server startup, creating 

a crash loop that prevents the server from running.

WE ARE RECEIVING THESE MALFORMED EMAILS REGULARLY, making this a recurring production issue.

TECHNICAL DETAILS

-----------------

Stack Trace:

    SIGSEGV received at address (nil)

    #0  __strncpy_sse2_unaligned ()

    #1  strncpy (__len=1024, __src=0x0, __dest=...)

    #2  html_to_ascii (...) at lib/html_to_ascii.c:161

        tag_start = 0x0  <-- NULL POINTER

        tag_end = 0x... ">Hello,</p>..."

    #3  fixed_output (...) at server/msgbase.c:1185

    #4  mime_decode (...) at lib/mime_parser.c:224

    ...

    #10 ft_index_message (msgnum=..., op=1) 

        at server/modules/fulltext/serv_fulltext.c:88

Root Cause:

In html_to_ascii() at line 161:

1. Function finds tag_end pointing to a '>' character

2. Attempts to find the matching '<' character for tag_start

3. When the '<' is missing, tag_start remains NULL

4. Calls strncpy(tag, tag_start, 1024) with NULL source pointer

5. SEGMENTATION FAULT (sigsegv)

Malformed HTML Example (triggers crash):

    p>Hello,</p>

    <p>I have information that is privileged...</p>

Should be:

    <p>Hello,</p>

    <p>I have information that is privileged...</p>

Note: The HTML is otherwise well-formed - only the opening '<' is missing.

STEPS TO REPRODUCE

------------------

1. Send email to Citadel with HTML body starting with text followed by '>' 

   (missing opening '<')

2. Restart Citadel server or wait for full-text indexing

3. Server crashes during message indexing

4. Server enters crash loop - crashes on every restart attempt

5. systemd reports: "citadel.service: Failed with result 'core-dump'"

EXPECTED BEHAVIOR

-----------------

Server should handle malformed HTML gracefully by either:

• Skipping malformed tags and continuing processing

• Logging warning and indexing message without HTML conversion

• Failing safely without crashing

ACTUAL BEHAVIOR

---------------

• Server crashes with SIGSEGV

• Enters unrecoverable crash loop

• Cannot start until problematic message is manually removed

• Service completely unavailable

IMPACT ASSESSMENT

-----------------

Business Impact: SEVERE

• Complete service outage - no email access during crash loop

• Manual intervention required - cannot auto-recover

• Data loss risk - requires database manipulation to fix

• Recurring issue - we have recently started to receive these malformed emails regularly

WORKAROUND

----------

The only current workaround is to manually delete the full-text index files:

    systemctl stop citadel

    rm -f /usr/local/citadel/data/cdb.0c    # Full-text index database

    rm -f /usr/local/citadel/data/cdb.0d    # Index control file

    systemctl start citadel

This forces Citadel to rebuild the full-text index from scratch. During the 

rebuild process, the indexer appears to handle the malformed message differently 

(possibly with better error handling) and skips it, allowing the server to start.

However, this workaround:

• Requires root access

• Causes service downtime

• Forces complete index rebuild (time-consuming on large databases)

• Temporarily loses full-text search capability during rebuild

• Only works until the next malformed email arrives

• Is NOT a sustainable solution when receiving these emails regularly

ATTEMPTED SOLUTIONS

-------------------

  Berkeley DB recovery (db_recover -c) - Did not resolve crash

  Citadel database cleanup script (database_cleanup.sh) - Performed full 

  Berkeley DB dump/load of ALL database files (cdb.00 through cdb.0d), removed 

  old databases and transaction logs, successfully reloaded everything - CRASH 

  PERSISTED UNCHANGED

  Database dump/reload manually (db_dump | db_load) - Preserved the malformed 

  message, crash persisted

  Full-text index deletion (cdb.0c, cdb.0d) - Allows server to start by 

  forcing index rebuild

Note: The fact that a complete database cleanup (dump/load of all 14 database 

files) did not resolve the issue demonstrates that:

1. The database files themselves are not corrupted

2. The malformed HTML content is preserved exactly through dump/load

3. Standard database recovery procedures are ineffective for this bug

4. The issue is with the content being valid Berkeley DB data but containing 

   HTML that crashes the parser

PROPOSED FIX

------------

The html_to_ascii() function should validate tag_start before use:

    // Around line 161 in lib/html_to_ascii.c

    if (tag_start == NULL || tag_end == NULL) {

        // Handle malformed HTML gracefully

        // Log warning, skip tag, or use safe default

        continue;  // or appropriate error handling

    }

    strncpy(tag, tag_start, tag_len);

Additionally, consider:

1. Input validation before HTML parsing

2. Defensive coding for all pointer operations in parser

3. Graceful degradation when HTML is malformed

REQUEST

-------

This is a CRITICAL PRODUCTION ISSUE causing recurring outages. We urgently 

request:

1. A patch for the NULL pointer check in html_to_ascii()

2. Guidance on additional defensive measures

3. Consider making full-text indexing more resilient to malformed input

Thank you for your attention to this severe bug.

ericsondatasystems

 don’t see those messages in that queue. Previously, I had to remove those messages directly from the DB because Citadel was crashing with a segmentation fault. Is it possible I didn’t remove them from all the required places?

In the Citadel console I also see:

paź 15 14:55:40 xxx citserver[328259]: citserver[328259]: msgbase: message #11040 was not found

paź 15 14:55:40 xxx citserver[328259]: msgbase: message #11040 was not found

Could these be ghost/orphaned messages?

śro paź 15 2025 00:36:59 UTC od IGnatius T Foobar Temat: Re: smtp queue

2. After my problem with server hang (few weekes ago I send here
information about that) I notice that in "Aida" directory there are
planty of meassages to today. Server is generating it reapetly after few

minutes:

Sounds like someone dumped a big pile of boomerang spam at your site, maybe?

The queue is in a hidden room called __CitadelSMTPspoolout__

You can delete from there.

navigate to https://easyinstall.citadel.org/

you'll find the site is down

i am trying to install citadel using sudo curl https://easyinstall.citadel.org/install | bash

Instructions for which i found on https://www.citadel.org/easyinstall.html

anyone know why site is down

also, is a copy of the easyinstall script available elsewhere?

thanks - ericsondatasystems

Hi,

two problems to solve - could you please give me a tip:

1. After sending email, massage goes to SMTP queue. I should expect that it will be send stright after. In my server email will be send after 8-10minutes. There is big delay. I notice also that:

2. After my problem with server hang (few weekes ago I send here information about that) I notice that in "Aida" directory there are planty of meassages to today. Server is generating it reapetly after few minutes:

2099538367@Uncensored

Specify please what you want to solve the problem. Do you want to have access to my server? Stack trace? How I obtain it?

Stack trace would be far more helpful than what you are currently trying to do.   Since you seem to have the ability to produce core dumps, you are halfway there already.

After a core dump, do this:

gdb /usr/local/citadel/citserver core.xxxxxxxxx   (use the name of your core file)

The debugger will open, and show you where it crashed.   Now type:

thread apply all bt

And post the output.  it should show lots of filenames and line numbers.

Specify please what you want to solve the problem. Do you want to have access to my server? Stack trace? How I obtain it?

Sat Sep 20 2025 18:08:00 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

Sorry again.

This night happened again. More coredumps. I updated the citadel. It installed and compiled a version but it remained the same version number. Did that attended my problem?

If not a send the new coredumps.

https://luisgo.pro/down/coredumps3.zip

I hope that you solved or are solving this problem.

Let see that if or not happen again.

Unfortunately, your core dumps are not useful to anyone because they are specific to your citserver binary.  I suppose if you include your citserver binary it might be useful, otherwise a stack trace is better.

In your case, we're just seeing so many weird things on your system that don't happen anywhere else.  It's difficult to pin down why.  I would strongly suggest that you give some consideration to moving to the containerized version of the Citadel system.  It is lovingly built for you in a carefully controlled environment.   Perhaps switch to the container version, and if necessary, do a ctdldump/ctdlload on your existing data.

And of course take lots of backups before doing anything.

https://luisgo.pro/down/coredumps6.zip

Including citserver 

Centos 9, last version.

Dear All,

I submit another coredump and this time I include the Citadel server binary.

https://luisgo.pro/down/coredumps5.zip

Thank you,

Luís Gonçalves.

Subject pretty much says it all: Are modems still supported in Citadel?

SP

I send you priv messages.

J.K.

Do you still have the unedited dump?  If you could upload the removed records it would be an invaluable aid in figuring out what went wrong.

Ok, I was able to turn it on. dumped all DB to file. Removed somehow all records with "11040" and load it back to server. Looks like it do the trick.

J.K.

wto wrz 23 2025 06:41:37 UTC od kwadrat Temat: Segmentation fault (core dumped)

I've got problem with turning on citadel server. It was working but from a day not. When I tried to turn it on have:

 

citserver[76547]: smtpclient: start full queue run , last_queue_job_processed=0 , last_queue_job_submitted=0

citserver[76547]: smtpclient: 1 messages to be processed

citserver[76547]: msgbase: CtdlFetchMessage(11041, 1)

citserver[76547]: smtpclient: attempting delivery of message <11041> now

citserver[76547]: smtpclient: smtp_attempt_delivery(11040, 0@test.com)

citserver[76547]: msgbase: CtdlOutputMsg(msgnum=11040, mode=1, section=<>)

citserver[76547]: msgbase: CtdlFetchMessage(11040, 1)

citserver[76547]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 1, 0, 0, 1

citserver[76547]: smtpclient: dkim-signing for selector <hdfxv> in domain <XXXXX.XXXXXX.XXXXX>

citserver[76547]: SIGSEGV received in thread 133741677622976 at address 0x1

citserver[76547]: Stack frame #0: ./citserver(+0x34a9d) [0x5dced6cfaa9d]

citserver[76547]: Stack frame #1: /lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x79a32ce45330]

citserver[76547]: Stack frame #2: ./citserver(+0x723f7) [0x5dced6d383f7]

citserver[76547]: Stack frame #3: ./citserver(+0x72d6d) [0x5dced6d38d6d]

citserver[76547]: Stack frame #4: ./citserver(+0x776b3) [0x5dced6d3d6b3]

citserver[76547]: Stack frame #5: ./citserver(+0x78280) [0x5dced6d3e280]

citserver[76547]: Stack frame #6: ./citserver(+0x7899d) [0x5dced6d3e99d]

citserver[76547]: Stack frame #7: ./citserver(+0x78a44) [0x5dced6d3ea44]

citserver[76547]: Stack frame #8: ./citserver(+0x32bdf) [0x5dced6cf8bdf]

citserver[76547]: Stack frame #9: ./citserver(+0x1c052) [0x5dced6ce2052]

citserver[76547]: Stack frame #10: ./citserver(+0x35d7b) [0x5dced6cfbd7b]

citserver[76547]: Stack frame #11: /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x79a32ce9caa4]

citserver[76547]: Stack frame #12: /lib/x86_64-linux-gnu/libc.so.6(+0x129c6c) [0x79a32cf29c6c]

Segmentation fault (core dumped)

 

From what I undersatnd there is a problem with reading email id=11041. Is there a way to remove it from database manually? It's Berkley DB from what I notice but i'm not able to do this... Could someone give  me a tip?

 

Best Regards

J.K.

I've got problem with turning on citadel server. It was working but from a day not. When I tried to turn it on have:

citserver[76547]: smtpclient: start full queue run , last_queue_job_processed=0 , last_queue_job_submitted=0

citserver[76547]: smtpclient: 1 messages to be processed

citserver[76547]: msgbase: CtdlFetchMessage(11041, 1)

citserver[76547]: smtpclient: attempting delivery of message <11041> now

citserver[76547]: smtpclient: smtp_attempt_delivery(11040, 0@test.com)

citserver[76547]: msgbase: CtdlOutputMsg(msgnum=11040, mode=1, section=<>)

citserver[76547]: msgbase: CtdlFetchMessage(11040, 1)

citserver[76547]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 1, 0, 0, 1

citserver[76547]: smtpclient: dkim-signing for selector <hdfxv> in domain <XXXXX.XXXXXX.XXXXX>

citserver[76547]: SIGSEGV received in thread 133741677622976 at address 0x1

citserver[76547]: Stack frame #0: ./citserver(+0x34a9d) [0x5dced6cfaa9d]

citserver[76547]: Stack frame #1: /lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x79a32ce45330]

citserver[76547]: Stack frame #2: ./citserver(+0x723f7) [0x5dced6d383f7]

citserver[76547]: Stack frame #3: ./citserver(+0x72d6d) [0x5dced6d38d6d]

citserver[76547]: Stack frame #4: ./citserver(+0x776b3) [0x5dced6d3d6b3]

citserver[76547]: Stack frame #5: ./citserver(+0x78280) [0x5dced6d3e280]

citserver[76547]: Stack frame #6: ./citserver(+0x7899d) [0x5dced6d3e99d]

citserver[76547]: Stack frame #7: ./citserver(+0x78a44) [0x5dced6d3ea44]

citserver[76547]: Stack frame #8: ./citserver(+0x32bdf) [0x5dced6cf8bdf]

citserver[76547]: Stack frame #9: ./citserver(+0x1c052) [0x5dced6ce2052]

citserver[76547]: Stack frame #10: ./citserver(+0x35d7b) [0x5dced6cfbd7b]

citserver[76547]: Stack frame #11: /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x79a32ce9caa4]

citserver[76547]: Stack frame #12: /lib/x86_64-linux-gnu/libc.so.6(+0x129c6c) [0x79a32cf29c6c]

Segmentation fault (core dumped)

From what I undersatnd there is a problem with reading email id=11041. Is there a way to remove it from database manually? It's Berkley DB from what I notice but i'm not able to do this... Could someone give  me a tip?

Best Regards

J.K.

Sorry again.

This night happened again. More coredumps. I updated the citadel. It installed and compiled a version but it remained the same version number. Did that attended my problem?

If not a send the new coredumps.

https://luisgo.pro/down/coredumps3.zip

I hope that you solved or are solving this problem.

Let see that if or not happen again.

Unfortunately, your core dumps are not useful to anyone because they are specific to your citserver binary.  I suppose if you include your citserver binary it might be useful, otherwise a stack trace is better.

In your case, we're just seeing so many weird things on your system that don't happen anywhere else.  It's difficult to pin down why.  I would strongly suggest that you give some consideration to moving to the containerized version of the Citadel system.  It is lovingly built for you in a carefully controlled environment.   Perhaps switch to the container version, and if necessary, do a ctdldump/ctdlload on your existing data.

And of course take lots of backups before doing anything.

Sorry again.

This night happened again. More coredumps. I updated the citadel. It installed and compiled a version but it remained the same version number. Did that attended my problem?

If not a send the new coredumps.

https://luisgo.pro/down/coredumps3.zip

I hope that you solved or are solving this problem.

Let see that if or not happen again.

Thank you,

Luís Gonçalves.

Wed Sep 17 2025 09:36:18 UTC from luisgo Subject: Re: Concerning tha last two posts in Support Room.

From a previous post (see above):

 

Centos 9, VPS (KVM), 3 GBytes RAM, 18 GBytes available disk, 2 virtual processors.

Last version of Citadel (1020)

Linux xxxxxx.xxx 5.14.0-612.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Aug 30 16:03:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 63

model name      : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz

stepping        : 2

cpu MHz         : 2599.925

cache size      : 4096 KB

physical id     : 0

siblings        : 1

core id         : 0

cpu cores       : 1

apicid          : 0

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 15

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc nopl cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes hypervisor lahf_lm abm pti xsaveopt xsavec xsaves

bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data bhi spectre_v2_user its

bogomips        : 5199.85

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

 

processor       : 1

vendor_id       : GenuineIntel

cpu family      : 6

model           : 63

model name      : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz

stepping        : 2

cpu MHz         : 2599.925

cache size      : 4096 KB

physical id     : 1

siblings        : 1

core id         : 0

cpu cores       : 1

apicid          : 1

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 15

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc nopl cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes hypervisor lahf_lm abm pti xsaveopt xsavec xsaves

bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data bhi spectre_v2_user its

bogomips        : 5199.85

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

 

power management:

Tue Sep 16 2025 14:22:31 UTC from Motobike Subject: Re: Concerning tha last two posts in Support Room.

 

Di Sep 16 2025 13:58:28 UTC von luisgo Betreff: Re: Concerning tha last two posts in Support Room.

I provided the core dumps for you all evaluate.

As I said in a past post is a Centos 9.

Tue Sep 16 2025 03:36:14 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

What system you are using ? This dont looks like a general
citadel problem for me. 

Agreed, this doesn't seem like a failure mode related to Citadel itself.
Something else is wrong with the system. Any log messages giving a clue?

 

Wich filesystem you are using ? ext4 or something else ? do you use disk encryption ? 

Maybe there are changes in kernel for system drivers ? when you have done the last update to your kernel ? wich kernel you are using ?

 

So many questions :D

Greetings

Mike

 

 

It is Centos 9 and the last version of openssl.

I analyzed better and in my server SMTP requires TLS. And I suppose that those errors are from servers with bad TLS or something near that. Because the server messages before those errors are from connections from outside servers. Also I have another port in SMTP-SSL, originally from other service. Perhaps that.

Thu Sep 18 2025 06:27:08 UTC from Motobike Subject: Re: Another problem.

You installed and configured TLS 1.3 on your Server ?

 

Do Sep 18 2025 03:21:40 UTC von luisgo Betreff: Another problem.

Dear All,

In the citserver log appear all the time:

Sep 18 05:03:00 luisgo.pro citserver[17435]: crypto: SSL_accept failed: wrong version number

 

See please:

https://luisgo.pro/images/cert1.jpg

https://luisgo.pro/images/cert2.jpg

Sorry in Portuguese.

Thank you,

Luís Gonçalves.

 

 

You installed and configured TLS 1.3 on your Server ?

Do Sep 18 2025 03:21:40 UTC von luisgo Betreff: Another problem.

Dear All,

In the citserver log appear all the time:

Sep 18 05:03:00 luisgo.pro citserver[17435]: crypto: SSL_accept failed: wrong version number

 

See please:

https://luisgo.pro/images/cert1.jpg

https://luisgo.pro/images/cert2.jpg

Sorry in Portuguese.

Thank you,

Luís Gonçalves.

 

Dear All,

In the citserver log appear all the time:

Sep 18 05:03:00 luisgo.pro citserver[17435]: crypto: SSL_accept failed: wrong version number

See please:

https://luisgo.pro/images/cert1.jpg

https://luisgo.pro/images/cert2.jpg

Sorry in Portuguese.

Thank you,

Luís Gonçalves.

From a previous post (see above):

Centos 9, VPS (KVM), 3 GBytes RAM, 18 GBytes available disk, 2 virtual processors.

Last version of Citadel (1020)

Linux xxxxxx.xxx 5.14.0-612.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Aug 30 16:03:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 63

model name      : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz

stepping        : 2

cpu MHz         : 2599.925

cache size      : 4096 KB

physical id     : 0

siblings        : 1

core id         : 0

cpu cores       : 1

apicid          : 0

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 15

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc nopl cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes hypervisor lahf_lm abm pti xsaveopt xsavec xsaves

bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data bhi spectre_v2_user its

bogomips        : 5199.85

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

processor       : 1

vendor_id       : GenuineIntel

cpu family      : 6

model           : 63

model name      : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz

stepping        : 2

cpu MHz         : 2599.925

cache size      : 4096 KB

physical id     : 1

siblings        : 1

core id         : 0

cpu cores       : 1

apicid          : 1

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 15

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc nopl cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes hypervisor lahf_lm abm pti xsaveopt xsavec xsaves

bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data bhi spectre_v2_user its

bogomips        : 5199.85

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

Tue Sep 16 2025 14:22:31 UTC from Motobike Subject: Re: Concerning tha last two posts in Support Room.

 

Di Sep 16 2025 13:58:28 UTC von luisgo Betreff: Re: Concerning tha last two posts in Support Room.

I provided the core dumps for you all evaluate.

As I said in a past post is a Centos 9.

Tue Sep 16 2025 03:36:14 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

What system you are using ? This dont looks like a general
citadel problem for me. 

Agreed, this doesn't seem like a failure mode related to Citadel itself.
Something else is wrong with the system. Any log messages giving a clue?

 

Wich filesystem you are using ? ext4 or something else ? do you use disk encryption ? 

Maybe there are changes in kernel for system drivers ? when you have done the last update to your kernel ? wich kernel you are using ?

 

So many questions :D

Greetings

Mike

 

Di Sep 16 2025 13:58:28 UTC von luisgo Betreff: Re: Concerning tha last two posts in Support Room.

I provided the core dumps for you all evaluate.

As I said in a past post is a Centos 9.

Tue Sep 16 2025 03:36:14 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

What system you are using ? This dont looks like a general
citadel problem for me. 

Agreed, this doesn't seem like a failure mode related to Citadel itself.
Something else is wrong with the system. Any log messages giving a clue?

 

Wich filesystem you are using ? ext4 or something else ? do you use disk encryption ? 

Maybe there are changes in kernel for system drivers ? when you have done the last update to your kernel ? wich kernel you are using ?

So many questions :D

Greetings

Mike

I provided the core dumps for you all evaluate.

As I said in a past post is a Centos 9.

Tue Sep 16 2025 03:36:14 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

What system you are using ? This dont looks like a general
citadel problem for me. 

Agreed, this doesn't seem like a failure mode related to Citadel itself.
Something else is wrong with the system. Any log messages giving a clue?

Dear Motobike,

That script did not worked. I used instead dump and load separatelly.

Thank you,

Luís Gonçalves.

Mon Sep 15 2025 12:53:23 UTC from Motobike Subject: Re: Concerning tha last two posts in Support Room.

 

Mo Sep 15 2025 12:30:08 UTC von luisgo Betreff: Re: Concerning tha last two posts in Support Room.

Dear Motobike,

Which script?

Thank you,

Luís Gonçalves.

 

Sun Sep 14 2025 20:17:27 UTC from Motobike Subject: Re: Concerning tha last two posts in Support Room.

Hi Luis,

i shrink database with the database script inside the program dir. That works fine for me. my database i now round about 90GB and i dont have any problems. 

 

Greetings

Mike

 

 

Look inside your citadel directory

 

root@citadel:/usr/local/citadel# ./database_cleanup.sh

 

You ever tryed this script ?

 

Greetings 

Mike

 

Thank you IGnatius T Foobar,

compiling went well,now it runs on recent version.

Fr Sep 12 2025 01:40:06 UTC von IGnatius T Foobar Betreff: Re: upgrade with easy install on raspberry pi os

 

Any idea what this mean?

maybe there is no longer support for a 32 bit OS?

32-bit support is ending soon, but it hasn't ended yet.

Try removing the file /usr/local/ctdlsupport/db-easyinstall.sum (or stash it somewhere safe) and attempt the build again.   This will force it to rebuild Berkeley DB.

 

Mo Sep 15 2025 12:30:08 UTC von luisgo Betreff: Re: Concerning tha last two posts in Support Room.

Dear Motobike,

Which script?

Thank you,

Luís Gonçalves.

 

Sun Sep 14 2025 20:17:27 UTC from Motobike Subject: Re: Concerning tha last two posts in Support Room.

Hi Luis,

i shrink database with the database script inside the program dir. That works fine for me. my database i now round about 90GB and i dont have any problems. 

 

Greetings

Mike

 

 

Look inside your citadel directory

root@citadel:/usr/local/citadel# ./database_cleanup.sh

Dear Motobike,

Which script?

Thank you,

Luís Gonçalves.

Sun Sep 14 2025 20:17:27 UTC from Motobike Subject: Re: Concerning tha last two posts in Support Room.

Hi Luis,

i shrink database with the database script inside the program dir. That works fine for me. my database i now round about 90GB and i dont have any problems. 

 

Greetings

Mike

 

Centos 9, VPS, 3 GBytes RAM, 18 GBytes available disk, 2 virtual processors.

Last version of Citadel (1020)

Linux xxxxxx.xxx 5.14.0-612.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Aug 30 16:03:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 63

model name      : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz

stepping        : 2

cpu MHz         : 2599.925

cache size      : 4096 KB

physical id     : 0

siblings        : 1

core id         : 0

cpu cores       : 1

apicid          : 0

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 15

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc nopl cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes hypervisor lahf_lm abm pti xsaveopt xsavec xsaves

bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data bhi spectre_v2_user its

bogomips        : 5199.85

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

processor       : 1

vendor_id       : GenuineIntel

cpu family      : 6

model           : 63

model name      : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz

stepping        : 2

cpu MHz         : 2599.925

cache size      : 4096 KB

physical id     : 1

siblings        : 1

core id         : 0

cpu cores       : 1

apicid          : 1

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 15

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc nopl cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes hypervisor lahf_lm abm pti xsaveopt xsavec xsaves

bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data bhi spectre_v2_user its

bogomips        : 5199.85

clflush size    : 64

cache_alignment : 64

address sizes   : 40 bits physical, 48 bits virtual

power management:

Sun Sep 14 2025 20:37:11 UTC from Motobike Subject: Re: Concerning tha last two posts in Support Room.

 

So Sep 14 2025 20:14:07 UTC von luisgo Betreff: Concerning tha last two posts in Support Room.

Dear All,

Due to problems reported in the last two posts in this room I am loosing emails. Perhaps due to Citadel misses to write to the database.

Thank you,

Luís Gonçalves.

What system you are using ? This dont looks like a general citadel problem for me. 

So Sep 14 2025 20:14:07 UTC von luisgo Betreff: Concerning tha last two posts in Support Room.

Dear All,

Due to problems reported in the last two posts in this room I am loosing emails. Perhaps due to Citadel misses to write to the database.

Thank you,

Luís Gonçalves.

What system you are using ? This dont looks like a general citadel problem for me. 

Hi Luis,

i shrink database with the database script inside the program dir. That works fine for me. my database i now round about 90GB and i dont have any problems. 

Greetings

Mike

Dear All,

Due to problems reported in the last two posts in this room I am loosing emails. Perhaps due to Citadel misses to write to the database.

Thank you,

Luís Gonçalves.

Dear All,

I am still waiting about the procedure to shrink the Database. How to configure the hidden configuration variable?

Luís Gonçalves.

Dear All,

Second time in a day that I have to delete the database and put back the backup.

Despite I said in a past post that I have not such much disk space I still have 18GBytes.

Every month I make a bakup of my Thunderbird and I delete one month of my older emails. (in a time range of 4 months). But as is normal the database does not reduce. The database is increasing despite it is said that the new emails will occupy the old ones.

My intuition is that this email replacement is not such solid procedure. Sorry to say.

See please the coredump files below:

https://luisgo.pro/down/coredumps2.zip

Thank you,

Luís Gonçalves.

Any idea what this mean?

maybe there is no longer support for a 32 bit OS?

32-bit support is ending soon, but it hasn't ended yet.

Try removing the file /usr/local/ctdlsupport/db-easyinstall.sum (or stash it somewhere safe) and attempt the build again.   This will force it to rebuild Berkeley DB.

Hi there,

after a while I wanted to upgrade my citadel installation on a raspberrypi with bookworm 32 bit official raspberrypi os

with easy install

My previous version is: Citadel Server 996 mit WebCit #define CLIENT_VERSION 996 /* This version of WebCit */
Server Build: 996

compiling leads to this error:

gcc -I/usr/local/ctdlsupport/include  -g -DBUILD_ID=\"25254\" -DCTDLDIR=\"/usr/local/citadel\"  -I/usr/local/ctdlsupport/include  -g -L/usr/local/ctdlsupport/lib -Wl,--rpath -Wl,/usr/local/ctdlsupport/lib  -lssl -lcrypto -lz utils/ctdldump.c server/visit.c -DVISIT_DUMP_LOAD server/backends/berkeley_db/berkeley_db.o server/backends/common/database.o -lcitadel -lz -ldb -lpthread -o ctdldump
/usr/bin/ld: /usr/local/ctdlsupport/lib/libdb.a(os_yield.o): in function `__os_yield_ctdl':
os_yield.c:(.text+0x94): undefined reference to `pthread_yield'
collect2: error: ld returned 1 exit status
gmake: *** [Makefile:59: ctdldump] Fehler 1

 Citadel Easy Install is aborting.
 The last few lines above this message may indicate what went wrong.

Any idea what this mean?

maybe there is no longer support for a 32 bit OS?

I perhaps solved.

In my server I have ports switched between services. And the services in firewalld were not well configured.

Until now is working.

Sorry about all.

Thu Sep 04 2025 15:34:13 UTC from luisgo Subject: Re: Urgent

https://luisgo.pro/down/coredumps.zip

 

Thu Sep 04 2025 15:18:39 UTC from luisgo Subject: Re: Urgent

With Webcit seems be stable. Some problem with external comunication to Thunderbird. 

 

 

Thu Sep 04 2025 14:38:41 UTC from luisgo Subject: Re: Urgent

I downgraded glibc and it did not worked.

A recompilation did not worked also.

 

Thu Sep 04 2025 13:58:07 UTC from luisgo Subject: Urgent

Dear all,

I updated my Centos 9 and the citadel simply stops with Core Dump.

When I do the easy install gives the following:

Error

D▒s▒

 

 

I suppose that was the "yum update". Some of the libraries used by citadel were updated. I suppose some incompatibility.

 

How I force a new compile of citadel?

 

Thanks,

 

Luís Gonçaçalves.

 

 

 

https://luisgo.pro/down/coredumps.zip

Thu Sep 04 2025 15:18:39 UTC from luisgo Subject: Re: Urgent

With Webcit seems be stable. Some problem with external comunication to Thunderbird. 

 

 

Thu Sep 04 2025 14:38:41 UTC from luisgo Subject: Re: Urgent

I downgraded glibc and it did not worked.

A recompilation did not worked also.

 

Thu Sep 04 2025 13:58:07 UTC from luisgo Subject: Urgent

Dear all,

I updated my Centos 9 and the citadel simply stops with Core Dump.

When I do the easy install gives the following:

Error

D▒s▒

 

 

I suppose that was the "yum update". Some of the libraries used by citadel were updated. I suppose some incompatibility.

 

How I force a new compile of citadel?

 

Thanks,

 

Luís Gonçaçalves.

 

 

With Webcit seems be stable. Some problem with external comunication to Thunderbird. 

Thu Sep 04 2025 14:38:41 UTC from luisgo Subject: Re: Urgent

I downgraded glibc and it did not worked.

A recompilation did not worked also.

 

Thu Sep 04 2025 13:58:07 UTC from luisgo Subject: Urgent

Dear all,

I updated my Centos 9 and the citadel simply stops with Core Dump.

When I do the easy install gives the following:

Error

D▒s▒

 

 

I suppose that was the "yum update". Some of the libraries used by citadel were updated. I suppose some incompatibility.

 

How I force a new compile of citadel?

 

Thanks,

 

Luís Gonçaçalves.

 

I downgraded glibc and it did not worked.

A recompilation did not worked also.

Thu Sep 04 2025 13:58:07 UTC from luisgo Subject: Urgent

Dear all,

I updated my Centos 9 and the citadel simply stops with Core Dump.

When I do the easy install gives the following:

Error

D▒s▒

 

 

I suppose that was the "yum update". Some of the libraries used by citadel were updated. I suppose some incompatibility.

 

How I force a new compile of citadel?

 

Thanks,

 

Luís Gonçaçalves.

Dear all,

I updated my Centos 9 and the citadel simply stops with Core Dump.

When I do the easy install gives the following:

Error

D▒s▒

Mo Sep 01 2025 17:53:00 UTC von IGnatius T Foobar Betreff: Re: BCC

There is a problem with BCC recieptiens.

They will all appear in Mail header as > Envelope-to:

Is there a option to hide recieptens in header ?

Can you clarify please, is this on outbound or inbound email?

Hi Ignatius,

this happens on Outbound Mails. It doesnt matter wich Client you use. All BCC mails will show up for everybody who receive the mail. Not in client, but if you look in header from the mail... all will show up @envelopeto

Greetings

Mike

Dear Foobar,

I have not so much space.

Do not see this as I putting bad reputation on the software.

Could you please explain how to set "c_shrink_db_files"?

Thank you,

Luís Gonçalves.

Mon Sep 01 2025 17:57:32 UTC from IGnatius T Foobar Subject: Re: Shrink my databases.

I want to shrink my databases. I tried with the shell script "database_cleanup.sh" and it gives a error even with citadel shutdown and with plenty of disk space (I renamed /tmp in the script).

How to set the "c_shrink_db_files" variable (before compilation)?

I want only an one time shrink.

c_shrink_db_files is not a compile-time option.  We've worked hard to have as few of those as possible.

If you set the c_shrink_db_files configuration option, the database will be compacted at shutdown time.

If you're not satisfied with the results, I would suggest using ctdldump and ctdlload, not database_cleanup.sh.   But really, Citadel is good with disk space -- it will reuse the "holes" in those files to write new data.  At some point your disk utilization will stabilize if you have both new data coming in and old data being purged.  If you have plenty of disk space and your system is running cleanly I would advise leaving it alone.

I want to shrink my databases. I tried with the shell script "database_cleanup.sh" and it gives a error even with citadel shutdown and with plenty of disk space (I renamed /tmp in the script).

How to set the "c_shrink_db_files" variable (before compilation)?

I want only an one time shrink.

c_shrink_db_files is not a compile-time option.  We've worked hard to have as few of those as possible.

If you set the c_shrink_db_files configuration option, the database will be compacted at shutdown time.

If you're not satisfied with the results, I would suggest using ctdldump and ctdlload, not database_cleanup.sh.   But really, Citadel is good with disk space -- it will reuse the "holes" in those files to write new data.  At some point your disk utilization will stabilize if you have both new data coming in and old data being purged.  If you have plenty of disk space and your system is running cleanly I would advise leaving it alone.

There is a problem with BCC recieptiens.

They will all appear in Mail header as > Envelope-to:

Is there a option to hide recieptens in header ?

Can you clarify please, is this on outbound or inbound email?

Dear All,

I want to shrink my databases. I tried with the shell script "database_cleanup.sh" and it gives a error even with citadel shutdown and with plenty of disk space (I renamed /tmp in the script).

How to set the "c_shrink_db_files" variable (before compilation)?

I want only an one time shrink.

Thank you,

Luís Gonçalves.