I send you priv messages.

J.K.

Subject pretty much says it all: Are modems still supported in Citadel?

SP

Dear All,

I submit another coredump and this time I include the Citadel server binary.

https://luisgo.pro/down/coredumps5.zip

Thank you,

Luís Gonçalves.

https://luisgo.pro/down/coredumps6.zip

Including citserver 

Centos 9, last version.

Specify please what you want to solve the problem. Do you want to have access to my server? Stack trace? How I obtain it?

Sat Sep 20 2025 18:08:00 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

Sorry again.

This night happened again. More coredumps. I updated the citadel. It installed and compiled a version but it remained the same version number. Did that attended my problem?

If not a send the new coredumps.

https://luisgo.pro/down/coredumps3.zip

I hope that you solved or are solving this problem.

Let see that if or not happen again.

Unfortunately, your core dumps are not useful to anyone because they are specific to your citserver binary.  I suppose if you include your citserver binary it might be useful, otherwise a stack trace is better.

In your case, we're just seeing so many weird things on your system that don't happen anywhere else.  It's difficult to pin down why.  I would strongly suggest that you give some consideration to moving to the containerized version of the Citadel system.  It is lovingly built for you in a carefully controlled environment.   Perhaps switch to the container version, and if necessary, do a ctdldump/ctdlload on your existing data.

And of course take lots of backups before doing anything.

Specify please what you want to solve the problem. Do you want to have access to my server? Stack trace? How I obtain it?

Stack trace would be far more helpful than what you are currently trying to do.   Since you seem to have the ability to produce core dumps, you are halfway there already.

After a core dump, do this:

gdb /usr/local/citadel/citserver core.xxxxxxxxx   (use the name of your core file)

The debugger will open, and show you where it crashed.   Now type:

thread apply all bt

And post the output.  it should show lots of filenames and line numbers.

Hi,

two problems to solve - could you please give me a tip:

1. After sending email, massage goes to SMTP queue. I should expect that it will be send stright after. In my server email will be send after 8-10minutes. There is big delay. I notice also that:

next attempt to send have date "sty 01 1970". Server date is correct. Is this connected?

2. After my problem with server hang (few weekes ago I send here information about that) I notice that in "Aida" directory there are planty of meassages to today. Server is generating it reapetly after few minutes:

How I can remove this from queue? Where is is coz it should be somewhere and can not find it.

J.K.

navigate to https://easyinstall.citadel.org/

you'll find the site is down

i am trying to install citadel using sudo curl https://easyinstall.citadel.org/install | bash

Instructions for which i found on https://www.citadel.org/easyinstall.html

anyone know why site is down

also, is a copy of the easyinstall script available elsewhere?

thanks - ericsondatasystems

 don’t see those messages in that queue. Previously, I had to remove those messages directly from the DB because Citadel was crashing with a segmentation fault. Is it possible I didn’t remove them from all the required places?

In the Citadel console I also see:

paź 15 14:55:40 xxx citserver[328259]: citserver[328259]: msgbase: message #11040 was not found

paź 15 14:55:40 xxx citserver[328259]: msgbase: message #11040 was not found

Could these be ghost/orphaned messages?

śro paź 15 2025 00:36:59 UTC od IGnatius T Foobar Temat: Re: smtp queue

2. After my problem with server hang (few weekes ago I send here
information about that) I notice that in "Aida" directory there are
planty of meassages to today. Server is generating it reapetly after few

minutes:

Sounds like someone dumped a big pile of boomerang spam at your site, maybe?

The queue is in a hidden room called __CitadelSMTPspoolout__

You can delete from there.

SUMMARY

-------

Citadel Server (Version 1020, build 25289) crashes with SIGSEGV when processing 

emails containing malformed HTML with missing opening angle brackets. The server 

enters an unrecoverable crash loop on startup, requiring manual database 

intervention to restore service.

SEVERITY: CRITICAL

------------------

• Impact: Complete server unavailability (crash loop)

• Reproducibility: 100% reproducible

• Recovery: Requires manual database manipulation

• Root Cause: NULL pointer dereference in libcitadel.so.4

ENVIRONMENT

-----------

Citadel Version: 1020 (build 25289)

libcitadel: libcitadel.so.4 (unnumbered)

OS: Ubuntu 24 (Linux x86_64)

Crash Location: lib/html_to_ascii.c:161

PROBLEM DESCRIPTION

-------------------

When Citadel receives email containing HTML where the opening '<' character is 

missing from a tag, the html_to_ascii() function crashes with a NULL pointer 

dereference. This occurs during full-text indexing at server startup, creating 

a crash loop that prevents the server from running.

WE ARE RECEIVING THESE MALFORMED EMAILS REGULARLY, making this a recurring production issue.

TECHNICAL DETAILS

-----------------

Stack Trace:

    SIGSEGV received at address (nil)

    #0  __strncpy_sse2_unaligned ()

    #1  strncpy (__len=1024, __src=0x0, __dest=...)

    #2  html_to_ascii (...) at lib/html_to_ascii.c:161

        tag_start = 0x0  <-- NULL POINTER

        tag_end = 0x... ">Hello,</p>..."

    #3  fixed_output (...) at server/msgbase.c:1185

    #4  mime_decode (...) at lib/mime_parser.c:224

    ...

    #10 ft_index_message (msgnum=..., op=1) 

        at server/modules/fulltext/serv_fulltext.c:88

Root Cause:

In html_to_ascii() at line 161:

1. Function finds tag_end pointing to a '>' character

2. Attempts to find the matching '<' character for tag_start

3. When the '<' is missing, tag_start remains NULL

4. Calls strncpy(tag, tag_start, 1024) with NULL source pointer

5. SEGMENTATION FAULT (sigsegv)

Malformed HTML Example (triggers crash):

    p>Hello,</p>

    <p>I have information that is privileged...</p>

Should be:

    <p>Hello,</p>

    <p>I have information that is privileged...</p>

Note: The HTML is otherwise well-formed - only the opening '<' is missing.

STEPS TO REPRODUCE

------------------

1. Send email to Citadel with HTML body starting with text followed by '>' 

   (missing opening '<')

2. Restart Citadel server or wait for full-text indexing

3. Server crashes during message indexing

4. Server enters crash loop - crashes on every restart attempt

5. systemd reports: "citadel.service: Failed with result 'core-dump'"

EXPECTED BEHAVIOR

-----------------

Server should handle malformed HTML gracefully by either:

• Skipping malformed tags and continuing processing

• Logging warning and indexing message without HTML conversion

• Failing safely without crashing

ACTUAL BEHAVIOR

---------------

• Server crashes with SIGSEGV

• Enters unrecoverable crash loop

• Cannot start until problematic message is manually removed

• Service completely unavailable

IMPACT ASSESSMENT

-----------------

Business Impact: SEVERE

• Complete service outage - no email access during crash loop

• Manual intervention required - cannot auto-recover

• Data loss risk - requires database manipulation to fix

• Recurring issue - we have recently started to receive these malformed emails regularly

WORKAROUND

----------

The only current workaround is to manually delete the full-text index files:

    systemctl stop citadel

    rm -f /usr/local/citadel/data/cdb.0c    # Full-text index database

    rm -f /usr/local/citadel/data/cdb.0d    # Index control file

    systemctl start citadel

This forces Citadel to rebuild the full-text index from scratch. During the 

rebuild process, the indexer appears to handle the malformed message differently 

(possibly with better error handling) and skips it, allowing the server to start.

However, this workaround:

• Requires root access

• Causes service downtime

• Forces complete index rebuild (time-consuming on large databases)

• Temporarily loses full-text search capability during rebuild

• Only works until the next malformed email arrives

• Is NOT a sustainable solution when receiving these emails regularly

ATTEMPTED SOLUTIONS

-------------------

  Berkeley DB recovery (db_recover -c) - Did not resolve crash

  Citadel database cleanup script (database_cleanup.sh) - Performed full 

  Berkeley DB dump/load of ALL database files (cdb.00 through cdb.0d), removed 

  old databases and transaction logs, successfully reloaded everything - CRASH 

  PERSISTED UNCHANGED

  Database dump/reload manually (db_dump | db_load) - Preserved the malformed 

  message, crash persisted

  Full-text index deletion (cdb.0c, cdb.0d) - Allows server to start by 

  forcing index rebuild

Note: The fact that a complete database cleanup (dump/load of all 14 database 

files) did not resolve the issue demonstrates that:

1. The database files themselves are not corrupted

2. The malformed HTML content is preserved exactly through dump/load

3. Standard database recovery procedures are ineffective for this bug

4. The issue is with the content being valid Berkeley DB data but containing 

   HTML that crashes the parser

PROPOSED FIX

------------

The html_to_ascii() function should validate tag_start before use:

    // Around line 161 in lib/html_to_ascii.c

    if (tag_start == NULL || tag_end == NULL) {

        // Handle malformed HTML gracefully

        // Log warning, skip tag, or use safe default

        continue;  // or appropriate error handling

    }

    strncpy(tag, tag_start, tag_len);

Additionally, consider:

1. Input validation before HTML parsing

2. Defensive coding for all pointer operations in parser

3. Graceful degradation when HTML is malformed

REQUEST

-------

This is a CRITICAL PRODUCTION ISSUE causing recurring outages. We urgently 

request:

1. A patch for the NULL pointer check in html_to_ascii()

2. Guidance on additional defensive measures

3. Consider making full-text indexing more resilient to malformed input

Thank you for your attention to this severe bug.

ericsondatasystems

Dear All,

This is happening to me. 

I think that is happening since the last upgrade (some days ago) when I send a message it has the date one hour after I sent.

I only noticed now. I upgraded at 6 Nov, 17H39 Central Europe Time. I can verify in some emails in the "Sent Items" after that (6 Nov) are registered one hour after than when sent. 

Happens with all the emails clients.

Are this happening to someone else?

Thank you,

Luís Gonçalves.

Dear All,

Concerning this problem I improved the realibility of Citadel doing the following.

I noticed that the available memory (with "top") was not too much (some tens of Megabytes) and it had a lot of Buffer/Cache Memory. And the Swap has about 200 MBytes or more.

I suspected that when Citadel needed memory was not freed fast enough. Then I remembered that in Apache, I has configured disk cache with some time out to free it.

I disallowed the disk cache in Apache and the time out. I noticed that now I have more free memory (hundreds od Megabytes) and the swap is almost not used. Some times the free memory goes to more than a GByte showing that memory is freed.

Thank you,

Luís Gonçalves.   

Wed Oct 01 2025 06:46:58 UTC from luisgo Subject: Re: Concerning tha last two posts in Support Room.

Specify please what you want to solve the problem. Do you want to have access to my server? Stack trace? How I obtain it?

 

Sat Sep 20 2025 18:08:00 UTC from IGnatius T Foobar Subject: Re: Concerning tha last two posts in Support Room.

Sorry again.

This night happened again. More coredumps. I updated the citadel. It installed and compiled a version but it remained the same version number. Did that attended my problem?

If not a send the new coredumps.

https://luisgo.pro/down/coredumps3.zip

I hope that you solved or are solving this problem.

Let see that if or not happen again.

Unfortunately, your core dumps are not useful to anyone because they are specific to your citserver binary.  I suppose if you include your citserver binary it might be useful, otherwise a stack trace is better.

In your case, we're just seeing so many weird things on your system that don't happen anywhere else.  It's difficult to pin down why.  I would strongly suggest that you give some consideration to moving to the containerized version of the Citadel system.  It is lovingly built for you in a carefully controlled environment.   Perhaps switch to the container version, and if necessary, do a ctdldump/ctdlload on your existing data.

And of course take lots of backups before doing anything.