SUMMARY

-------

Citadel Server (Version 1020, build 25289) crashes with SIGSEGV when processing 

emails containing malformed HTML with missing opening angle brackets. The server 

enters an unrecoverable crash loop on startup, requiring manual database 

intervention to restore service.

SEVERITY: CRITICAL

------------------

• Impact: Complete server unavailability (crash loop)

• Reproducibility: 100% reproducible

• Recovery: Requires manual database manipulation

• Root Cause: NULL pointer dereference in libcitadel.so.4

ENVIRONMENT

-----------

Citadel Version: 1020 (build 25289)

libcitadel: libcitadel.so.4 (unnumbered)

OS: Ubuntu 24 (Linux x86_64)

Crash Location: lib/html_to_ascii.c:161

PROBLEM DESCRIPTION

-------------------

When Citadel receives email containing HTML where the opening '<' character is 

missing from a tag, the html_to_ascii() function crashes with a NULL pointer 

dereference. This occurs during full-text indexing at server startup, creating 

a crash loop that prevents the server from running.

WE ARE RECEIVING THESE MALFORMED EMAILS REGULARLY, making this a recurring production issue.

TECHNICAL DETAILS

-----------------

Stack Trace:

    SIGSEGV received at address (nil)

    #0  __strncpy_sse2_unaligned ()

    #1  strncpy (__len=1024, __src=0x0, __dest=...)

    #2  html_to_ascii (...) at lib/html_to_ascii.c:161

        tag_start = 0x0  <-- NULL POINTER

        tag_end = 0x... ">Hello,</p>..."

    #3  fixed_output (...) at server/msgbase.c:1185

    #4  mime_decode (...) at lib/mime_parser.c:224

    ...

    #10 ft_index_message (msgnum=..., op=1) 

        at server/modules/fulltext/serv_fulltext.c:88

Root Cause:

In html_to_ascii() at line 161:

1. Function finds tag_end pointing to a '>' character

2. Attempts to find the matching '<' character for tag_start

3. When the '<' is missing, tag_start remains NULL

4. Calls strncpy(tag, tag_start, 1024) with NULL source pointer

5. SEGMENTATION FAULT (sigsegv)

Malformed HTML Example (triggers crash):

    p>Hello,</p>

    <p>I have information that is privileged...</p>

Should be:

    <p>Hello,</p>

    <p>I have information that is privileged...</p>

Note: The HTML is otherwise well-formed - only the opening '<' is missing.

STEPS TO REPRODUCE

------------------

1. Send email to Citadel with HTML body starting with text followed by '>' 

   (missing opening '<')

2. Restart Citadel server or wait for full-text indexing

3. Server crashes during message indexing

4. Server enters crash loop - crashes on every restart attempt

5. systemd reports: "citadel.service: Failed with result 'core-dump'"

EXPECTED BEHAVIOR

-----------------

Server should handle malformed HTML gracefully by either:

• Skipping malformed tags and continuing processing

• Logging warning and indexing message without HTML conversion

• Failing safely without crashing

ACTUAL BEHAVIOR

---------------

• Server crashes with SIGSEGV

• Enters unrecoverable crash loop

• Cannot start until problematic message is manually removed

• Service completely unavailable

IMPACT ASSESSMENT

-----------------

Business Impact: SEVERE

• Complete service outage - no email access during crash loop

• Manual intervention required - cannot auto-recover

• Data loss risk - requires database manipulation to fix

• Recurring issue - we have recently started to receive these malformed emails regularly

WORKAROUND

----------

The only current workaround is to manually delete the full-text index files:

    systemctl stop citadel

    rm -f /usr/local/citadel/data/cdb.0c    # Full-text index database

    rm -f /usr/local/citadel/data/cdb.0d    # Index control file

    systemctl start citadel

This forces Citadel to rebuild the full-text index from scratch. During the 

rebuild process, the indexer appears to handle the malformed message differently 

(possibly with better error handling) and skips it, allowing the server to start.

However, this workaround:

• Requires root access

• Causes service downtime

• Forces complete index rebuild (time-consuming on large databases)

• Temporarily loses full-text search capability during rebuild

• Only works until the next malformed email arrives

• Is NOT a sustainable solution when receiving these emails regularly

ATTEMPTED SOLUTIONS

-------------------

  Berkeley DB recovery (db_recover -c) - Did not resolve crash

  Citadel database cleanup script (database_cleanup.sh) - Performed full 

  Berkeley DB dump/load of ALL database files (cdb.00 through cdb.0d), removed 

  old databases and transaction logs, successfully reloaded everything - CRASH 

  PERSISTED UNCHANGED

  Database dump/reload manually (db_dump | db_load) - Preserved the malformed 

  message, crash persisted

  Full-text index deletion (cdb.0c, cdb.0d) - Allows server to start by 

  forcing index rebuild

Note: The fact that a complete database cleanup (dump/load of all 14 database 

files) did not resolve the issue demonstrates that:

1. The database files themselves are not corrupted

2. The malformed HTML content is preserved exactly through dump/load

3. Standard database recovery procedures are ineffective for this bug

4. The issue is with the content being valid Berkeley DB data but containing 

   HTML that crashes the parser

PROPOSED FIX

------------

The html_to_ascii() function should validate tag_start before use:

    // Around line 161 in lib/html_to_ascii.c

    if (tag_start == NULL || tag_end == NULL) {

        // Handle malformed HTML gracefully

        // Log warning, skip tag, or use safe default

        continue;  // or appropriate error handling

    }

    strncpy(tag, tag_start, tag_len);

Additionally, consider:

1. Input validation before HTML parsing

2. Defensive coding for all pointer operations in parser

3. Graceful degradation when HTML is malformed

REQUEST

-------

This is a CRITICAL PRODUCTION ISSUE causing recurring outages. We urgently 

request:

1. A patch for the NULL pointer check in html_to_ascii()

2. Guidance on additional defensive measures

3. Consider making full-text indexing more resilient to malformed input

Thank you for your attention to this severe bug.

ericsondatasystems