Thu Feb 20 2025 20:54:48 UTC from IGnatius T Foobar Subject: Re: citadel crashed when

Please enable core dumps on your system. (If you don't know how to do that, check your favorite search engine.)

After the Citadel Server crashes, you can then do this:

cd /usr/local/citadel
gdb ./citserver [path/to/core_file]

The debugger will start up and put you into the server program right after it crashed. Now do this:

thread apply all bt

And post the output of that command. It is also imperative that you are running the latest version. Your problem sounds like one we fixed a couple of versions ago where a corrupt incoming message (usually a spam) causes the server to crash whenever the indexer is run.

Hello!

Here is my core dump log:

root@mail:/usr/local/citadel# coredumpctl debug
          PID: 570622 (citserver)
          UID: 1001 (citadel)
          GID: 1003 (citadel)
       Signal: 6 (ABRT)
    Timestamp: Fri 2025-03-07 09:08:38 MSK (3min 29s ago)
 Command Line: /usr/local/citadel/citserver
   Executable: /usr/local/citadel/citserver
Control Group: /system.slice/citadel.service
         Unit: citadel.service
        Slice: system.slice
      Boot ID: 70b64ff8afd2457181058ad8f9bb4a71
   Machine ID: 3b023f1e7c6d4965a3d0e2f737442736
     Hostname: mail.vodolaga.ru
      Storage: /var/lib/systemd/coredump/core.citserver.1001.70b64ff8afd2457181058ad8f9bb4a71.570622.1741327718000000.zst (present)
 Size on Disk: 542.9K
      Message: Process 570622 (citserver) of user 1001 dumped core.
                
               Stack trace of thread 570624:
               #0  0x00007fc28c71cebc __pthread_kill_implementation (libc.so.6 + 0x8aebc)
               #1  0x00007fc28c6cdfb2 __GI_raise (libc.so.6 + 0x3bfb2)
               #2  0x00007fc28c6b8472 __GI_abort (libc.so.6 + 0x26472)
               #3  0x00007fc28c711430 __libc_message (libc.so.6 + 0x7f430)
               #4  0x00007fc28c72683a malloc_printerr (libc.so.6 + 0x9483a)
               #5  0x00007fc28c72871c _int_free (libc.so.6 + 0x9671c)
               #6  0x00007fc28c72af1f __GI___libc_free (libc.so.6 + 0x98f1f)
               #7  0x000055633ce570ce fixed_output (citserver + 0x230ce)
               #8  0x00007fc28ced0b0e mime_decode (libcitadel.so.4 + 0x8b0e)
               #9  0x00007fc28ced124f recurseable_mime_parser (libcitadel.so.4 + 0x924f)
               #10 0x00007fc28ced0d48 the_mime_parser (libcitadel.so.4 + 0x8d48)
               #11 0x00007fc28ced14c6 mime_parser (libcitadel.so.4 + 0x94c6)
               #12 0x000055633ce5948f CtdlOutputPreLoadedMsg (citserver + 0x2548f)
               #13 0x000055633ce7d9ff ft_index_message (citserver + 0x499ff)
               #14 0x000055633ce7e088 do_fulltext_indexing (citserver + 0x4a088)
               #15 0x000055633ce621b5 PerformSessionHooks (citserver + 0x2e1b5)
               #16 0x000055633ce4d28d do_housekeeping (citserver + 0x1928d)
               #17 0x000055633ce64648 worker_thread (citserver + 0x30648)
               #18 0x00007fc28c71b1c4 start_thread (libc.so.6 + 0x891c4)
               #19 0x00007fc28c79b85c __clone3 (libc.so.6 + 0x10985c)
                
               Stack trace of thread 570622:
               #0  0x00007fc28c761505 __GI___clock_nanosleep (libc.so.6 + 0xcf505)
               #1  0x00007fc28c765e13 __GI___nanosleep (libc.so.6 + 0xd3e13)
               #2  0x00007fc28c791205 usleep (libc.so.6 + 0xff205)
               #3  0x000055633ce64b43 go_threading (citserver + 0x30b43)
               #4  0x000055633ce61519 main (citserver + 0x2d519)
               #5  0x00007fc28c6b924a __libc_start_call_main (libc.so.6 + 0x2724a)
               #6  0x00007fc28c6b9305 __libc_start_main_impl (libc.so.6 + 0x27305)
               #7  0x000055633ce45e51 _start (citserver + 0x11e51)
                
               Stack trace of thread 570623:
               #0  0x00007fc28c79095c __GI___select (libc.so.6 + 0xfe95c)
               #1  0x000055633ce64161 worker_thread (citserver + 0x30161)
               #2  0x00007fc28c71b1c4 start_thread (libc.so.6 + 0x891c4)
               #3  0x00007fc28c79b85c __clone3 (libc.so.6 + 0x10985c)
               ELF object binary architecture: AMD x86-64

GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
   <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/citadel/citserver...
[New LWP 570624]
[New LWP 570622]
[New LWP 570623]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/citadel/citserver'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7fc288b8c6c0 (LWP 570624))]
(gdb) thread apply all bt

Thread 3 (Thread 0x7fc28938d6c0 (LWP 570623)):
#0  0x00007fc28c79095c in __GI___select (nfds=39, readfds=0x7fc28938cbc0, writefds=0x0, exceptfds=0x0, timeout=0x7fc28938cbb0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x000055633ce64161 in worker_thread (blah=0x0) at server/sysdep.c:811
#2  0x00007fc28c71b1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#3  0x00007fc28c79b85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7fc289c1a980 (LWP 570622)):
#0  0x00007fc28c761505 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffe48735c60, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
#1  0x00007fc28c765e13 in __GI___nanosleep (req=req@entry=0x7ffe48735c60, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007fc28c791205 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3  0x000055633ce64b43 in go_threading () at server/threads.c:137
#4  0x000055633ce61519 in main (argc=1, argv=0x7ffe48736e68) at server/server_main.c:297

Thread 1 (Thread 0x7fc288b8c6c0 (LWP 570624)):
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007fc28c71cf1f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007fc28c6cdfb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007fc28c6b8472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007fc28c711430 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fc28c82b459 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007fc28c72683a in malloc_printerr (str=str@entry=0x7fc28c82e180 "free(): invalid next size (normal)") at ./malloc/malloc.c:5660
#6  0x00007fc28c72871c in _int_free (av=0x7fc284000030, p=0x7fc284026a30, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4592
#7  0x00007fc28c72af1f in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385
#8  0x000055633ce570ce in fixed_output (name=0x7fc2840344e0 "", filename=0x7fc2840364f0 "", partnum=0x7fc28cee3103 "1", disp=0x7fc2840374f8 "", content=0x7fc284039520, cbtype=0x7fc2840314c8 "text/html", cbcharset=0x7fc28
40324d0 "utf-8", length=42916, encoding=0x7fc288b892a0 "binary", cbid=0x7fc284038500 "", cbuserdata=0x7fc288b8b990) at server/msgbase.c:1177
#9  0x00007fc28ced0b0e in mime_decode (partnum=0x7fc28cee3622 "", part_start=0x7fc284015f39 "\n<!DOCTYPE html=0A    PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http:=\n//www.w3.org/TR/xhtml1/DTD/xhtml1-transition
al.dtd\">=0A<html xmlns=3D\"http:=\n//www.w3.org/1999/xhtml\">=0A=0A<head>=0A    <"..., length=<optimized out>, content_type=0x7fc2840314c8 "text/html", charset=0x7fc2840324d0 "utf-8", encoding=<optimized out>, dispositi
on=0x7fc2840374f8 "", id=0x7fc284038500 "", name=0x7fc2840344e0 "", filename=0x7fc2840364f0 "", CallBack=0x55633ce56f00 <fixed_output>, PreMultiPartCallBack=0x0, PostMultiPartCallBack=0x0, userdata=0x7fc288b8b990, dont_d
ecode=0) at lib/mime_parser.c:238
#10 0x00007fc28ced124f in recurseable_mime_parser (partnum=partnum@entry=0x7fc28cee3622 "", content_start=0x7fc284015f39 "\n<!DOCTYPE html=0A    PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http:=\n//www.w3.org/TR
/xhtml1/DTD/xhtml1-transitional.dtd\">=0A<html xmlns=3D\"http:=\n//www.w3.org/1999/xhtml\">=0A=0A<head>=0A    <"..., content_end=content_end@entry=0x7fc284022933 "", CallBack=CallBack@entry=0x55633ce56f00 <fixed_output>,
PreMultiPartCallBack=PreMultiPartCallBack@entry=0x55633ce56dac <fixed_output_pre>, PostMultiPartCallBack=PostMultiPartCallBack@entry=0x55633ce56e56 <fixed_output_post>, userdata=0x7fc288b8b990, dont_decode=0, m=0x7fc284
02e4b0) at lib/mime_parser.c:712
#11 0x00007fc28ced0d48 in the_mime_parser (partnum=partnum@entry=0x7fc28cee3622 "", content_start=<optimized out>, content_end=0x7fc284022933 "", CallBack=0x55633ce56f00 <fixed_output>, PreMultiPartCallBack=0x55633ce56da
c <fixed_output_pre>, PostMultiPartCallBack=0x55633ce56e56 <fixed_output_post>, userdata=0x7fc288b8b990, dont_decode=0) at lib/mime_parser.c:811
#12 0x00007fc28ced14c6 in mime_parser (content_start=<optimized out>, content_end=<optimized out>, CallBack=<optimized out>, PreMultiPartCallBack=<optimized out>, PostMultiPartCallBack=<optimized out>, userdata=<optimize
d out>, dont_decode=0) at lib/mime_parser.c:836
#13 0x000055633ce5948f in CtdlOutputPreLoadedMsg (TheMessage=0x7fc284025a20, mode=0, headers_only=0, do_proto=0, crlf=1, flags=0) at server/msgbase.c:2074
#14 0x000055633ce7d9ff in ft_index_message (msgnum=14539, op=1) at server/modules/fulltext/serv_fulltext.c:86
#15 0x000055633ce7e088 in do_fulltext_indexing () at server/modules/fulltext/serv_fulltext.c:242
#16 0x000055633ce621b5 in PerformSessionHooks (EventType=12) at server/serv_extensions.c:439
#17 0x000055633ce4d28d in do_housekeeping () at server/housekeeping.c:127
#18 0x000055633ce64648 in worker_thread (blah=0x0) at server/sysdep.c:951
#19 0x00007fc28c71b1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x00007fc28c79b85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)

Sat Mar 08 2025 20:50:06 UTC from IGnatius T Foobar Subject: Re: citadel crashed when

 If you can manage to save a copy of the original RFC822 message that'd do it quickly, but we'll find it one way or another.

How to save this message from DB?

Tue Feb 25 2025 16:31:07 UTC from IGnatius T Foobar Subject: Re: Setup queries

Also should the citadel binary have been built? When I compile 1009 I'm not getting one.

The 'citadel' binary is the text mode client.  If you are manually compiling the components of the Citadel system, you'll find that build in the textclient/ directory.

I've got no textclient directory, nothing in the build directory called 'citadel'.

What ist best pratice for journaling mails ? Wich Mail Storage is useful ?

Mailstore or other programs who can handle journaling mails from citadel ?

Cheers

Dear Maintainers,

I am being under attack since for two days now.

I start citadel and some time after appears .logs files in the data directory until disk full and citadel become unusable.

Please give me a solution. This way citadel become unusable.

What you need? The log files?

Thanks,

Luís.

Not that it helps you any, but that is one of the main reasons i shut down all my public facing sites and now even my personal stuff has to go thru a password protected proxy, THEN a 2nd password in the app.

I let the proxy deal with the flood instead of my actual apps.  Safer that way.

For a while it was pulled back to vpn only..   but the flood subsided enough to do the proxy instead. 

Fri Mar 14 2025 08:06:54 UTC from luisgo Subject: Continually under attack.

Dear Maintainers,

I am being under attack since for two days now.

I start citadel and some time after appears .logs files in the data directory until disk full and citadel become unusable.

Please give me a solution. This way citadel become unusable.

What you need? The log files?

Thanks,

Luís.

 

Sun Mar 16 2025 03:37:41 UTC from IGnatius T Foobar Subject: Re: citadel crashed when

How to save this message from DB?

If you are able to get your Citadel server running for a little while, you can just do

cd /usr/local/citadel
./sendcommand MSG2 14539

But if you can't, you might try extracting it from the database:

cd /usr/local/citadel
./ctdldump -y | grep 'msgtext|14539'

The first method did not help to extract the message, not a single one. I indicated any number, the answer was "not found".

The second method received a piece of some text encrypted with base64. I decrypted it, there are pieces of some e-mails.

Where can I send this fragment? I do not want to show personal data on the forum.

Citadel Server has deleted 32571 users, 0 messages, 1848174144 rooms, 32571
visit records, 1902681392 use table entries, and 0 EUID indices due to expire
policy set on those objects.

Message in Aide. What I must do?

Sun Mar 16 2025 03:40:25 UTC from IGnatius T Foobar Subject: Re: Continually under attack.

data directory until disk full and citadel become unusable.

Please give me a solution. This way citadel become unusable.

You're either being spammed hard or someone has acquired the password to an account on your system. Didn't this happen to you before? I wonder if maybe the account they used didn't get locked down?

Really the only way to find out what's going on is to watch your syslogs and see what citserver is doing.

I changed the password of "Admin" users and I cheked "Disable self-service user account creation; Hint: do not select both!"

Let see what happens.

Mon Mar 17 2025 17:14:22 UTC from IGnatius T Foobar Subject: Re: Continually under attack.

  Citadel Server has deleted 32571 users, 0 messages,
1848174144 rooms, 32571 visit records, 1902681392 use table
entries, and 0 EUID indices due to expire policy set on those
objects.   Message in Aide. What I must do?

Lock down your system. This strongly suggests that someone is in there and is using your system to try and relay messages or something like that. Maybe you can watch the syslog to see what account is being used, but if they compromised an admin account they might be able to create more accounts.

If you're not running a public site, be sure to disable self service account creation. If you are running a public site, look around during operation and see who's logged in. Definitely don't allow new accounts permission to send Internet email if you're running an open site.

If there's a problem with the software we can find it, but this really just sounds like someone is in there who's not supposed to be, and the software is doing what it's built to do.

I don't actually have any specific questions at this moment, but wanted to say that I'm here.  I just setup a new Citadel instance on my network, to replace a ridiculously difficult to keep running email server, and it just instantly took over and is handling all incoming mail like an absolute champ.  I even configured it for multiple domain handling, and found it to be incredibly easy to do so. 
I have been a Citadel user since the mid 80's, even wrote one once.  I've been known across the entire multiverse of Citadels as either "The Magician" or "PanaSonic" for the most part.  Since someone I ran into recently actually called me out by the latter, that's what I chose to use here, today.

Greetings to all the Citadel 'ops and users.

I'm sure the questions will be flowing soon.

sure, you can use that.. 

I just hit "Reply Quoted" in here, to attempt to reply to your message, and instead it skipped to the next room, and lost my Ungoto history.

A new interface would be welcome, this version of WebCit seems particularly lacking.  I guess my first question, then, will be, can I try out said new interface, and contribute to it? :-D

Presently just using it for an incoming email server, I need to figure out how to deal with outgoing... then i want to test it's email/calendar/contacts services with some particularly old legacy devices to see if it's compatible.

That it "just works" as an incoming email server is a big plus, though.  I had a more difficult time getting it running in docker than I did setting up the email lol

I have seen that too.. but not in a reproducible manner. 

Mon Mar 24 2025 15:48:09 UTC from PanaSonic Subject: Re: hello there

 

I just hit "Reply Quoted" in here, to attempt to reply to your message, and instead it skipped to the next room, and lost my Ungoto history.