Citadel Support

[#] Tue Sep 26 2023 12:49:41 EDT from IGnatius T Foobar

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

Perhaps we can get more information.

First, flog yourself with a wet noodle for disregarding common sense about taking backups. You should be backing up every day, not just when you're about to perform an upgrade. Use a filesystem like `btrfs` or `zfs` which can make instant snapshots, then rsync those snapshots to a backup location.

Then shut down the systemd unit file for Citadel (`systemctl stop citadel`).
It's ok to leave the web interface running.

Run the Citadel Server in the foreground with maximum debugging:

cd /usr/local/citadel
./citserver -x9

If there is really a problem with the server software, it will crash with maximum logging right in front of you. You can post the last screenful or so of the output and perhaps it will tell us more.

For "even more" information, you can try running it in the debugger:

cd /usr/local/citadel
gdb ./citserver
run -x9

If it crashes, post the last screenful up to the crash, then do `thread apply all bt` and stash that output too, just in case we need it.

[#] Tue Sep 26 2023 15:19:09 EDT from cjonline

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

./citserver -x9

citserver[1776]: extensions: finished initializing modules (threading=1)

citserver[1776]: sysdep: new client socket 39

citserver[1776]: fulltext: indexing started. msgs 6996--7111

citserver[1776]: msgbase: CtdlFetchMessage(6997, 1)

citserver[1776]: fulltext: ft_index_message() adding msg 6997

citserver[1776]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 0, 0, 0, 1

citserver[1776]: msgbase: fixed_output() part 1: (text/html) (35175 bytes)

citserver[1776]: context: session (SMTPs-MTA) started from 46.148.40.151 (46.148.40.151) uid=-1

citserver[1776]: crypto: using certificate chain keys/citadel.cer

citserver[1776]: crypto: using private key keys/citadel.key

citserver[1776]: fulltext: wordbreaking message 6997 (79 bytes)

citserver[1776]: fulltext: indexing message 6997 [0 tokens]

citserver[1776]: msgbase: CtdlFetchMessage(6998, 1)

citserver[1776]: fulltext: ft_index_message() adding msg 6998

citserver[1776]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 0, 0, 0, 1

citserver[1776]: msgbase: fixed_output() part 1: (text/html) (53736 bytes)

citserver[1776]: fulltext: wordbreaking message 6998 (0 bytes)

Segmentation fault

------------

gdb ./citserver
run -x9

citserver[2086]: extensions: finished initializing modules (threading=1)

[New Thread 0xb3f5c200 (LWP 2089)]

[New Thread 0xb3e5b200 (LWP 2090)]

citserver[2086]: fulltext: indexing started. msgs 6996--7111

citserver[2086]: msgbase: CtdlFetchMessage(6997, 1)

citserver[2086]: fulltext: ft_index_message() adding msg 6997

citserver[2086]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 0, 0, 0, 1

citserver[2086]: msgbase: fixed_output() part 1: (text/html) (35175 bytes)

citserver[2086]: fulltext: wordbreaking message 6997 (79 bytes)

citserver[2086]: fulltext: indexing message 6997 [0 tokens]

citserver[2086]: msgbase: CtdlFetchMessage(6998, 1)

citserver[2086]: fulltext: ft_index_message() adding msg 6998

citserver[2086]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 0, 0, 0, 1

citserver[2086]: msgbase: fixed_output() part 1: (text/html) (53736 bytes)

citserver[2086]: fulltext: wordbreaking message 6998 (0 bytes)

Thread 2 "citserver" received signal SIGSEGV, Segmentation fault.

[Switching to Thread 0xb3f5c200 (LWP 2089)]

array_len (arr=0x0) at lib/array.c:71

71 lib/array.c: No such file or directory.

--------
thread apply all bt

Thread 3 (Thread 0xb3e5b200 (LWP 2090)):

#0 __GI___select (timeout=0xbeffe4ea, exceptfds=0x0, writefds=0x0, readfds=0xb3e5ab54, nfds=39) at ../sysdeps/unix/sysv/linux/select.c:41

#1 __GI___select (nfds=39, readfds=0xb3e5ab54, writefds=0x0, exceptfds=0x0, timeout=0xb3e5ab4c) at ../sysdeps/unix/sysv/linux/select.c:37

#2 0x000278f8 in worker_thread (blah=0x0) at server/sysdep.c:730

#3 0xb6c98494 in start_thread (arg=0xb3e5b200) at pthread_create.c:486

#4 0xb6a3be98 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 0xb3f5c200 (LWP 2089)):

#0 array_len (arr=0x0) at lib/array.c:71

#1 0x00059000 in ft_index_message (msgnum=6998, op=1) at server/modules/fulltext/serv_fulltext.c:97

#2 0x00059604 in do_fulltext_indexing () at server/modules/fulltext/serv_fulltext.c:237

#3 0x00039474 in PerformSessionHooks (EventType=12) at server/serv_extensions.c:469

#4 0x00037278 in do_housekeeping () at server/housekeeping.c:130

#5 0x00027e90 in worker_thread (blah=0x0) at server/sysdep.c:876

#6 0xb6c98494 in start_thread (arg=0xb3f5c200) at pthread_create.c:486

#7 0xb6a3be98 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0xb6fefe00 (LWP 2086)):

#0 __GI___nanosleep (remaining=0x0, requested_time=0xbeffe564) at ../sysdeps/unix/sysv/linux/nanosleep.c:28

#1 __GI___nanosleep (requested_time=0xbeffe564, requested_time@entry=0xbeffe55c, remaining=remaining@entry=0x0)

at ../sysdeps/unix/sysv/linux/nanosleep.c:25

#2 0xb6a358e0 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:32

#3 0x00036b18 in go_threading () at server/threads.c:96

#4 0x00025888 in main (argc=2, argv=0xbefff744) at server/server_main.c:293

[#] Wed Sep 27 2023 00:28:29 EDT from IGnatius T Foobar

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

All right, this is your lucky day. That was an easy problem to spot and I've published a fix.

Somehow you got a couple of zero-length messages into your system (how?) and the indexer choked on a null word list. Go ahead and do the upgrade again.
Make sure you get at least version 995 of the server.

And start taking daily backups. :)

[#] Wed Sep 27 2023 05:23:46 EDT from cjonline

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

Hi, Still not working.. same issue -

DB version is 995

----

citserver[18815]: msgbase: fixed_output() part 1: (text/html) (53736 bytes)

citserver[18815]: context: session (SMTP-MSA) started from 77.90.185.59 (77.90.185.59) uid=-1

citserver[18815]: fulltext: wordbreaking message 6998 (0 bytes)

Segmentation fault

----

citserver[18848]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 0, 0, 0, 1

citserver[18848]: msgbase: fixed_output() part 1: (text/html) (53736 bytes)

citserver[18848]: fulltext: wordbreaking message 6998 (0 bytes)

Thread 2 "citserver" received signal SIGSEGV, Segmentation fault.

[Switching to Thread 0xb3f5c200 (LWP 18854)]

array_len (arr=0x0) at lib/array.c:71

71 lib/array.c: No such file or directory.

-----

Thread 2 (Thread 0xb3f5c200 (LWP 18854)):

#0 array_len (arr=0x0) at lib/array.c:71

#1 0x0005900c in ft_index_message (msgnum=6998, op=1) at server/modules/fulltext/serv_fulltext.c:97

#2 0x00059624 in do_fulltext_indexing () at server/modules/fulltext/serv_fulltext.c:241

#3 0x00039480 in PerformSessionHooks (EventType=12) at server/serv_extensions.c:469

#4 0x00037284 in do_housekeeping () at server/housekeeping.c:130

#5 0x00027ec8 in worker_thread (blah=0x0) at server/sysdep.c:876

#6 0xb6c98494 in start_thread (arg=0xb3f5c200) at pthread_create.c:486

#7 0xb6a3be98 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0xb6fefe00 (LWP 18848)):

#0 __GI___nanosleep (remaining=0x0, requested_time=0xbeffe564) at ../sysdeps/unix/sysv/linux/nanosleep.c:28

#1 __GI___nanosleep (requested_time=0xbeffe564, requested_time@entry=0xbeffe55c, remaining=remaining@entry=0x0)

at ../sysdeps/unix/sysv/linux/nanosleep.c:25

#2 0xb6a358e0 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:32

#3 0x00036b50 in go_threading () at server/threads.c:96

#4 0x000258c0 in main (argc=2, argv=0xbefff744) at server/server_main.c:295

Wed Sep 27 2023 00:28:29 EDT from IGnatius T Foobar Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

All right, this is your lucky day. That was an easy problem to spot and I've published a fix.

Somehow you got a couple of zero-length messages into your system (how?) and the indexer choked on a null word list. Go ahead and do the upgrade again.
Make sure you get at least version 995 of the server.

And start taking daily backups. :)

[#] Thu Sep 28 2023 05:19:24 EDT from cjonline

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

Hi IG,

please can you help ?

Craig.

[#] Thu Sep 28 2023 07:26:19 EDT from Nurb432

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

He will. Hes got a day job remember :)

Thu Sep 28 2023 05:19:24 EDT from cjonline Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

Hi IG,

please can you help ?

Craig.

[#] Thu Sep 28 2023 22:21:22 EDT from IGnatius T Foobar

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

All right, one more try. Make sure you get version 996 of the server.

That fix we put in to get around your mysterious null message needed to be a few lines earlier in the code. I hope this fixes it.

[#] Fri Sep 29 2023 05:40:59 EDT from cjonline

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

Thank you so much! that fixed my issue.. will create backups now going forward.

You are the best!

Craig.

Thu Sep 28 2023 22:21:22 EDT from IGnatius T Foobar Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

All right, one more try. Make sure you get version 996 of the server.

That fix we put in to get around your mysterious null message needed to be a few lines earlier in the code. I hope this fixes it.

[#] Fri Sep 29 2023 16:48:00 EDT from IGnatius T Foobar

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

Spread the word!

[#] Fri Sep 29 2023 17:24:33 EDT from LadySerenaKitty

Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

Word the spread!

Fri Sep 29 2023 16:48:00 EDT from IGnatius T Foobar Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)

Spread the word!

[#] Fri Sep 29 2023 17:46:15 EDT from Nurb432

all this spreading going on....

Log in using a user name and password

New user? Register now

Log in using OpenID

Log in using Google

Log in using Yahoo

Log in using AOL or AIM

Please wait...