Subject: Re: When I add a mail filter based on message size, citserver crashes
Ok, this is fixed in https://code.citadel.org/citadel.git/commit/?id=a2dc9c66382e65082492c255a0bf27d187e383bb
The fixed code will appear in the next released version.
In the mean time, try the following workaround: create a rule that forces the inbox handler to load the message headers. For example, "if subject equals xyxyxyxyxyx, file into trash, and continue". Doesn't matter what it is, as long as it appears in the list above your reject rule.
You can remove that dummy rule after the next upgrade. Watch for a new release of Citadel Server over the next few days.
Subject: Re: When I add a mail filter based on message size, citserver crashes
Hello,
Thank you for your attention on such short notice. We also appreciate you spending your valuable personal time to develop and maintain a good product like Citadel.
I know that NetBSD is not one of the primary OSes for Citadel. I am just an entusiast compiling Citadel for fun on NetBSD. I do not expect support, just sending this information hoping that it will be useful...
BTW, I compiled Citadel under Linux and it is working fine.
The problem is that citserver uses 100% CPU under NetBSD. I found out that the issue is with pthreads. The only way I found to mitigate this is to exclude threading from the code completely (see the patch below). Setting max_threads to 1 does not
mitigate the issue (still 100% CPU)
Additional information:
My versions:
* NetBSD 10.0
* Citadel 4.0.1004
* libcitadel 4.0.1004
* db 18.1.40
Output of ldd:
/opt/citadel/sbin/citserver:
-lssl.15 => /usr/lib/libssl.so.15
-lcrypto.15 => /usr/lib/libcrypto.so.15
-lcrypt.1 => /usr/lib/libcrypt.so.1
-lc.12 => /usr/lib/libc.so.12
-lz.1 => /usr/lib/libz.so.1
-liconv.2
=> /usr/pkg/lib/libiconv.so.2
-lcitadel.4 => /opt/citadel/lib/libcitadel.so.4
-lpthread.1 => /usr/lib/libpthread.so.1
-lical.3 => /usr/pkg/lib/libical.so.3
-licuuc.75 => /usr/pkg/lib/libicuuc.so.75
-licudata.75 => /usr/pkg/lib/libicudata.so.75
-lstdc++.9 => /usr/lib/libstdc++.so.9
-lm.0 => /usr/lib/libm.so.0
-lgcc_s.1 => /usr/lib/libgcc_s.so.1
-licui18n.75 => /usr/pkg/lib/libicui18n.so.75
-lldap.6 => /usr/lib/libldap.so.6
-llber.5 => /usr/lib/liblber.so.5
-lgssapi.12 => /usr/lib/libgssapi.so.12
-lkrb5.28 => /usr/lib/libkrb5.so.28
-lhx509.7 => /usr/lib/libhx509.so.7
-lasn1.10 => /usr/lib/libasn1.so.10
-lcom_err.8 => /usr/lib/libcom_err.so.8
-lroken.20 => /usr/lib/libroken.so.20
-lutil.7 => /usr/lib/libutil.so.7
-lwind.1 => /usr/lib/libwind.so.1
-lheimbase.2 => /usr/lib/libheimbase.so.2
-lheimntlm.6 => /usr/lib/libheimntlm.so.6
-lexpat.1 => /usr/pkg/lib/libexpat.so.1
-lcurl.4
=> /usr/pkg/lib/libcurl.so.4
-lnghttp2.14 => /usr/pkg/lib/libnghttp2.so.14
-lidn2.0 => /usr/pkg/lib/libidn2.so.0
-lunistring.5 => /usr/pkg/lib/libunistring.so.5
-lintl.1 => /usr/lib/libintl.so.1
-lresolv.3 => /usr/lib/libresolv.so.3
-ldb18-18.1 => /usr/pkg/lib/libdb18-18.1.so
My test patch (unified) which got citserver going without 100% CPU:
# o< --------------------------------------------------------------------------
--- server/threads.c.orig 2024-11-13 10:52:14.366557357 +0000
+++ server/threads.c 2024-11-13 10:55:38.939885643 +0000
@@ -67,14 +67,10 @@
// Second call to module init functions now that threading is up
initialize_modules(1);
- // Begin with one worker thread. We will expand the pool if necessary
- CtdlThreadCreate(worker_thread);
-
- // The supervisor thread monitors worker threads and spawns more of them if it finds that they are all in use.
+ //NOTE: citserv
on NetBSD is using 100% CPU, so disabling threading!
+ //event loop without threads (although pthread code initialized)
while (!server_shutting_down) {
- if ((active_workers == num_workers) && (num_workers < CtdlGetConfigInt("c_max_workers"))) {
- CtdlThreadCreate(worker_thread);
- }
+ worker_thread(NULL);
usleep(1000000);
}
# o< --------------------------------------------------------------------------
Some other tweaks to get it compiled under NetBSD
# in order to find libdb v18
sed -i 's/^BACKEND_LDFLAGS=.*/BACKEND_LDFLAGS=-ldb18/' configure
sed -i 's@^#include <db.h>@#include <db18/db.h>@' server/backends/berkeley_db/berkeley_db.c
# all the -R (rpaths) are needed for NetBSD, because it does not do traditional shared libraries lookup
CFLAGS="-I/usr/pkg/include -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -I/opt/citadel/include -L/opt/citadel/lib -Wl,-R/opt/citadel/lib" \
LDFLAGS="-L/usr/pkg/lib
-Wl,-R/usr/pkg/lib -L/opt/citadel/lib -Wl,-R/opt/citadel/lib" \
./configure --ctdldir=/opt/citadel/sbin
Cheers and have fun!
Subject: Re: When I add a mail filter based on message size, citserver crashes
Thank you for your attention on such short notice. We also
appreciate you spending your valuable personal time to develop
and maintain a good product like Citadel.
Seeing people using and enjoying the Citadel System is very rewarding.
Please be advised that we've pushed a maintenance release (version 1006) that contains the fix to the bug you uncovered. It is available in both Docker and Easy Install packages.
Subject: Re: When I add a mail filter based on message size, citserver crashes
Hello,
I pulled the latest docker image and fired up Citadel. I wrote the rule and it works. Thanks again.
Regards,
Subject: Re: citserver NetBSD 100% CPU from pthreads
The problem is that citserver uses 100% CPU under NetBSD. I found out thatthe issue is with pthreads. The only way I found to >mitigate this is to exclude threading from the code completely (see the patch below). Setting max_threads to 1 does not
mitigate the issue (still 100% CPU)
That is indeed an interesting workaround. It would be interesting to see a stack trace to know where the active loop is taking place. Although disabling multithreaded execution isn't a workable solution for the mainline, it will actually work at a very small scale, it will simply answer one request at a time. We had someone trying that under HP-UX about 20 years ago with similar results.
It would be interesting to see a stack traceIt seams the high load is coming from the main thread (in top PID 8846 and
to know where the active loop is taking place.
LID 8846) ... Almost sure this is about how NetBSD handles pthreads differently
than Linux, but I am not so deep in thread programming, so cannot say for sure..
My test results are:
* without any server load (just started the server)
* disabled all services (IMAP, XMPP, etc.) not to interfere the results
* my versions: libcitadel 4.0.1004, citadel 4.0.1004, db 18.1.40
top output (one thread per line) shows:
PID LID USERNAME PRI STATE TIME WCPU CPU NAME COMMAND
8864 8864 citadel 30 CPU/0 5:48 98.19% 98.19% - citserver
8864 9543 citadel 85 select/1 0:00 0.00% 0.00% - citserver
8864 10738 citadel 85 select/1 0:00 0.00% 0.00% - citserver
4528
4528 user 85 poll/1 0:00 0.00% 0.00% - sudo
11229 11229 user 85 poll/1 0:00 0.00% 0.00% - sudo
6537 6537 root 85 wait/1 0:00 0.00% 0.00% - gdb
gdb output (started process, sent ^C, thread apply all bt) shows:
user@machine:~$ sudo gdb /opt/citadel/sbin/citserver
GNU gdb (GDB) 11.0.50.20200914-git
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources
online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/citadel/sbin/citserver...
(gdb) run -ucitadel -h/var/db/citadel
Starting program: /opt/citadel/sbin/citserver -ucitadel -h/var/db/citadel
citserver[8864]: crypto: keys/citadel.key exists and is readable
citserver[8864]: crypto: keys/citadel.cer exists and is readable
citserver[8864]: citserver: checking directory access
citserver[8864]: citserver: opening databases
citserver[8864]: db: initialized Berkeley DB backend
citserver[8864]: citserver: seeding the pseudo-random number generator
citserver[8864]: citserver: initializing configuration system
citserver[8864]: citserver: creating base rooms (if necessary)
citserver[8864]: control: sanity checking the recorded highest message and room numbers
citserver[8864]:
control: sanity checking the recorded highest user number
citserver[8864]: control: sanity checks complete
citserver[8864]: main: upgrading modules
citserver[8864]: Existing database version on disk is 1004
citserver[8864]: extensions: unix domain socket 'citadel.socket': registered.
citserver[8864]: extensions: unix domain socket 'citadel-admin.socket': registered.
citserver[8864]: extensions: TCP port 127.0.0.1:504: (citadel-TCP) registered.
citserver[8864]: main: initializing server extensions
citserver[8864]: extensions: service IMAP has been manually disabled, skipping
citserver[8864]: extensions: service IMAPS has been manually disabled, skipping
citserver[8864]: extensions: service NNTP has been manually disabled, skipping
citserver[8864]: extensions: service NNTPS has been manually disabled, skipping
citserver[8864]: extensions: service POP3 has been manually disabled, skipping
citserver[8864]:
extensions: service POP3S has been manually disabled, skipping
citserver[8864]: rssclient: using libcurl/8.10.1 OpenSSL/3.0.12 zlib/1.2.13 libidn2/2.3.7 nghttp2/1.62.1
citserver[8864]: extensions: service SMTP-MTA has been manually disabled, skipping
citserver[8864]: extensions: service SMTPs-MTA has been manually disabled, skipping
citserver[8864]: extensions: service SMTP-MSA has been manually disabled, skipping
citserver[8864]: extensions: unix domain socket 'lmtp.socket': registered.
citserver[8864]: extensions: unix domain socket 'lmtp-unfiltered.socket': registered.
citserver[8864]: Existing database version on disk is 1004
citserver[8864]: extensions: service DICT_TCP has been manually disabled, skipping
citserver[8864]: extensions: service XMPP has been manually disabled, skipping
citserver[8864]: main: changing uid to 301
[New LWP 10738 of process 8864]
[New LWP 9543 of process
8864]
^C[New process 8864]
Thread 4 received signal SIGINT, Interrupt.
[Switching to process 8864]
0x00007a7d6c846e2a in _sys___select50 () from /usr/lib/libc.so.12
(gdb) thread apply all bt
Thread 3 (LWP 9543 of process 8864 ""):
#0 0x00007a7d6c846e2a in _sys___select50 () from /usr/lib/libc.so.12
#1 0x00007a7d6ea086f1 in __select50 () from /usr/lib/libpthread.so.1
#2 0x00000000004264e7 in worker_thread (blah=0x0) at server/sysdep.c:720
#3 0x00007a7d6ea0c89f in pthread.create_tramp () from /usr/lib/libpthread.so.1
#4 0x00007a7d6c892f80 in ?? () from /usr/lib/libc.so.12
#5 0x0000000000000000 in ?? ()
Thread 2 (LWP 10738 of process 8864 ""):
#0 0x00007a7d6c846e2a in _sys___selec
t50 () from /usr/lib/libc.so.12
#1 0x00007a7d6ea086f1 in __select50 () from /usr/lib/libpthread.so.1
#2 0x00000000004264e7 in worker_thread (blah=0x0) at server/sysdep.c:720
#3 0x00007a7d6ea0c89f in pthread.create_tramp
() from /usr/lib/libpthread.so.1
#4 0x00007a7d6c892f80 in ?? () from /usr/lib/libc.so.12
#5 0x0000000000400000 in ?? ()
#6 0x00007a7d65400000 in ?? ()
#7 0x0000001003a0efff in ?? ()
#8 0x00007a7d652000c0 in ?? ()
#9 0x00000000001fff40 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 1 (LWP 8864 of process 8864 ""):
--Type <RET> for more, q to quit, c to continue without paging--
#0 0x00007a7d6c9694cd in usleep () from /usr/lib/libc.so.12
#1 0x0000000000426c06 in go_threading () at server/threads.c:78
#2 0x0000000000424b78 in main (argc=3, argv=0x7f7fffca4d58) at server/server_main.c:298
(gdb)
Cheers !
Although disabling multithreaded execution isn't a workable solutionthat for sure...
for the mainline
, it will actually work at a very small scale, it will simply answer onewell, this is good news for me :) thanks for the info
request at a time.
i was thinking i may install [in a very distant future] a home server with
almost no load...
speaking of load: I tested the no-threaded server with 'loadtest -n5' and did
not see crashing or some db corruptions, so it actually may be useful.
Subject: Re: citserver NetBSD 100% CPU from pthreads
speaking of load: I tested the no-threaded server with 'loadtest -n5' anddid
not see crashing or some db corruptions, so it actually may be useful.
It shouldn't crash, and it definitely shouldn't corrupt the database. What you might notice, for example, is that if a very large message (say, several megabytes or more) is being submitted or transmitted, all the other sessions will block until it's finished. If it weren't for that, the whole server could be single threaded and we wouldn't have to worry about locking resources while writing to them, or anything like that ... it would be developer heaven
:)
Your stack trace suggests what I had suspected -- the select() system call is perhaps behaving differently on NetBSD than it does on FreeBSD and Linux.
Your stack trace suggests what I had suspected -- the select() system callah, the select() then!
is perhaps behaving differently on NetBSD than it does on FreeBSD and Linux.
take a quick peak at their man page select(2):
https://man.netbsd.org/NetBSD-10.0/select.2
especially this:
# o< --------------------------------------------------------------------------
NOTES
It is recommended to use the poll(2) interface instead, which tends to be
more portable and efficient.
# o< --------------------------------------------------------------------------
these people tend to hate select + plus their BUGS section in select(2) is
bigger than the rest of the man page :)
... it would be developer heaven :)hehe :) Valhala :)
that is why i like good'ol multiprocessing with fork()-s :) I know, I know -
nobody recommends it and also there are voices to remove it from POSIX
completely :)
Thanks, I will give that a try
Subject: textclient not honouring ansi_color=off in citadelrc
Not a big deal, just to report it:
My Versions:
* the last text client downloaded from easyinstall.citadel.org 2024-11-16
Problem description:
* editing $HOME/.citadelrc for textclient configuration
* entering ansi_color=off
* starting client
* Expected result: client to start without colors
* Result: client still has ansi colors
Additional information:
In commands.c the variable rc_ansi_color is initialised to:
rc_ansi_color = 1;
And later on the logic for reading ~/.citadelrc is:
if (!strncasecmp(buf, "ansi_color=", 11)) {
if ( (!strncasecmp(&buf[11], "on", 2))
|| (!strncasecmp(&buf[11], "auto", 4))
) {
rc_ansi_color = 1;
}
}
Conclusion: it could never reach value rc_ansi_color = 0;
even if in ~/.citadelrc configuration is ansi_color=off
Cheers!
Subject: Re: No working install posible on debian bookworm or other raspberry pi os
Mon Nov 11 2024 10:52:10 UTC from alcomys Subject: No working install posible on debian bookworm or other raspberry pi osHi Guys,
I have been busy day's to get citadel running on my raspberry pi but without success.
The setup and needed download when use easy install are good to go, but when it comes to login to the admin it goes wrong.
When installing the easy install the standard way so use admin and the password citadel for the web cit to login i got always an password error.
If i run the easy installer again and replace the admin and password with some of my own i have the same problem with password error.
I did searched google for an answer but there is less information on how to get it work on Debian bookworm.
Maybe someone did had the same problems as i have? and solved them?
Or maybe someone can help me out here? because it is very frustrating when trying days without success.
Thanks in Advance.
Alcomys
Is there nobody who have had the same problem as me with the password not working in Citadel Web-cit?
I did try different distro's for my raspberry pi 4-B and all have the same problem with the password.
Thanks in advance.
Subject: Re: No working install posible on debian bookworm or other raspberry pi os
More like most of us dont use RPI hardware. If i get a chance this weekend ill try latest easy install on a real ARM SBC.
Does "different distros" include Armbian?
Mon Nov 18 2024 20:34:54 UTC from alcomys Subject: Re: No working install posible on debian bookworm or other raspberry pi os
Is there nobody who have had the same problem as me with the password not working in Citadel Web-cit?
I did try different distro's for my raspberry pi 4-B and all have the same problem with the password.
Thanks in advance.
Ok my plans for tonight fell thru and i gave it a shot.
It wont be 1:1 the same as your hardware, other than being an ARM chip ( different family but same concept ), but Armbian is available for the RPI as well. However, i ran into a different issue, where around 95% of the process, the restart of the server hit 100% cpu and hung ( not seen that one before ). This was not a virgin install, so if i have time tomorrow ill blow the OS away and start over.
As a side note, while im not a docker fan, i did look and RPI4's can run docker. So that is another option for you, instead of using easyinstall.
And while someone else would need to chime in, i would assume you could go into the DB manually and reset the password to a known value. Just in case something weird is going on with your terminal when you entered it during hte install.
Subject: Re: No working install posible on debian bookworm or other raspberry pi os
Mon Nov 18 2024 21:15:51 UTC from Nurb432 Subject: Re: No working install posible on debian bookworm or other raspberry pi osMore like most of us dont use RPI hardware. If i get a chance this weekend ill try latest easy install on a real ARM SBC.
Does "different distros" include Armbian?
Mon Nov 18 2024 20:34:54 UTC from alcomys Subject: Re: No working install posible on debian bookworm or other raspberry pi os
Is there nobody who have had the same problem as me with the password not working in Citadel Web-cit?
I did try different distro's for my raspberry pi 4-B and all have the same problem with the password.
Thanks in advance.
Hi Nurb432,
Thanks for the replay.
Yes i have Armbian on my raspberry pi 4.
But i did try also other distro like ubuntu,Debian,Openelec all have the same problem that the password when logging in Web-Cit is wrong.
So the install has no problems but there is something with the password.
Subject: Re: No working install posible on debian bookworm or other raspberry pi os
With a bit of luck i can try again today. I expect a quiet day at work but you never know. Someone like IG can verify my thoughts about the DB. I have not had to look at the DB before.
Tue Nov 19 2024 11:03:35 UTC from alcomys Subject: Re: No working install posible on debian bookworm or other raspberry pi os
Hi Nurb432,Thanks for the replay.
Yes i have Armbian on my raspberry pi 4.
But i did try also other distro like ubuntu,Debian,Openelec all have the same problem that the password when logging in Web-Cit is wrong.
So the install has no problems but there is something with the password.
Thanks for the release.
I suppose that in some emails clients there is a maximum from
which is cut.
But 70 (??) is too few.
Ok, it will be 256 in the next update.
Ok i wont be of much help on the bad password thing. Tried an easy install on a fresh minimal Armbian OS, same result as before and it froze up on me during the restart. I am sure someone will chime in, but might be worth looking at the docker option in the meantime.
I could try the OEM OS, but its not anything you can install on a RPI so not worth it to try.