Please wait...
0 files
Subject: Re: Citadel Server stopped sending mails out
Dear All,
eventually found the issue. Somehow Citadel was no longer able to resolve names. Removed systemd-resolved and completely reconfigured. Afterwards resolution was working again resolving the issue.
Now the only remaining question is how to address the error messages related to message #97535.
Any ideas here?
P.
Dear All,
couple of days back server stopped sending mails. Notices because of an error message coming back with SMTP code 412. Receiving is fine, no issues.
In the logs I keep seeing this message:
Sep 23 12:56:17 citserver[5959]: citserver[5959]: msgbase: message #97535 was not found
Sep 23 12:56:17 citserver[5959]: citserver[5959]: smtpclient: 97535 does not exist
Sep 23 12:56:17 citserver[5959]: msgbase: message #97535 was not found
Sep 23 12:56:17 citserver[5959]: smtpclient: 97535 does not existThought maybe database had issues so restored a backup from before last time message was sent successfully. But this did not change anything. No mail goes out and the error also reappears. I therefore guess it is unrelated.
When I login as admin in Webcit and go to outbound SMTP queue it is showing:
As there are mails pending that have not been delivered, I would have expected to see them here.
Any ideas would be much welcome.
Thanks,
P.
Subject: Re: Citadel Server stopped sending mails out
Now the only remaining question is how to address the error messages related
to message #97535.
Any ideas here?
There might be a message in the queue that has that number but it isn't there anymore. If that is the case, the mail delivery system will give up after five days and you will stop receiving that message. If you feel like tinkering, the queue is in a hidden room called __CitadelSMTPspoolout__ and you can poke around in there looking for it.
Subject: Re: Rebooted my RHEL 8.8 instance and now Citadel crashes upon start...
So...after about the 2nd or 3rd re-Easy Install, I was able to make changes in the db from :8443 without it crashing, and it's been up and just fine ever since.
About RHEL...I inherited it with the project. There's a subscription already in place, and I've never messed with RHEL much, so now's my chance, I guess.
My only previous linux experience had been with the Ubuntu and Centos flavors.
Thank you for the help.
Cheers,
JP
Subject: Re: Rebooted my RHEL 8.8 instance and now Citadel crashes upon start...
Hi, Can you please tell me how you fixed this? getting similar messages on rasbian.
Craig.
Subject: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Hi,
I've been using citadel for a few months and decided to upgrade to the latest and greatest version with easy install.. I have done this before on previous version with success, however, this time the install ran through fine, but at the point where it stops the services and restarts the install failed to connect.. it bombed to the shell. I tried to start citadel from command line but it starts then segfaults a few seconds later... please help as this is my main email.
thanks
Craig. (log below)
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: rssclient: using libcurl/7.64.0 OpenSSL/1.1.1n zlib/1.2.11 libidn2/2.2.0 libpsl/0.20.2 (+lib
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: extensions: TCP port 0.0.0.0:25: (SMTP-MTA) registered.
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: extensions: TCP port 0.0.0.0:465: (SMTPs-MTA) registered.
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: extensions: TCP port 0.0.0.0:587: (SMTP-MSA) registered.
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: extensions: unix domain socket 'lmtp.socket': registered.
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: extensions: unix domain socket 'lmtp-unfiltered.socket': registered.
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: citserver[9846]: Existing database version on disk is 994
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: Existing database version on disk is 994
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: citserver[9846]: extensions: service DICT_TCP has been manually disabled, skipping
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: citserver[9846]: extensions: TCP port 0.0.0.0:5222: (XMPP) registered.
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: extensions: service DICT_TCP has been manually disabled, skipping
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: citserver[9846]: main: changing uid to 112
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: extensions: TCP port 0.0.0.0:5222: (XMPP) registered.
Sep 26 14:51:16 victor.cajsoft.co.uk citserver[9846]: main: changing uid to 112
Sep 26 14:51:17 victor.cajsoft.co.uk citserver[9846]: citserver[9846]: context: session (SMTP-MSA) started from 77.90.185.59 (77.90.185.59) uid=-1
Sep 26 14:51:17 victor.cajsoft.co.uk citserver[9846]: context: session (SMTP-MSA) started from 77.90.185.59 (77.90.185.59) uid=-1
Sep 26 14:51:18 victor.cajsoft.co.uk systemd[1]: citadel.service: Main process exited, code=killed, status=11/SEGV
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit citadel.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 11.
Sep 26 14:51:18 victor.cajsoft.co.uk systemd[1]: citadel.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit citadel.service has entered the 'failed' state with result 'signal'.
Sep 26 14:51:18 victor.cajsoft.co.uk systemd[1]: citadel.service: Service RestartSec=100ms expired, scheduling restart.
Sep 26 14:51:18 victor.cajsoft.co.uk systemd[1]: citadel.service: Scheduled restart job, restart counter is at 5.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- the configured Restart= setting for the unit.
Sep 26 14:51:18 victor.cajsoft.co.uk systemd[1]: Stopped Citadel Server.
-- Subject: A stop job for unit citadel.service has finished
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A stop job for unit citadel.service has finished.
--
-- The job identifier is 24163 and the job result is done.
Sep 26 14:51:18 victor.cajsoft.co.uk systemd[1]: citadel.service: Start request repeated too quickly.
Sep 26 14:51:18 victor.cajsoft.co.uk systemd[1]: citadel.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit citadel.service has entered the 'failed' state with result 'signal'.
Sep 26 14:51:18 victor.cajsoft.co.uk systemd[1]: Failed to start Citadel Server.
-- Subject: A start job for unit citadel.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit citadel.service has finished with a failure.
--
-- The job identifier is 24163 and the job result is failed.
-- Automatic restarting of the unit citadel.service has been scheduled, as the result for
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Cant help with the crash, but you did make a backup right? Could always roll back and do some testing with coming suggestions.
Tue Sep 26 2023 10:03:11 EDT from cjonline Subject: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)please help as this is my main email.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Would you believe it.. NO I didnt as the previous upgrade I did I made a backup but it went smoothly so thought it wouldnt be needed :(
how I was wrong !
Craig.
Tue Sep 26 2023 10:11:16 EDT from Nurb432 Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)Cant help with the crash, but you did make a backup right? Could always roll back and do some testing with coming suggestions.
Tue Sep 26 2023 10:03:11 EDT from cjonline Subject: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)please help as this is my main email.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Darn. i always do that even if i have tested it. you never know. power fail, OS freak out = corrupted files = ack!
Once the others see this i'm sure you will be getting suggestions.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
I know.. I made a BIG mistake.. I usually do back up but thought this time it would be ok!!
I've tried clearing down the citadel folders /usr/local/citadel and /usr/local/webcit, then ran the easy install.. it runs fine but crashes at the setup phase. I suspect it's something to do with the latest version and the new db functions?
Tue Sep 26 2023 10:52:42 EDT from Nurb432 Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)Darn. i always do that even if i have tested it. you never know. power fail, OS freak out = corrupted files = ack!
Once the others see this i'm sure you will be getting suggestions.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
once IG sees the post im sure he will have ideas.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Perhaps we can get more information.
First, flog yourself with a wet noodle for disregarding common sense about taking backups. You should be backing up every day, not just when you're about to perform an upgrade. Use a filesystem like `btrfs` or `zfs` which can make instant snapshots, then rsync those snapshots to a backup location.
Then shut down the systemd unit file for Citadel (`systemctl stop citadel`).
It's ok to leave the web interface running.
Run the Citadel Server in the foreground with maximum debugging:
cd /usr/local/citadel
./citserver -x9
If there is really a problem with the server software, it will crash with maximum logging right in front of you. You can post the last screenful or so of the output and perhaps it will tell us more.
For "even more" information, you can try running it in the debugger:
cd /usr/local/citadel
gdb ./citserver
run -x9
If it crashes, post the last screenful up to the crash, then do `thread apply all bt` and stash that output too, just in case we need it.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
./citserver -x9
citserver[1776]: extensions: finished initializing modules (threading=1)
citserver[1776]: sysdep: new client socket 39
citserver[1776]: fulltext: indexing started. msgs 6996--7111
citserver[1776]: msgbase: CtdlFetchMessage(6997, 1)
citserver[1776]: fulltext: ft_index_message() adding msg 6997
citserver[1776]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 0, 0, 0, 1
citserver[1776]: msgbase: fixed_output() part 1: (text/html) (35175 bytes)
citserver[1776]: context: session (SMTPs-MTA) started from 46.148.40.151 (46.148.40.151) uid=-1
citserver[1776]: crypto: using certificate chain keys/citadel.cer
citserver[1776]: crypto: using private key keys/citadel.key
citserver[1776]: fulltext: wordbreaking message 6997 (79 bytes)
citserver[1776]: fulltext: indexing message 6997 [0 tokens]
citserver[1776]: msgbase: CtdlFetchMessage(6998, 1)
citserver[1776]: fulltext: ft_index_message() adding msg 6998
citserver[1776]: msgbase: CtdlOutputPreLoadedMsg(TheMessage=not null, 0, 0, 0, 1
citserver[1776]: msgbase: fixed_output() part 1: (text/html) (53736 bytes)
citserver[1776]: fulltext: wordbreaking message 6998 (0 bytes)
Segmentation fault
run -x9
--------
thread apply all bt
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
All right, this is your lucky day. That was an easy problem to spot and I've published a fix.
Somehow you got a couple of zero-length messages into your system (how?) and the indexer choked on a null word list. Go ahead and do the upgrade again.
Make sure you get at least version 995 of the server.
And start taking daily backups. :)
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Hi, Still not working.. same issue -
DB version is 995
----
citserver[18815]: msgbase: fixed_output() part 1: (text/html) (53736 bytes)
citserver[18815]: context: session (SMTP-MSA) started from 77.90.185.59 (77.90.185.59) uid=-1
citserver[18815]: fulltext: wordbreaking message 6998 (0 bytes)
Segmentation fault
Wed Sep 27 2023 00:28:29 EDT from IGnatius T Foobar Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
All right, this is your lucky day. That was an easy problem to spot and I've published a fix.
Somehow you got a couple of zero-length messages into your system (how?) and the indexer choked on a null word list. Go ahead and do the upgrade again.
Make sure you get at least version 995 of the server.
And start taking daily backups. :)
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Hi IG,
please can you help ?
Craig.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
He will. Hes got a day job remember :)
Thu Sep 28 2023 05:19:24 EDT from cjonline Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)Hi IG,
please can you help ?
Craig.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
All right, one more try. Make sure you get version 996 of the server.
That fix we put in to get around your mysterious null message needed to be a few lines earlier in the code. I hope this fixes it.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Thank you so much! that fixed my issue.. will create backups now going forward.
You are the best!
Craig.
Thu Sep 28 2023 22:21:22 EDT from IGnatius T Foobar Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
All right, one more try. Make sure you get version 996 of the server.
That fix we put in to get around your mysterious null message needed to be a few lines earlier in the code. I hope this fixes it.
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)
Word the spread!
Fri Sep 29 2023 16:48:00 EDT from IGnatius T Foobar Subject: Re: Citadel wont start after EASY INSTALL upgrade to 944 (rasbian)Spread the word!