Please wait...
0 files
Well,
A number of users are attempting to clean their email accounts of Deleted Messages and Trash and the citadel user "citadel" user account 1001 is seeing application crashes:
This system has been functional since last year with nary an issue. 956 was initial system. Currently at 972.
It appears that webcit-http did a core dump yesterday with an issue in libcitadel. I could provide log snippets on request. The core dumps today appear related to the imap-cleanup procedure. A number of users are clearing the Deleted Messages and Trash in their accounts as well as Main Floor Trashcan. We have had two core dumps in as many hours. A couple of users have more than 10K emails to empty....
I
Running 972 on a Redhat 9 EL
LDAP Microsoft AD on 2008R2 Forest/Domain
Aide has created 820 MILLION accounts
User Creation Notice
New user account <> has been created, from host []
Happens at various times periods, anywhere from 1 to 6 per minutes.
I suspect our friends at Microsoft as the culprit as several other Citadel server dont exhibit this behavior. There are several AD authentication schemas but not at the version with the spurious account creations
Anyone see this before.
And yes the plan is to go to 996 as rapidly as possible........
Cheers
I'd like to hear more about how you're using Citadel with AD because that's definitely a supported configuration but I don't run it in production, so if there are things we need to adjust to make it easier day to day, we can work on that.
I can send some logs and configs from the misbehaving system.
I also would like to maillog to show the local recipient. Is there a way to expand the level of the logging to show this?
Subject: Re: Follow up on massive account creations
I also would like to maillog to show the local recipient. Is there a
way to expand the level of the logging to show this?
Yes. Citadel Server sends logs at all logging levels, including "debug" (the highest). Adjust your syslog configuration to capture log messages from citserver at the debug level and you will definitely see alias expansions.
Just an observation i have been meaning to mention for a while and its nothing critical at all:
When a room description is larger than can display, and you hover over, the full text popup normally only displays for a second or so, then the popup is gone. ( at least for me. Chrome. Several versions. Linux )
Subject: V 996 The DB Log count has increased drammitcally
The DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.
The citadel application core dumps on "systemctl stop citadel" when executing a backup.
I this a concern or and induication that something is misconfigured? All Citadel related files are owned citadel and group citadel.
Subject: Re: V 996 The DB Log count has increased drammitcally
OK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient. I need to look into the core dump issue on stopping Citadel......
Fri Dec 15 2023 14:09:05 EST from stormytramp Subject: V 996 The DB Log count has increased drammitcallyThe DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.
The citadel application core dumps on "systemctl stop citadel" when executing a backup.
I this a concern or and induication that something is misconfigured? All Citadel related files are owned citadel and group citadel.
Subject: Re: V 996 The DB Log count has increased drammitcally
The DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.
This suggests that something is very rapidly writing to your database.
Check to make sure your site is not under attack by spammers, and that you don't have any runaway scripts running, anything like that.
And this might be a long shot, but by chance did you run the load test utility and forgot to stop it?
The citadel application core dumps on "systemctl stop citadel" when executing a backup.
That normally doesn't happen anymore, but if you're able to produce a core dump, run it through the debugger and let's see where it crashed.
Subject: Re: V 996 The DB Log count has increased drammitcally
OK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient.
Well, sort of. :)
The fact that it's committing 16 GB at a time and not crashing is the new and improved database. The fact that you are easily recovering from that without corruption when you do take it down is also the new and improved database. But the fact that it's happening is indicative of unwanted server activity.
You could also run one of those log files through the `strings` utility to see what's inside them.
Subject: Re: V 996 The DB Log count has increased drammitcally
Well, this is embarrassing...
So, I was (thinking I was) cleaning up the issues from the VM crash, inadvertently copied over the DB & log files over the running server DB files and crashed the system.....
Once I restored from the backup, those log files are no longer being created. I didnt run the load test, so I can only assume that I missed something during the upgrade. I also found the the webcit systemd files were not created, so I took care of that. Again, I don't think it was the easy install procedure because I created a new server to check out the process. So all looks as it should.
Another note, regarding the mail logging facility, I have been able to get the logs to reflect all the pertinent data looking for in rsyslog (Redhat 9)... I did not modify the citserver systemd files, but the rsyslog file. Am I doing the wrong files for the details Im looking for?
Cheers
Sat Dec 16 2023 18:41:02 EST from IGnatius T Foobar Subject: Re: V 996 The DB Log count has increased drammitcallyOK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient.
Well, sort of. :)
The fact that it's committing 16 GB at a time and not crashing is the new and improved database. The fact that you are easily recovering from that without corruption when you do take it down is also the new and improved database. But the fact that it's happening is indicative of unwanted server activity.
You could also run one of those log files through the `strings` utility to see what's inside them.
Subject: mail -syslogs - Log facility does not show data from, to, size, etc.
Subject: Re: mail -syslogs - Log facility does not show data from, to, size, etc.
Sorry
Meant to say that these log enries from mail facility are not showing the data sought
Cheers
Tue Dec 19 2023 04:32:13 EST from stormytramp Subject: mail -syslogs - Log facility does not show data from, to, size, etc.
Subject: Re: V 996 The DB Log count has increased drammitcally
Another note, regarding the mail logging facility, I have been able to get
the logs to reflect all the pertinent data looking for in rsyslog (Redhat
9)... I did not modify the citserver systemd files, but the rsyslog file.Am
I doing the wrong files for the details Im looking for?
That sounds correct.
Citadel Server will generate syslog messages at all levels. If you are running it in the foreground on a terminal, the `-x` option merely indicates what logging level is sent to the screen. But if you are capturing log messages using syslog, Citadel Server will always generate syslog messages at all levels.
It is up to your syslog daemon (rsyslog or whatever) to decide which messages it wants to capture and where it wants to store them.
Back when I ran my main system at home, I'd send all my debug level syslogs to /dev/tty12 so I could just toggle over there whenever I wanted to watch it. Nowadays it's a virtual machine running on a host in a remote data center so I don't do that anymore, but it was fun to watch.
I have Citadel installed on a Raspberry Pi ... this is the 3rd time I have installed it, yet to finish due to me breaking things ...
I tried to move the Database and Log files to my mounted 1TB ssd, but then the pi would crash on boot. So I wiped it to start over.
How do I safely and correctly move these to the mounted ssd so that I dont have to worry about having enough space?
Thanks
Bobby
You have a couple of options.
If the Pi is new enough (RPi 4 or later) you can just install the operating system directly to the SSD and eliminate the SD card altogether. That is how I run mine. There is also a way to re-flash a Pi 3 to do that, but I bricked mine doing it, so maybe don't try that :)
If you're running the containerized version of Citadel (which avoids all that tedious mucking about with Easy Install) you can put the data volume anywhere you want. For example, if your SSD is on /myssd you can create a directory called /myssd/citadel and then run the Docker command with "--volume=/myssd/citadel:/citadel-data" and it'll put everything there.
If for some reason you are unwilling to do either of the above, you can just make /usr/local/citadel a symlink to some other location on your disk.
Very informative answers, Thank you very much.
I am running a PI4, but it is a board or kit that adds the SSD, so when ever I flash the SD, I think have to mount the SSD in the /etc/fstab, so I am unsure how I would install the OS and boot from the SSD.
Not running a container: I am following a How to using Easy Install script
For the symlink option (ln) would this be correct?
ln -s citadel-path mounted-drive-path
Then after that I assume I move the files to the mounted-drive-path that are in the original folder? Or does it automatically make new ones/move them?
I am new to this, I tinker.