Please wait...
0 files
Subject: V 996 The DB Log count has increased drammitcally
The DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.
The citadel application core dumps on "systemctl stop citadel" when executing a backup.
I this a concern or and induication that something is misconfigured? All Citadel related files are owned citadel and group citadel.
Subject: Re: V 996 The DB Log count has increased drammitcally
OK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient. I need to look into the core dump issue on stopping Citadel......
Fri Dec 15 2023 14:09:05 EST from stormytramp Subject: V 996 The DB Log count has increased drammitcallyThe DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.
The citadel application core dumps on "systemctl stop citadel" when executing a backup.
I this a concern or and induication that something is misconfigured? All Citadel related files are owned citadel and group citadel.
Subject: Re: V 996 The DB Log count has increased drammitcally
The DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.
This suggests that something is very rapidly writing to your database.
Check to make sure your site is not under attack by spammers, and that you don't have any runaway scripts running, anything like that.
And this might be a long shot, but by chance did you run the load test utility and forgot to stop it?
The citadel application core dumps on "systemctl stop citadel" when executing a backup.
That normally doesn't happen anymore, but if you're able to produce a core dump, run it through the debugger and let's see where it crashed.
Subject: Re: V 996 The DB Log count has increased drammitcally
OK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient.
Well, sort of. :)
The fact that it's committing 16 GB at a time and not crashing is the new and improved database. The fact that you are easily recovering from that without corruption when you do take it down is also the new and improved database. But the fact that it's happening is indicative of unwanted server activity.
You could also run one of those log files through the `strings` utility to see what's inside them.
Subject: Re: V 996 The DB Log count has increased drammitcally
Well, this is embarrassing...
So, I was (thinking I was) cleaning up the issues from the VM crash, inadvertently copied over the DB & log files over the running server DB files and crashed the system.....
Once I restored from the backup, those log files are no longer being created. I didnt run the load test, so I can only assume that I missed something during the upgrade. I also found the the webcit systemd files were not created, so I took care of that. Again, I don't think it was the easy install procedure because I created a new server to check out the process. So all looks as it should.
Another note, regarding the mail logging facility, I have been able to get the logs to reflect all the pertinent data looking for in rsyslog (Redhat 9)... I did not modify the citserver systemd files, but the rsyslog file. Am I doing the wrong files for the details Im looking for?
Cheers
Sat Dec 16 2023 18:41:02 EST from IGnatius T Foobar Subject: Re: V 996 The DB Log count has increased drammitcallyOK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient.
Well, sort of. :)
The fact that it's committing 16 GB at a time and not crashing is the new and improved database. The fact that you are easily recovering from that without corruption when you do take it down is also the new and improved database. But the fact that it's happening is indicative of unwanted server activity.
You could also run one of those log files through the `strings` utility to see what's inside them.
Subject: mail -syslogs - Log facility does not show data from, to, size, etc.
Subject: Re: mail -syslogs - Log facility does not show data from, to, size, etc.
Sorry
Meant to say that these log enries from mail facility are not showing the data sought
Cheers
Tue Dec 19 2023 04:32:13 EST from stormytramp Subject: mail -syslogs - Log facility does not show data from, to, size, etc.
Subject: Re: V 996 The DB Log count has increased drammitcally
Another note, regarding the mail logging facility, I have been able to get
the logs to reflect all the pertinent data looking for in rsyslog (Redhat
9)... I did not modify the citserver systemd files, but the rsyslog file.Am
I doing the wrong files for the details Im looking for?
That sounds correct.
Citadel Server will generate syslog messages at all levels. If you are running it in the foreground on a terminal, the `-x` option merely indicates what logging level is sent to the screen. But if you are capturing log messages using syslog, Citadel Server will always generate syslog messages at all levels.
It is up to your syslog daemon (rsyslog or whatever) to decide which messages it wants to capture and where it wants to store them.
Back when I ran my main system at home, I'd send all my debug level syslogs to /dev/tty12 so I could just toggle over there whenever I wanted to watch it. Nowadays it's a virtual machine running on a host in a remote data center so I don't do that anymore, but it was fun to watch.
I have Citadel installed on a Raspberry Pi ... this is the 3rd time I have installed it, yet to finish due to me breaking things ...
I tried to move the Database and Log files to my mounted 1TB ssd, but then the pi would crash on boot. So I wiped it to start over.
How do I safely and correctly move these to the mounted ssd so that I dont have to worry about having enough space?
Thanks
Bobby
You have a couple of options.
If the Pi is new enough (RPi 4 or later) you can just install the operating system directly to the SSD and eliminate the SD card altogether. That is how I run mine. There is also a way to re-flash a Pi 3 to do that, but I bricked mine doing it, so maybe don't try that :)
If you're running the containerized version of Citadel (which avoids all that tedious mucking about with Easy Install) you can put the data volume anywhere you want. For example, if your SSD is on /myssd you can create a directory called /myssd/citadel and then run the Docker command with "--volume=/myssd/citadel:/citadel-data" and it'll put everything there.
If for some reason you are unwilling to do either of the above, you can just make /usr/local/citadel a symlink to some other location on your disk.
Very informative answers, Thank you very much.
I am running a PI4, but it is a board or kit that adds the SSD, so when ever I flash the SD, I think have to mount the SSD in the /etc/fstab, so I am unsure how I would install the OS and boot from the SSD.
Not running a container: I am following a How to using Easy Install script
For the symlink option (ln) would this be correct?
ln -s citadel-path mounted-drive-path
Then after that I assume I move the files to the mounted-drive-path that are in the original folder? Or does it automatically make new ones/move them?
I am new to this, I tinker.
Would it be best to use docker? Is there a how-to for Citadel and/or apache for setting them up in containers?
I found this, gonna give it a go
https://www.makeuseof.com/how-to-boot-raspberry-pi-ssd-permanent-storage/
Docker instructions.. just add the extra bit of parameters above to move it to SSD.
https://www.citadel.org/docker.html
Unrelated to citadel, i have a board where i boot from SD, then have it 'switch' the root over to my SSD. Device wont support booting from SSD, but this way other than the slower boot time, it 'runs' from SSD and get the speed where it matters.
If you cant get anything else to work as far as the PI booting properly, hit me up over in the unix room or something and ill get you instructions on how i did it ( its not hard, and i have a script that will do the grunt work )
I am running a PI4, but it is a board or kit that adds the SSD, so when ever I flash the SD, I think have to mount the SSD in the /etc/fstab, so I am unsure how I would install the OS and boot from the SSD.
Docker instructions.. just add the extra bit of parameters above to
move it to SSD.
I believe he said it was a new installation. If that is the case, there's no need to move anything. Just specify the name of the directory when starting the container, and you're off to the races.
Docker is really the best way to run Citadel these days. Full instructions are at [ https://www.citadel.org/docker.html ] and should cover most use cases, at least those which would be applicable to someone new to containers. For those who are experts with containers, there's a section later on in the document explaining the container's parameters and the seasoned expert will already know what to do.
I'm in the process of trying to acquire some new server hardware in my data center. When I do, I'm going to try to run pretty much everything in containers using a Kubernetes cluster.
For the symlink option (ln) would this be correct?
ln -s citadel-path mounted-drive-path
Something like that. If you already have Citadel installed, you'll want to do something like this. (Obviously make sure Citadel Server is STOPPED while it's moving.) Let's say your SSD is mounted as /ssd and you already have Citadel Server installed to /usr/local/citadel. You could do this:
cd /usr/local
mv -v citadel /ssd
ln -s /ssd/citadel /usr/local/citadel
This will move the contents of /usr/local/citadel to /ssd/citadel and then make /usr/local/citadel a symbolic link to /ssd/citadel.
If the whole system is new, however, assuming you don't have the means to use the SSD as the system drive, you might consider simply mounting your SSD as /usr/local and then installing Citadel using any of the normal methods.
Unrelated to citadel, i have a board where i boot from SD, then have
it 'switch' the root over to my SSD. Device wont support booting
from SSD, but this way other than the slower boot time, it 'runs'
from SSD and get the speed where it matters.
When the Pi first came out, I had a Pi 1 that I was using as a small NAS.
I moved the root filesystem over to an external drive and just kept /boot on the SD card. It worked pretty well, but the procedure to get there wasn't exactly entry level simple.