Language:

en_US

switch to room list switch to menu My folders
Go to page: [1] 2 3 4 5 ... Last
↑↑↑ Old messages ↑↑↑            ↓↓↓ New messages ↓↓↓
[#] Wed Nov 15 2023 10:32:36 UTC from dswarhurst

Subject: Citadel Easy Install crashes with segmentation fault

[Reply] [ReplyQuoted] [Headers] [Print]

Hi,

I am running a fully patched release of raspian bullseye on a raspberry pi. Installing Citadel using the Easy Install method.

Having done all the compiles and answered all the configuration questions,  Citadel restarts then issues the following message:

Restarting Citadel server to apply changes

bash: line 643: 26257 Segmentation fault      /usr/local/citadel/setup < /dev/tty > /dev/tty 2> /dev/tty

 Citadel Easy Install is aborting. 

 The last few lines above this message may indicate what went wrong. 


 

Does anyone know how to fix this?

Many thanks.


[#] Wed Nov 15 2023 19:39:05 UTC from IGnatius T Foobar

Subject: Re: Citadel Easy Install crashes with segmentation fault

[Reply] [ReplyQuoted] [Headers] [Print]

You could try running setup in the debugger and tell us where it's failing.
But it might be easier to just run the container version of Citadel, it's guaranteed to work.

[#] Wed Nov 29 2023 10:20:09 UTC from stormytramp

Subject: First the Cisco firewall lost the NAT address to Citadel

[Reply] [ReplyQuoted] [Headers] [Print]

Here is a good one....

Last Wednesday, just as end of day, the Cisco ASA firewall went down on the NAT for Citadel..... version 972

 

While there are automated backups running everyday, everything continued to work..... the backup MX took over spooling email traffic, and the Citadel continued perating within the environment, no mail was flowing due to the disconnected NAT.

On Friday, the IP of the OUTSIDE Citadel got listed at Spamhaus ZEN.

Then the later Friday, the AD LDAP connection failed.

Backups continued, however, the database was encountering problems which was corruption. 

Monday morning, client called NO emails.

Found the AD down, and was unable to connect to AD server. Citadel was unresponsive. ran sysemctl stop on Citadel apps (incl webcit) and restarted. AD restored.

Found the firewall was not NATted. Reset the the firewall (full power reset as ASA wasnt reachable)

Mail started to flow, but AD failed again causing 551 user unknown, emails rejected.

De-spooled MX backup server sent emails which were rejected (3.5GB)

Mail from Friday to Monday lost.

Got Citadel up again with AD, but seems the database was corrupted. Ran db_recovery, no joy. Ran database_cleanup, mail outbound worked, inbound appeared to work as headers came through, but no body. Attempted to restore from backups going back each day from Monday. Several cdb failed db_verify.

So all the way back to Wednesday were restored. All clients were were backed up to PST files. And tried to update to 996. System easyinstall ran fine.....until the setup portion of the install where it hung. No activity for several hours and no joy.

Im restoring the backup of the 972 as I cant wait to figure out where issues are as I must create new cdb and load all the PSTs to return ops.

I cant figure out where it went wrong.....



[#] Thu Nov 30 2023 17:18:46 UTC from stormytramp

Subject: Update

[Reply] [ReplyQuoted] [Headers] [Print]

Well,

A number of users are attempting to clean their email accounts of Deleted Messages and Trash and the citadel user "citadel" user account 1001 is seeing application crashes:

This system has been functional since last year with nary an issue. 956 was initial system. Currently at 972.

It appears that webcit-http did a core dump yesterday with an issue in libcitadel. I could provide log snippets on request. The core dumps today appear related to the imap-cleanup procedure. A number of users are clearing the Deleted Messages and Trash in their accounts as well as Main Floor Trashcan. We have had two core dumps in as many hours. A couple of users have more than 10K emails to empty....

 

I

 



[#] Fri Dec 01 2023 12:54:09 UTC from stormytramp

Subject: Spurious account creations

[Reply] [ReplyQuoted] [Headers] [Print]



[#] Fri Dec 01 2023 13:02:36 UTC from stormytramp

Subject: Spurious account creations

[Reply] [ReplyQuoted] [Headers] [Print]

Running 972 on a Redhat 9 EL

LDAP Microsoft AD on 2008R2 Forest/Domain

 

Aide has created 820 MILLION accounts

 

User Creation Notice

New user account <> has been created, from host []

 

 

Happens at various times periods, anywhere from 1 to 6 per minutes.

 

I suspect our friends at Microsoft as the culprit as several other Citadel server dont exhibit this behavior. There are several AD authentication schemas but not at the version with the spurious  account creations

 

Anyone see this before.

 

And yes the plan is to go to 996 as rapidly as possible........

 

Cheers



[#] Fri Dec 01 2023 19:19:02 UTC from IGnatius T Foobar

Subject: Re: Spurious account creations

[Reply] [ReplyQuoted] [Headers] [Print]

Sorry to hear about that. Yes, if you are <993 you definitely want to upgrade because you'll get the new database layer that is way better at handling unexpected system crashes and outside-of-citadel events.

I'd like to hear more about how you're using Citadel with AD because that's definitely a supported configuration but I don't run it in production, so if there are things we need to adjust to make it easier day to day, we can work on that.

[#] Fri Dec 01 2023 23:55:59 UTC from stormytramp

Subject: Follow up on massive account creations

[Reply] [ReplyQuoted] [Headers] [Print]

I can send some logs and configs from the misbehaving system.

I also would like to maillog to show the local recipient. Is there a way to expand the level of the logging to show this?



[#] Sat Dec 02 2023 18:49:30 UTC from IGnatius T Foobar

Subject: Re: Follow up on massive account creations

[Reply] [ReplyQuoted] [Headers] [Print]

I also would like to maillog to show the local recipient. Is there a
way to expand the level of the logging to show this?

Yes. Citadel Server sends logs at all logging levels, including "debug" (the highest). Adjust your syslog configuration to capture log messages from citserver at the debug level and you will definitely see alias expansions.

[#] Tue Dec 05 2023 22:36:56 UTC from Nurb432

[Reply] [ReplyQuoted] [Headers] [Print]

Just an observation i have been meaning to mention for a while and its nothing critical at all:

When a room description is larger than can display, and you hover over, the full text popup normally only displays for a second or so, then the popup is gone. ( at least for me. Chrome. Several versions. Linux )



[#] Sat Dec 09 2023 15:43:33 UTC from IGnatius T Foobar

[Reply] [ReplyQuoted] [Headers] [Print]

Hate to say this but it's probably not gonna get fixed. All the effort is going into the rewrite right now.

[#] Fri Dec 15 2023 19:09:05 UTC from stormytramp

Subject: V 996 The DB Log count has increased drammitcally

[Reply] [ReplyQuoted] [Headers] [Print]

The DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.

 

The citadel application core dumps on "systemctl stop citadel" when executing a backup.

 

I this a concern or and induication that something is misconfigured? All Citadel related files are owned citadel and group citadel.



[#] Fri Dec 15 2023 19:23:19 UTC from stormytramp

Subject: Re: V 996 The DB Log count has increased drammitcally

[Reply] [ReplyQuoted] [Headers] [Print]

OK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient. I need to look into the core dump issue on stopping Citadel......

Fri Dec 15 2023 14:09:05 EST from stormytramp Subject: V 996 The DB Log count has increased drammitcally

The DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.

 

The citadel application core dumps on "systemctl stop citadel" when executing a backup.

 

I this a concern or and induication that something is misconfigured? All Citadel related files are owned citadel and group citadel.



 



[#] Sat Dec 16 2023 23:37:32 UTC from IGnatius T Foobar

Subject: Re: V 996 The DB Log count has increased drammitcally

[Reply] [ReplyQuoted] [Headers] [Print]

The DB xxxx12345.log files in "data" have dramically increased in number, and are creating new ones by 10+ every second.

This suggests that something is very rapidly writing to your database.

Check to make sure your site is not under attack by spammers, and that you don't have any runaway scripts running, anything like that.

And this might be a long shot, but by chance did you run the load test utility and forgot to stop it?

The citadel application core dumps on "systemctl stop citadel" when executing a backup.

That normally doesn't happen anymore, but if you're able to produce a core dump, run it through the debugger and let's see where it crashed.

 



[#] Sat Dec 16 2023 23:41:02 UTC from IGnatius T Foobar

Subject: Re: V 996 The DB Log count has increased drammitcally

[Reply] [ReplyQuoted] [Headers] [Print]

OK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient. 

Well, sort of.  :)

The fact that it's committing 16 GB at a time and not crashing is the new and improved database.  The fact that you are easily recovering from that without corruption when you do take it down is also the new and improved database.  But the fact that it's happening is indicative of unwanted server activity.

You could also run one of those log files through the `strings` utility to see what's inside them.



[#] Mon Dec 18 2023 23:41:27 UTC from stormytramp

Subject: Re: V 996 The DB Log count has increased drammitcally

[Reply] [ReplyQuoted] [Headers] [Print]

Well, this is embarrassing...

So, I was (thinking I was) cleaning up the issues from the VM crash, inadvertently copied over the DB & log files over the running server DB files and crashed the system.....

 

Once I restored from the backup, those log files are no longer being created. I didnt run the load test, so I can only assume that I missed something during the upgrade. I also found the the webcit systemd files were not created, so I took care of that. Again, I don't think it was the easy install procedure because I created a new server to check out the process. So all looks as it should.

 

Another note, regarding the mail logging facility, I have been able to get the logs to reflect all the pertinent data looking for in rsyslog (Redhat 9)... I did not modify the citserver systemd files, but the rsyslog file. Am I doing the wrong files for the details Im looking for?

 

Cheers

 

 

Sat Dec 16 2023 18:41:02 EST from IGnatius T Foobar Subject: Re: V 996 The DB Log count has increased drammitcally

OK, This log.xxxxxx process count goes to about 1600 log files then they are processed and committed (?) so I am thinking this is by design making the 996 DB more resilient. 

Well, sort of.  :)

The fact that it's committing 16 GB at a time and not crashing is the new and improved database.  The fact that you are easily recovering from that without corruption when you do take it down is also the new and improved database.  But the fact that it's happening is indicative of unwanted server activity.

You could also run one of those log files through the `strings` utility to see what's inside them.



 



[#] Tue Dec 19 2023 09:32:13 UTC from stormytramp

Subject: mail -syslogs - Log facility does not show data from, to, size, etc.

[Reply] [ReplyQuoted] [Headers] [Print]



[#] Tue Dec 19 2023 09:33:38 UTC from stormytramp

Subject: Re: mail -syslogs - Log facility does not show data from, to, size, etc.

[Reply] [ReplyQuoted] [Headers] [Print]

Sorry

Meant to say that these log enries from mail facility are not showing the data sought

 

Cheers

Tue Dec 19 2023 04:32:13 EST from stormytramp Subject: mail -syslogs - Log facility does not show data from, to, size, etc.


 



[#] Tue Dec 19 2023 14:29:47 UTC from IGnatius T Foobar

Subject: Re: V 996 The DB Log count has increased drammitcally

[Reply] [ReplyQuoted] [Headers] [Print]

Another note, regarding the mail logging facility, I have been able to get

the logs to reflect all the pertinent data looking for in rsyslog (Redhat

9)... I did not modify the citserver systemd files, but the rsyslog file.
Am
I doing the wrong files for the details Im looking for?

That sounds correct.

Citadel Server will generate syslog messages at all levels. If you are running it in the foreground on a terminal, the `-x` option merely indicates what logging level is sent to the screen. But if you are capturing log messages using syslog, Citadel Server will always generate syslog messages at all levels.
It is up to your syslog daemon (rsyslog or whatever) to decide which messages it wants to capture and where it wants to store them.

Back when I ran my main system at home, I'd send all my debug level syslogs to /dev/tty12 so I could just toggle over there whenever I wanted to watch it. Nowadays it's a virtual machine running on a host in a remote data center so I don't do that anymore, but it was fun to watch.

[#] Tue Dec 19 2023 16:24:07 UTC from bobbydharrell

Subject: Setup Questions

[Reply] [ReplyQuoted] [Headers] [Print]

I have Citadel installed on a Raspberry Pi ... this is the 3rd time I have installed it, yet to finish due to me breaking things ... 
I tried to move the Database and Log files to my mounted 1TB ssd, but then the pi would crash on boot. So I wiped it to start over.

 

How do I safely and correctly move these to the mounted ssd so that I dont have to worry about having enough space?

 

Thanks

Bobby



Go to page: [1] 2 3 4 5 ... Last