Language:
switch to room list switch to menu My folders
Go to page: First ... 6 7 8 9 [10] 11 12 13 14 ... Last
[#] Tue Feb 14 2017 18:52:55 EST from IGnatius T Foobar @ Uncensored

Subject: Re: Error configuring citadel suite.

[Reply] [ReplyQuoted] [Headers] [Print]

My idea would be: the "how I was installed" file should live where the db lives. And when you fire any migration script, it needs to know where the db is located anyway. And it should contain a "automatically generated, no trespassing!" note....

Also, for maintenablitly, it should be dist2easy and easy2dist. It is still maintenance hell, I once wrote a generic install script for Retroshare, with distro guessing, etc.

To be honest, I don't have a lot of interest in going from Easy Install back to the distribution packages.  Once a site is running Easy Install, they've already satisfied the dependencies and have gotten their system into a state where they will always be able to pull down the latest-and-greatest version directly from the Citadel project.  Why would they ever go back?

What I'm thinking of doing is just adding some code to Easy Install that, if it doesn't find a Citadel installation in /usr/local/citadel, also looks for a Citadel installation in the FSSTND directories.  If it finds one, it can offer a prompt something like this:

Easy Install has found an existing Citadel installation in /var/lib/citadel.
Please select what you wish to do:
1) Move this data to /usr/local/citadel and run an Easy Install upgrade.
2) Copy this data to /usr/local/citadel and run an Easy Install upgrade.
3) Ignore this data and perform a new Citadel installation in /usr/local/citadel. 

We'll get the usual complaints from people who didn't RTFM and didn't take a backup, but those people can die in a car fire.



[#] Wed Feb 15 2017 10:38:44 EST from bennabiy @ Uncensored

Subject: Re: Citserver stops responding

[Reply] [ReplyQuoted] [Headers] [Print]

Since I have not managed to get it to produce a core yet, I think I am better off manually starting it in gdb.

 

Tue Feb 14 2017 18:46:08 EST from IGnatius T Foobar @ Uncensored Subject: Re: Citserver stops responding
Tue Feb 14 2017 11:28:02 AM EST from bennabiy @ Uncensored Subject: Re: Citserver stops responding

How would you recommend starting citserver to try to capture it? It is not something I can duplicate except by waiting...

Well, there are a couple of ways.  You can either start citserver from inside the debugger, or you can try to get a core dump.  If your problem is occurring on a predictable daily basis, you're probably better off starting with the debugger.  Stop citserver and then start it like this:

cd /usr/local/citadel

gdb ./citserver

(gdb) run

Citadel Server will run in the foreground.  When you have the problem where it gets stuck, press Ctrl-C.  You will get some diagnostics and a new gdb prompt.  At that point you have to get the stack traces for every thread:

(gdb) thread apply all bt

Most of your threads will be blocked on select() , those are idle threads.  If the server is truly stuck, there will be at least one thread doing something unusual.  Where it is will tell us what's stuck.



 



[#] Wed Feb 15 2017 11:13:42 EST from bennabiy @ Uncensored

Subject: Re: Citserver stops responding

[Reply] [ReplyQuoted] [Headers] [Print]

Ok, here is the backtrace-

citserver[1211]: SSL_read got error 5

citserver[1211]: Ending SSL/TLS

 

Program received signal SIGPIPE, Broken pipe.

[Switching to Thread 0x7ffff2f79700 (LWP 1380)]

0x00007ffff6a10a3d in write () at ../sysdeps/unix/syscall-template.S:81

81../sysdeps/unix/syscall-template.S: No such file or directory.

(gdb) thread apply all bt

 

Thread 8 (Thread 0x7ffff2f79700 (LWP 1380)):

#0  0x00007ffff6a10a3d in write () at ../sysdeps/unix/syscall-template.S:81

#1  0x00007ffff765e155 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#2  0x00007ffff765c1ec in BIO_write () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#3  0x00007ffff7986f12 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#4  0x00007ffff7988b83 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#5  0x00007ffff7984d42 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#6  0x0000000000443af0 in endtls () at modules/crypto/serv_crypto.c:707

#7  0x0000000000442ffa in client_read_sslbuffer (buf=0x7fffe00ec850, timeout=1) at modules/crypto/serv_crypto.c:420

#8  0x0000000000443240 in client_readline_sslbuffer (Line=0x7fffd8009700, IOBuf=0x7fffe00ec850, Pos=0x7fffe008e580, timeout=1) at modules/crypto/serv_crypto.c:486

#9  0x00000000004166ff in CtdlClientGetLine (Target=0x7fffd8009700) at sysdep.c:777

#10 0x0000000000464900 in imap_command_loop () at modules/imap/serv_imap.c:1527

#11 0x00000000004178ca in worker_thread (blah=0x0) at sysdep.c:1453

#12 0x0000000000434c4f in CTC_backend (supplied_start_routine=0x417188 <worker_thread>) at threads.c:121

#13 0x00007ffff6a0a064 in start_thread (arg=0x7ffff2f79700) at pthread_create.c:309

#14 0x00007ffff543d62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

 

Thread 7 (Thread 0x7ffff3186700 (LWP 1339)):

#0  0x00007ffff6a10a9d in read () at ../sysdeps/unix/syscall-template.S:81

#1  0x00007ffff765e0dc in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#2  0x00007ffff765c0ec in BIO_read () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#3  0x00007ffff7986b44 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#4  0x00007ffff7987e6d in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#5  0x00007ffff7984f84 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#6  0x0000000000442f9d in client_read_sslbuffer (buf=0x7fffe82c8360, timeout=1) at modules/crypto/serv_crypto.c:410

#7  0x0000000000443240 in client_readline_sslbuffer (Line=0x7fffe01abf90, IOBuf=0x7fffe82c8360, Pos=0x7fffe8338bb0, timeout=1) at modules/crypto/serv_crypto.c:486

---Type <return> to continue, or q <return> to quit---

#8  0x00000000004166ff in CtdlClientGetLine (Target=0x7fffe01abf90) at sysdep.c:777

#9  0x0000000000464900 in imap_command_loop () at modules/imap/serv_imap.c:1527

#10 0x00000000004178ca in worker_thread (blah=0x0) at sysdep.c:1453

#11 0x0000000000434c4f in CTC_backend (supplied_start_routine=0x417188 <worker_thread>) at threads.c:121

#12 0x00007ffff6a0a064 in start_thread (arg=0x7ffff3186700) at pthread_create.c:309

#13 0x00007ffff543d62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

 

Thread 6 (Thread 0x7ffff307a700 (LWP 1219)):

#0  0x00007ffff5436893 in select () at ../sysdeps/unix/syscall-template.S:81

#1  0x0000000000417450 in worker_thread (blah=0x0) at sysdep.c:1319

#2  0x0000000000434c4f in CTC_backend (supplied_start_routine=0x417188 <worker_thread>) at threads.c:121

#3  0x00007ffff6a0a064 in start_thread (arg=0x7ffff307a700) at pthread_create.c:309

#4  0x00007ffff543d62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

 

Thread 5 (Thread 0x7ffff3287700 (LWP 1218)):

#0  0x00007ffff6a10a9d in read () at ../sysdeps/unix/syscall-template.S:81

#1  0x00007ffff765e0dc in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#2  0x00007ffff765c0ec in BIO_read () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

#3  0x00007ffff798cb45 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#4  0x00007ffff798b3da in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#5  0x00007ffff798b9f8 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

#6  0x00000000004437d6 in CtdlStartTLS (ok_response=0x0, nosup_response=0x0, error_response=0x0) at modules/crypto/serv_crypto.c:617

#7  0x000000000041a560 in CtdlModuleStartCryptoMsgs (ok_response=0x0, nosup_response=0x0, error_response=0x0) at serv_extensions.c:1477

#8  0x000000000048b12e in smtps_greeting () at modules/smtp/serv_smtp.c:201

#9  0x00000000004178ad in worker_thread (blah=0x0) at sysdep.c:1449

#10 0x0000000000434c4f in CTC_backend (supplied_start_routine=0x417188 <worker_thread>) at threads.c:121

#11 0x00007ffff6a0a064 in start_thread (arg=0x7ffff3287700) at pthread_create.c:309

---Type <return> to continue, or q <return> to quit---

#12 0x00007ffff543d62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

 

Thread 4 (Thread 0x7ffff3388700 (LWP 1217)):

#0  0x00007ffff5436893 in select () at ../sysdeps/unix/syscall-template.S:81

#1  0x0000000000417450 in worker_thread (blah=0x0) at sysdep.c:1319

#2  0x0000000000434c4f in CTC_backend (supplied_start_routine=0x417188 <worker_thread>) at threads.c:121

#3  0x00007ffff6a0a064 in start_thread (arg=0x7ffff3388700) at pthread_create.c:309

#4  0x00007ffff543d62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

 

Thread 3 (Thread 0x7ffff3489700 (LWP 1216)):

#0  0x00007ffff543dc03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81

#1  0x00007ffff735ebc2 in epoll_poll (loop=0x7fffe4000940, timeout=<optimized out>) at ev_epoll.c:153

#2  0x00007ffff7360f8d in ev_run (loop=0x7fffe4000940, flags=<optimized out>) at ev.c:3049

#3  0x0000000000450780 in db_event_thread (arg=0x0) at modules/eventclient/serv_eventclient.c:867

#4  0x0000000000434c4f in CTC_backend (supplied_start_routine=0x450663 <db_event_thread>) at threads.c:121

#5  0x00007ffff6a0a064 in start_thread (arg=0x7ffff3489700) at pthread_create.c:309

#6  0x00007ffff543d62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

 

Thread 2 (Thread 0x7ffff7f80700 (LWP 1215)):

#0  0x00007ffff543dc03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81

#1  0x00007ffff735ebc2 in epoll_poll (loop=0x7ffff7567500 <default_loop_struct>, timeout=<optimized out>) at ev_epoll.c:153

#2  0x00007ffff7360f8d in ev_run (loop=0x7ffff7567500 <default_loop_struct>, flags=<optimized out>) at ev.c:3049

#3  0x000000000045027d in client_event_thread (arg=0x0) at modules/eventclient/serv_eventclient.c:727

#4  0x0000000000434c4f in CTC_backend (supplied_start_routine=0x450125 <client_event_thread>) at threads.c:121

#5  0x00007ffff6a0a064 in start_thread (arg=0x7ffff7f80700) at pthread_create.c:309

#6  0x00007ffff543d62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

 

---Type <return> to continue, or q <return> to quit---

Thread 1 (Thread 0x7ffff7fe1700 (LWP 1211)):

#0  0x00007ffff540ef2d in nanosleep () at ../sysdeps/unix/syscall-template.S:81

#1  0x00007ffff5436fb4 in usleep (useconds=<optimized out>) at ../sysdeps/unix/sysv/linux/usleep.c:32

#2  0x0000000000434d96 in go_threading () at threads.c:178

#3  0x000000000040bf7f in main (argc=1, argv=0x7fffffffe5a8) at server_main.c:402

 

(gdb) 

 

This was the fastest I have seen it crash yet.

Tue Feb 14 2017 18:46:08 EST from IGnatius T Foobar @ Uncensored Subject: Re: Citserver stops responding
Tue Feb 14 2017 11:28:02 AM EST from bennabiy @ Uncensored Subject: Re: Citserver stops responding

How would you recommend starting citserver to try to capture it? It is not something I can duplicate except by waiting...

Well, there are a couple of ways.  You can either start citserver from inside the debugger, or you can try to get a core dump.  If your problem is occurring on a predictable daily basis, you're probably better off starting with the debugger.  Stop citserver and then start it like this:

cd /usr/local/citadel

gdb ./citserver

(gdb) run

Citadel Server will run in the foreground.  When you have the problem where it gets stuck, press Ctrl-C.  You will get some diagnostics and a new gdb prompt.  At that point you have to get the stack traces for every thread:

(gdb) thread apply all bt

Most of your threads will be blocked on select() , those are idle threads.  If the server is truly stuck, there will be at least one thread doing something unusual.  Where it is will tell us what's stuck.



 



[#] Wed Feb 15 2017 12:30:07 EST from bennabiy @ Uncensored

Subject: Re: Citserver stops responding

[Reply] [ReplyQuoted] [Headers] [Print]

I decided to put the backtraces in a pastebin (sorry I did not do it the first time) so far I have gotten it to crash twice, same error.

https://pastebin.mozilla.org/8979383

You can view the diff between them (obviously there will be many)

Let me know where to go from here...

 

Tue Feb 14 2017 18:46:08 EST from IGnatius T Foobar @ Uncensored Subject: Re: Citserver stops responding
Tue Feb 14 2017 11:28:02 AM EST from bennabiy @ Uncensored Subject: Re: Citserver stops responding

How would you recommend starting citserver to try to capture it? It is not something I can duplicate except by waiting...

Well, there are a couple of ways.  You can either start citserver from inside the debugger, or you can try to get a core dump.  If your problem is occurring on a predictable daily basis, you're probably better off starting with the debugger.  Stop citserver and then start it like this:

cd /usr/local/citadel

gdb ./citserver

(gdb) run

Citadel Server will run in the foreground.  When you have the problem where it gets stuck, press Ctrl-C.  You will get some diagnostics and a new gdb prompt.  At that point you have to get the stack traces for every thread:

(gdb) thread apply all bt

Most of your threads will be blocked on select() , those are idle threads.  If the server is truly stuck, there will be at least one thread doing something unusual.  Where it is will tell us what's stuck.



 



[#] Wed Feb 15 2017 14:46:28 EST from bennabiy @ Uncensored

Subject: Re: Citserver stops responding

[Reply] [ReplyQuoted] [Headers] [Print]

latest pastebin... https://pastebin.mozilla.org/8979396

Wed Feb 15 2017 12:30:07 EST from bennabiy @ Uncensored Subject: Re: Citserver stops responding

I decided to put the backtraces in a pastebin (sorry I did not do it the first time) so far I have gotten it to crash twice, same error.

https://pastebin.mozilla.org/8979383

You can view the diff between them (obviously there will be many)

Let me know where to go from here...

 

Tue Feb 14 2017 18:46:08 EST from IGnatius T Foobar @ Uncensored Subject: Re: Citserver stops responding
Tue Feb 14 2017 11:28:02 AM EST from bennabiy @ Uncensored Subject: Re: Citserver stops responding

How would you recommend starting citserver to try to capture it? It is not something I can duplicate except by waiting...

Well, there are a couple of ways.  You can either start citserver from inside the debugger, or you can try to get a core dump.  If your problem is occurring on a predictable daily basis, you're probably better off starting with the debugger.  Stop citserver and then start it like this:

cd /usr/local/citadel

gdb ./citserver

(gdb) run

Citadel Server will run in the foreground.  When you have the problem where it gets stuck, press Ctrl-C.  You will get some diagnostics and a new gdb prompt.  At that point you have to get the stack traces for every thread:

(gdb) thread apply all bt

Most of your threads will be blocked on select() , those are idle threads.  If the server is truly stuck, there will be at least one thread doing something unusual.  Where it is will tell us what's stuck.



 



 



[#] Thu Feb 16 2017 11:25:34 EST from bennabiy @ Uncensored

Subject: Re: BugHunt

[Reply] [ReplyQuoted] [Headers] [Print]

I started citserver again in gdb, this time with glibc-source installed (so that we do not get a misleading error)... will let you know what I find.



[#] Thu Feb 16 2017 17:21:28 EST from IGnatius T Foobar @ Uncensored

Subject: Re: BugHunt

[Reply] [ReplyQuoted] [Headers] [Print]

Quick hint here: if your server stopped on a SIGPIPE, you're not actually hung up. You have to bypass SIGPIPE in the debugger. This can be done by entering the following command in gdb *before* you run the program:

handle SIGPIPE nostop noprint pass

You'll get a SIGPIPE any time a remote host hangs up unexpectedly. This doesn't bother citserver but for some reason gdb likes to stop on it.

[#] Thu Feb 16 2017 17:34:15 EST from bennabiy @ Uncensored

Subject: Re: BugHunt

[Reply] [ReplyQuoted] [Headers] [Print]

I figured as much that it seemed odd that it was crashing so quickly in the debugger, and I noticed it referenced a source library which didn't exist.

 

Running again...

Thu Feb 16 2017 17:21:28 EST from IGnatius T Foobar @ Uncensored Subject: Re: BugHunt
Quick hint here: if your server stopped on a SIGPIPE, you're not actually hung up. You have to bypass SIGPIPE in the debugger. This can be done by entering the following command in gdb *before* you run the program:

handle SIGPIPE nostop noprint pass

You'll get a SIGPIPE any time a remote host hangs up unexpectedly. This doesn't bother citserver but for some reason gdb likes to stop on it.

 



[#] Thu Feb 16 2017 22:51:42 EST from rwolfe @ Uncensored

Subject: Citadel VM

[Reply] [ReplyQuoted] [Headers] [Print]

Hi all!  I am putting together a VM of the latest Citadel (built using the EasyInstall script) built on the latest CentOS 7.x release.  The VM is for Oracle VirtualBox and if anyone wants to give it a look see or a try, please let me know and I can make it available for downloading from my BBS and website.



[#] Fri Feb 17 2017 15:38:22 EST from IGnatius T Foobar @ Uncensored

Subject: Re: Citadel VM

[Reply] [ReplyQuoted] [Headers] [Print]

rwolfe, I don't know if you're too far along for these resources to be helpful to you, but I had a couple of scripts that I used to build appliances, with the idea that the appliances would be Easy Install based and therefore easy to update (i.e. without having to install a new appliance).

http://easyinstall.citadel.org/appliance-builder.sh

It also installed a console:

http://easyinstall.citadel.org/appliance-console.sh

These are designed for Debian not CentOS, though. Feel free to take whatever you want from these, or just maybe use them for ideas. My appliances included Citadel Server, WebCit, SpamAssassin, ClamAV, and a basic server administration console. One built today would certainly include ctdlsh as well (even though ctdlsh has not been officially released, it can still be very useful).

[#] Fri Feb 17 2017 15:38:53 EST from IGnatius T Foobar @ Uncensored

Subject: Re: BugHunt

[Reply] [ReplyQuoted] [Headers] [Print]

I figured as much that it seemed odd that it was crashing so quickly in
the
debugger, and I noticed it referenced a source library which didn't exist.


Ok sorry about that, should have told you sooner.

Let me know when you've got a core dump from a "real" server hang, and we'll work our way through it.

[#] Sun Feb 19 2017 08:39:38 EST from bennabiy @ Uncensored

Subject: Re: BugHunt

[Reply] [ReplyQuoted] [Headers] [Print]

Ok, so it is hung, but it has not crashed gdb yet. It is not responding and giving in my mail client an SSL/TLS handshake error.

It does not respond to nc localhost 504.

It is officially hung.

Fri Feb 17 2017 03:38:53 PM EST from IGnatius T Foobar @ Uncensored Subject: Re: BugHunt
I figured as much that it seemed odd that it was crashing so quickly in
the
debugger, and I noticed it referenced a source library which didn't exist.


Ok sorry about that, should have told you sooner.

Let me know when you've got a core dump from a "real" server hang, and we'll work our way through it.

 



[#] Sun Feb 19 2017 09:25:51 EST from bennabiy @ Uncensored

Subject: Re: BugHunt

[Reply] [ReplyQuoted] [Headers] [Print]

Here is a pastebin of the latest crash ( I had to kill it to get it back to gdb)

This is the result of thread apply all bt

http://pastebin.com/r3C83Zpc

and the result of thread apply all bt full

in 2 parts

http://pastebin.com/F07caWbc

and

 http://pastebin.com/mHjRQgY0

Sun Feb 19 2017 08:39:38 AM EST from bennabiy @ Uncensored Subject: Re: BugHunt

Ok, so it is hung, but it has not crashed gdb yet. It is not responding and giving in my mail client an SSL/TLS handshake error.

It does not respond to nc localhost 504.

It is officially hung.

Fri Feb 17 2017 03:38:53 PM EST from IGnatius T Foobar @ Uncensored Subject: Re: BugHunt
I figured as much that it seemed odd that it was crashing so quickly in
the
debugger, and I noticed it referenced a source library which didn't exist.


Ok sorry about that, should have told you sooner.

Let me know when you've got a core dump from a "real" server hang, and we'll work our way through it.

 



 



[#] Tue Feb 21 2017 19:29:37 EST from IGnatius T Foobar @ Uncensored

Subject: Re: BugHunt

[Reply] [ReplyQuoted] [Headers] [Print]

Ok, it's pretty clear what's happening, so now we just have to figure out why.

You've got hundreds of threads (which is a lot) all hung up on SSL reads.
428 of them are SMTPS and 46 of them are IMAPS.

Curiously, they're all hung up in the "greeting" banner for each protocol, which means they're not even clients that have been running for a while. When your server's been running for a while (or when it's hung) can you please do a "netatat -anp | grep citserver" so we can see what's on the other end of these connections?

Is there anything that could be making connections to ports 993 and 465, and closing the connection before speaking to Citadel? Maybe a monitoring system?

[#] Tue Feb 21 2017 22:56:33 EST from bennabiy @ Uncensored

Subject: Re: BugHunt

[Reply] [ReplyQuoted] [Headers] [Print]

One of my clients had this message (probably from outlook, but the closest thing I have seen to a valid error short of SSL/TLS handshake error yet.

I will try that when it hangs again. The only things what would be polling are email clients. I noticed that Outlook 2013 is a bear to setup with citadel (at least using non standard ports and ssl).

 

I do have a script which logs in each user each hour to correct vcard issues, but that succeeds each time.

 

I do have certain users which end up with about 50 email address entries in their vcard where every other one is fine... I wonder...

Tue Feb 21 2017 07:29:37 PM EST from IGnatius T Foobar @ Uncensored Subject: Re: BugHunt
Ok, it's pretty clear what's happening, so now we just have to figure out why.

You've got hundreds of threads (which is a lot) all hung up on SSL reads.
428 of them are SMTPS and 46 of them are IMAPS.

Curiously, they're all hung up in the "greeting" banner for each protocol, which means they're not even clients that have been running for a while. When your server's been running for a while (or when it's hung) can you please do a "netatat -anp | grep citserver" so we can see what's on the other end of these connections?

Is there anything that could be making connections to ports 993 and 465, and closing the connection before speaking to Citadel? Maybe a monitoring system?

 



sslerror.jpg (image/jpeg, 40890 bytes) [ View | Download ]
[#] Wed Feb 22 2017 17:50:41 EST from bennabiy @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

A little update...

 

I managed to check the connections and saw that certain locations had many open connections coming from one IP (a known IP, so not worried about that) where I have many users all connecting to the server. I noticed that I had the idle timeout on my server set to 15 minutes (900 seconds) which would account for many connections building up if their clients are polling faster than that, and possibly opening new connections instead of reusing the old connection.

 

Do you think that would cause it? Running out of workers, and then the connections piling up without anyone to talk to? I still do not know why there are that many SMTPS connecitons, my userbase is only about 275.



[#] Wed Feb 22 2017 18:06:04 EST from rwb0027 @ Uncensored

Subject: debconf install automation, ports can't be binded

[Reply] [ReplyQuoted] [Headers] [Print]

I can install Citadel and it works great.  Love it.  I need to push it onto VMs using bash scripts so I install it then use debconf-get from utils to identify the configuration questions, and boom I make this to skip the installation questions:

echo $(date) " PreInstall Config for Citadel..." >> /usr/dev/log
echo "citadel-server  citadel/Password        password        SomePassword" | debconf-set-selections
echo "citadel-server  citadel/Password_again  password        SomePassword" | debconf-set-selections
echo "citadel-webcit  citadel/WebcitOfferLang select  UNLIMITED" | debconf-set-selections
echo "citadel-server  citadel/LDAPServer      string  0.0.0.0" | debconf-set-selections 
echo "citadel-server  citadel/LDAPServerPort  string  389" | debconf-set-selections
echo "citadel-server  citadel/LDAPBaseDN      string  dc=example,dc=com" | debconf-set-selections
echo "citadel-server  citadel/BadUser error" | debconf-set-selections
echo "citadel-server  citadel/Administrator   string  admin" | debconf-set-selections
echo "citadel-webcit  citadel/WebcitApacheIntegration select  Internal" | debconf-set-selections
echo "citadel-webcit  citadel/WebcitHttpsPort string  8001" | debconf-set-selections
echo "citadel-server  citadel/ServerIPAddress string  0.0.0.0" | debconf-set-selections
echo "citadel-server  citadel/LDAPBindDNPassword      string  DrewOver01" | debconf-set-selections
echo "citadel-server  citadel/LoginType       select  Internal" | debconf-set-selections
echo "citadel-webcit  citadel/WebcitHttpPort  string  8000" | debconf-set-selections

It works, kind of.  If I run through the standard installer and use these answers to answer the questions normally, Citadel comes right up at 8000 and I can log in.
But when I stuff the config into debconf, suddenly the end of the install doesn't go so well:
 


 Message from syslogd@debian at Feb 22 22:59:22 ...
 citserver[13433]: Modules: Citadel had trouble on starting up. We couldn't bind all ports you configured to be provided by citadel server.#012 This means, citadel won't be the service provider for a specific service you configured it to.#012#012If you don't want citadel to provide these services, turn them off in WebCit via: "Admin->System Preferences->Network".#012#012The failed ports and sockets are: TCP port  0.0.0.0:504: (citadel-TCP) ;TCP port  0.0.0.0:143: (IMAP) ;TCP port  0.0.0.0:993: (IMAPS) ;TCP port  0.0.0.0:2020: (ManageSieve) ;TCP port  0.0.0.0:110: (POP3) ;TCP port  0.0.0.0:995: (POP3S) ;TCP port  0.0.0.0:25: (SMTP-MTA) ;TCP port  0.0.0.0:465: (SMTPs-MTA) ;TCP port  0.0.0.0:587: (SMTP-MSA) ;TCP port  0.0.0.0:5222: (XMPP) #012#012If you want citadel to provide you with that functionality, check the output of "netstat -lnp" on linux Servers or "netstat -na" on *BSD and stop the program that binds these ports.#012 You should eventually remove  their initscripts in /etc/init.d so that you won't get this trouble once more.#012 After that goto "Administration -> Shutdown Citadel" to make Citadel restart & retry to bind this port.#012#012#012To make both ways actualy take place restart the citserver with "sendcommand down"#012#012The errors returned by the system were:#012Error binding to [ 0.0.0.0] : No such file or directory; Error binding to [ 0.0.0.0] : No such file or directory; Error binding to [ 0.0.0.0] : No such file or directory; Error binding to [ 0.0.0.0] : No such file or directory; Error binding to [ 0.0.0.0] : File exists; Error binding to [ 0.0.0.0] : File exists; Error binding to [ 0.0.0.0] : File exists; Error binding to [ 0.0.0.0] : File exists; Error binding to [ 0.0.0.0] : File exists; Error binding to [ 0.0.0.0] : No such file or directory#012#012You can recheck the above if you follow this faq item:#012http://www.citadel.org/doku.php?id=faq:mastering_your_os:net#netstat
 
Broadcast message from systemd-journald@golfoscar.com (Wed 2017-02-22 22:59:22 UTC):
 
citserver[13433]: Modules: Citadel had trouble on starting up. We couldn't bind all ports you configured to be provided by citadel server. This means, citadel won't be the service provider for a specific service you configured it to. If you don't want citadel to provide these services, turn them off in WebCit via: "Admin->System Preferences->Network".
 
The failed ports and sockets are: TCP port  0.0.0.0:504: (citadel-TCP) ;TCP port  0.0.0.0:143: (IMAP) ;TCP port  0.0.0.0:993: (IMAPS) ;TCP port  0.0.0.0:2020: (ManageSieve) ;TCP port  0.0.0.0:110: (POP3) ;TCP port  0.0.0.0:995: (POP3S) ;TCP port  0.0.0.0:25: (SMTP-MTA) ;TCP port  0.0.0.0:465: (SMTPs-MTA) ;TCP port  0.0.0.0:587: (SMTP-MSA) ;TCP port  0.0.0.0:5222: (XMPP)
 
If you want citadel to provide you with that functionality, check the output of "netstat -lnp" on linux Servers or "netstat -na" on *BSD and stop the program that binds these ports. You should eventually remove  their initscripts in /etc/init.d so that you won't get this trouble once more. After that goto "Administration -> Shutdown Citadel" to make Citadel restart & retry to bind this port.To make both ways actualy take place restart the citserver with "sendcommand down" The errors returned by the system were:
Error binding to [ 0.0.0.0] : No such file or directory;
Error binding to [ 0.0.0.0] : No such file or directory;
Error binding to [ 0.0.0.0] : No such file or directory;
Error binding to [ 0.0.0.0] : No such file or directory;
Error binding to [ 0.0.0.0] : File exists;
Error binding to [ 0.0.0.0] : File exists;
Error binding to [ 0.0.0.0] : File exists;
Error binding to [ 0.0.0.0] : File exists;
Error binding to [ 0.0.0.0] : File exists;
Error binding to [ 0.0.0.0] : No such file or directory
 
 
Anyone have a quick solution?
 
 
 


[#] Wed Feb 22 2017 18:09:16 EST from rwb0027 @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

And as the prompts request:

root@debian:~# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2966/sshd
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      13783/webcit
tcp6       0      0 :::80                   :::*                    LISTEN      11747/apache2
tcp6       0      0 :::22                   :::*                    LISTEN      2966/sshd
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     12287    1/init              /run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     27667    11913/dovecot       /var/run/dovecot/stats
unix  2      [ ACC ]     STREAM     LISTENING     27672    11913/dovecot       /var/run/dovecot/ssl-params
unix  2      [ ACC ]     STREAM     LISTENING     27674    11913/dovecot       /var/run/dovecot/login/ssl-params
unix  2      [ ACC ]     STREAM     LISTENING     27678    11913/dovecot       /var/run/dovecot/replicator
unix  2      [ ACC ]     STREAM     LISTENING     30241    13433/citserver     /var/run/citadel/citadel.socket
unix  2      [ ACC ]     STREAM     LISTENING     27682    11913/dovecot       /var/run/dovecot/replication-notify
unix  2      [ ACC ]     STREAM     LISTENING     30243    13433/citserver     /var/run/citadel/citadel-admin.socket
unix  2      [ ACC ]     STREAM     LISTENING     30245    13433/citserver     /var/run/citadel/lmtp.socket
unix  2      [ ACC ]     STREAM     LISTENING     30247    13433/citserver     /var/run/citadel/lmtp-unfiltered.socket
unix  2      [ ACC ]     STREAM     LISTENING     27687    11913/dovecot       /var/run/dovecot/log-errors
unix  2      [ ACC ]     STREAM     LISTENING     8490     1/init              /run/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     27691    11913/dovecot       /var/run/dovecot/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     27695    11913/dovecot       /var/run/dovecot/ipc
unix  2      [ ACC ]     STREAM     LISTENING     27697    11913/dovecot       /var/run/dovecot/login/ipc-proxy
unix  2      [ ACC ]     STREAM     LISTENING     27701    11913/dovecot       /var/run/dovecot/indexer-worker
unix  2      [ ACC ]     STREAM     LISTENING     27705    11913/dovecot       /var/run/dovecot/indexer
unix  2      [ ACC ]     STREAM     LISTENING     27709    11913/dovecot       /var/run/dovecot/doveadm-server
unix  2      [ ACC ]     STREAM     LISTENING     27713    11913/dovecot       /var/run/dovecot/dns-client
unix  2      [ ACC ]     SEQPACKET  LISTENING     8512     1/init              /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     8516     1/init              /run/systemd/journal/stdout
unix  2      [ ACC ]     STREAM     LISTENING     27717    11913/dovecot       /var/run/dovecot/director-admin
unix  2      [ ACC ]     STREAM     LISTENING     27721    11913/dovecot       /var/run/dovecot/director-userdb
unix  2      [ ACC ]     STREAM     LISTENING     27725    11913/dovecot       /var/run/dovecot/dict
unix  2      [ ACC ]     STREAM     LISTENING     27729    11913/dovecot       /var/run/dovecot/config
unix  2      [ ACC ]     STREAM     LISTENING     27731    11913/dovecot       /var/run/dovecot/login/login
unix  2      [ ACC ]     STREAM     LISTENING     27733    11913/dovecot       /var/run/dovecot/token-login/tokenlogin
unix  2      [ ACC ]     STREAM     LISTENING     27737    11913/dovecot       /var/run/dovecot/auth-login
unix  2      [ ACC ]     STREAM     LISTENING     20058    6676/python         /var/run/fail2ban/fail2ban.sock
unix  2      [ ACC ]     STREAM     LISTENING     27741    11913/dovecot       /var/run/dovecot/auth-client
unix  2      [ ACC ]     STREAM     LISTENING     27745    11913/dovecot       /var/run/dovecot/auth-userdb
unix  2      [ ACC ]     STREAM     LISTENING     27749    11913/dovecot       /var/run/dovecot/auth-master
unix  2      [ ACC ]     STREAM     LISTENING     27753    11913/dovecot       /var/run/dovecot/auth-worker
unix  2      [ ACC ]     STREAM     LISTENING     27757    11913/dovecot       /var/run/dovecot/anvil
unix  2      [ ACC ]     STREAM     LISTENING     27761    11913/dovecot       /var/run/dovecot/anvil-auth-penalty
unix  2      [ ACC ]     STREAM     LISTENING     12284    1/init              /var/run/dbus/system_bus_socket

 



[#] Fri Feb 24 2017 11:05:12 EST from bennabiy @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

I don't want to speak too soon, but it seems the error is no longer popping up after lowering the idle timeout time to 3 minutes (from 15 minutes).

 

Any chance that could have been all it was?



[#] Fri Feb 24 2017 21:45:24 EST from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

 

Do you think that would cause it? Running out of workers, and then the connections piling up without anyone to talk to? I still do not know why there are that many SMTPS connecitons, my userbase is only about 275.

That could definitely be it ... stale connections piling up faster than the server can time them out.   And yes 15 minutes is the default timeout, but it probably doesn't need to be anywhere near that long, because SMTP connections should never go idle, and IMAP connections don't go idle as long as the client keeps polling for mail or sending keepalives or whatever.

Run it for a while with the lower timeout and let's see!

 



Go to page: First ... 6 7 8 9 [10] 11 12 13 14 ... Last