Citadel Development

Hi,

while building the 8.02 RPMS in OBS I got the following warnings:

Webcit:

W: incorrect-fsf-address /usr/share/citadel-webcit/static/wclib.js
The Free Software Foundation address in this file seems to be outdated or
misspelled.  Ask upstream to update the address, or if this is a license file,
possibly the entire file with a new copy available from the FSF.

W: zero-length /usr/share/citadel-webcit/static/t/room/edit/submit.html
W: zero-length /usr/share/citadel-webcit/static/t/sieve/empty.html
W: version-control-internal-file /usr/share/citadel-webcit/static/.gitignore
You have included file(s) internally used by a version control system in the
package. Move these files out of the package and rebuild it.

Citadel:

I: binary-or-shlib-calls-gethostbyname /usr/sbin/citadel-setup
I: binary-or-shlib-calls-gethostbyname /usr/sbin/citserver
The binary calls gethostbyname(). Please port the code to use getaddrinfo().

Regards
Stefan

Thu Nov 24 2011 11:40:31 ESTfrom Stefan @ Uncensored Subject: Warnings

Hi,

while building the 8.02 RPMS in OBS I got the following warnings:

Webcit:

W: incorrect-fsf-address /usr/share/citadel-webcit/static/wclib.js
The Free Software Foundation address in this file seems to be outdated or
misspelled.  Ask upstream to update the address, or if this is a license file,
possibly the entire file with a new copy available from the FSF.

W: zero-length /usr/share/citadel-webcit/static/t/room/edit/submit.html
W: zero-length /usr/share/citadel-webcit/static/t/sieve/empty.html


W: version-control-internal-file /usr/share/citadel-webcit/static/.gitignore
You have included file(s) internally used by a version control system in the
package. Move these files out of the package and rebuild it.

Citadel:

I: binary-or-shlib-calls-gethostbyname /usr/sbin/citadel-setup


I: binary-or-shlib-calls-gethostbyname /usr/sbin/citserver
The binary calls gethostbyname(). Please port the code to use getaddrinfo().

Regards
Stefan

well, thats probably a little bigger doing...

we're going to take the 3rd road: going with c-ares. most probably that migration won't be done before 8.20

please make shure you split sending + receiving into two functions; and collect all vars needed in both on a struct.

this will be essential to ease the work of migrating it to event driven programming. 

Sun Nov 27 2011 11:27:15 EST from IGnatius T Foobar @ Uncensored

On the other hand we have to determine whether going event driven is appropriate for this function, since it is expected to have interactive performance rather than background. 

in this case, "performance" means something else.

since an http request does a DNS request, opens an external TCP connection anywhere across the 'net which might be randomly slow... so performing this operation doesn't use local resources directyl; but it might take up to several seconds to be finished.

so by doing this "inline" you block a worker thread for a real long time; this worker could do some other usefull operations while the async io (libev..) is select(epoll-)'ing   for that external event to happen.

so for each external http request you basicaly have to fork another worker thread  (which does that workers job while its blocking...); which is the  "real" resource consumption of an non async aproach

Once the http thread finishes, the asyncio task has to wake up some worker to pick up the citserver<->webcit conversation again and reply to the user.

I don't expect the enduser to directly notice the difference between one of the approach on an idle system; no profiler will show it either.

I somewhere had a nice graph showing how much time it takes to retrieve an information from (each line up to 10^2 magnitudes in time more) the sequence looked something like that:

cpu register

cpu cache 

computers memory 

in cluster memory

------1

disk i/o

-----2

database i/o [berkeley DB in our case]

-----3

local network i/o (in the direct reach without more ip hops; so basicaly "in the same building")

external i/o

external i/o to anywhere around the world

the last one is what we do with that openid; waiting for that should consume as less resources as possible inside of the citadel application stack; so making the evio epoll() and collect the HTTP response until we get back on parsing & evaluating the reply  is definitely a must.

----[1|2|3] marks the boundaries in which we have to decide by using different event loops; everything above ----1 is expected to be "fast evough" to fit in everywhere in some of the loops now and then; while 2 & 3 are that different slow, that 2 is done by one queue which simply "waits" for the next slot to become available, while 3 uses poll() to aggregate from the kernel when something relevant for the above layers happenes.

To exploit an inline approach, an attacker simply would have to start many openid requests; each request swawns a thread waiting for an external resource (which in worst case the attacker also controls to send replies even more slow)  and our citserver would have to start forking workers until there is no more & the system is dead.

In comparison in the async IO case the resource useage is the memory required by the citadel context, the http context + the number of filedescriptors;

which we obviously have much more available than worker threads.

Webcit will currently remain the weak link in that chain; it might be subject to change later on.

oh, and unless you do stuff like...

for (foo = somestring; strlen(foo) > 0); foo++)

or some weird lots of DB-queries 

I wouldn't expect anything below ----2 to be byound of the threshhold _one_ user could notice; this just all gets important once you want to serve lots of users.

however, the async approach isn't about making something slower to the user; its primarily about using the cpu more effectively while waiting for the IO to that user.

in fact this makes stuff even faster to the user in some cases; 

i.e. i3wm.org uses libev + XCB to manage its conversation with the X11 server; which makes it in fact lots faster than other windowmanagers around; 

these other windowmanagers will send a resize directive to the X11 server and wait for its reply to continue their command flow; while i3wm uses XCB to serialize the X11 command (which is somewhat like CORBA just to name a comparison) and send it plus using libev to wait for the reply; it then can continue doing other stuff [using XCB to send more requests...) without having to wait for the first command to continue; once the reply is there, XCB deserialises that status sent by the X-server and continues this control flow and commands basing on the results of that command.

think of it as "multiplexing" the cpu since the cpu to file i/o  since is lots faster (see table of last post).

If you still have doubts of asyncio having negative impact on performance to the end-user, please let me know.

Starting with Citadel 8.01 (maybe 8.00?) and still present in 8.02, the text client no longer seems to obey the "fullscreen" or "local_screen_dimensions" directives in the citadel.rc (or .citadelrc).

I routinely run a 40+ line terminal screen (PuTTY, Terminal from my Mac Book), and starting with 8.01, with "fullscreen=yes" (personal .citadelrc) and "local_screen_dimensions=1" (personal .citadelrc and /usr/local/citadel/citadel.rc), I get a prompt at 24 lines, and no blue status bar at the bottom of my session.

And looking at the citadel.rc file that came with the citadel-8.02.tar.gz, I no longer see the "fullscreen" option, though "local_screen_dimensions" is still present.

Were the options removed from the actual text client? If so, can we have them back?

Thanks.

Same happens if I ssh in using the citadel user name.

Definitely not working.

Attached is screen shot. I've used PuTTY to ssh to my server, then run "citadel <myhost>", and the prompt appears on line 24.

ok, got the bug.

its iceweasel 8 to send requests bytewise (which is definitely a performance regression...) 

fix to make webcit cope with it committed.

please upgrade uncensored soon so I can stop using chrome again ;-)