Wed Feb 05 2025 22:05:23 UTCfrom IGnatius T Foobar Subject: Re: User metadata storage?

I was wondering if the database backend has a plugin system for adding
similar expansion packs. If it doesn't, what would it take to add it?

I'm not sure I understand what you want to do. Are you looking to run Citadel Server with a database other than Berkeley DB? If so then yes, we have the framework in place to add more storage drivers. LMDB is what we had in mind for the future.

Or are you looking to have new types of applications running on top of Citadel?

If so, you might like to play around with the "views" framework that we use to determine what type of application runs inside each room (mailbox, forum, calendar, address book, wiki, etc). If you're serious about doing that it wouldn't be too much of an effort to get a pluggable type of thing going to add more -- sort of like people do now with NextCloud and similar platforms.

Thanks! I was thinking of a training video type of app view where a user could add access to a floor by subscription, watch a video from an embedded link in a read-only wiki view, advance to the next room when done watching, take a multiple-choice quiz generated from ready-made questions in random order with the answers listed in random order as well, be disallowed access to any additional rooms on that floor until they pass the quiz, read a caption from another room as read-only wiki text, mail feedback in to the grader through a mailbox view, interact with other students via chatroom under supervision of their instructor with multiple sessions going concurrently and so on. Obviously, a video link will need a browser or custom app like I'm already making, but the views framework sounds like what I was thinking of.

Also, my question about the user metadata was: storing quiz data, subscriptions to multiple floors, chat logs and subscription tiers within the user account records in the database. If the database framework can handle it, making views for polls and other types of interactions might be more practical. If it's not possible, it's not a deal breaker. I wouldn't expect all of the views described, to work in WebCit nor WebCitNG to handle this directly either, unless the views framework is extensible.

Wed Feb 05 2025 17:40:15 UTC from wojciech.krolikowski Subject: Citadel server hangs 

Server is running on RPi 3B+ with 256GB SSD drive connected via USB3 - SSD cable.

I thought the RasPi 3 series could only handle USB 2.0 as a maximum. RasPi 4 added a different USB controller chip for USB 3.0, I think.

Also, my question about the user metadata was: storing quiz data, subscriptions to multiple floors, chat logs and subscription tiers within the user account records in the database. If the database framework can handle it, making views for polls and other types of interactions might be more practical. If it's not possible, it's not a deal breaker. I wouldn't expect all of the views described, to work in WebCit nor WebCitNG to handle this directly either, unless the views framework is extensible.

Right, so this isn't unreasonable, and there was a time when we had envisioned "any application type" running on top of Citadel, sort of like Lotus Notes.

Writing directly into the database is not how you'd approach that.  Basically you'd combine three functional areas of the Citadel system to do new applications:

1. The fundamental one is of course a room's View, which tells the system what type of application is running in the room.  There are a few which are variants or compatible with each other; for example you could view some rooms as a Forum, a Mailbox, or a Blog.  The "native view" is stored in the system and tells us how it was created and will appear to new users; the "account local view" tells us how the user prefers to view it.  For example, a user might change the view of a room from a Forum to a Blog, but they wouldn't change it to (for example) a calendar or wiki, because they have different metadata.

In fact, take a look at how we do Wiki rooms for a great example of how to "write an application" on top of Citadel!

2. Custom MIME types.  Just about everything gets stored on top of the message base.  Anything that needs an "index" is done using the EUID of individual items stored in the message base.  That's how we look up individual items like calendar events and Wiki topics without having to slow-scan the entire room.

Look at how we store the outbound SMTP queue for a great example of how we do that.

3. Application hooks.  This is the most difficult to implement because it requires extending Citadel Server, but maybe we can plug in an extension language if you want to do a lot of them.  Basically we have all sorts of things that can trigger application hooks, like saving a message (both before-save and after-save hooks), user login and logout, etc.  If you need back end application logic, this is the way to do it.

I have an issue. Sometimes server just hangs. No errors in journalctl, systemctl shows that it is running, ps - no issues, no zombies. But I can't get or send email, webcit is not responding.

Are you using the Docker container or the Easy Install deployment?

Hi,

Easy Install on Rasbian. I dig down into logs and find that Citadel hangs when SpamAssassin is response slowly. SpamD is working as I can connect via spamc, but it is really slow. When I restart spamD it is working without issues and fast as it should. But Citadel remain hanging. Then I have to kick Citadel service and everything get back on tracks. Currently CRON reboots spamD everyday at midnight, it is working, but I this is not a solution.

Citadel and spamD are running 2 different RPis.

Answering previous question: cable/adapter is USB3, RPi has only USB2, but I mention that as sometimes USB2.0<->SATA interface is causing transfer speed issues. I have 5 RPis in my "server room" and I had such problems in the past (mainly related to cache read/write). Switching to USB3 adapters solved problems with SSD drives.

Have an absolutely great day :)

Thu Feb 06 2025 23:37:40 UTC from IGnatius T Foobar Subject: Re: Citadel server hangs

I have an issue. Sometimes server just hangs. No errors in journalctl, systemctl shows that it is running, ps - no issues, no zombies. But I can't get or send email, webcit is not responding.

Are you using the Docker container or the Easy Install deployment?

 

Thu Feb 06 2025 23:36:19 UTCfrom IGnatius T Foobar Subject: Re: User metadata storage?

Right, so this isn't unreasonable, and there was a time when we had envisioned "any application type" running on top of Citadel, sort of like Lotus Notes.

3. Application hooks.  This is the most difficult to implement because it requires extending Citadel Server, but maybe we can plug in an extension language if you want to do a lot of them.  Basically we have all sorts of things that can trigger application hooks, like saving a message (both before-save and after-save hooks), user login and logout, etc.  If you need back end application logic, this is the way to do it.

I'll probably have to do this but I'll definitely be keeping in contact before pushing merge requests. My first modification will likely involve external POSIX filters to allow encryption and compression to be maintained as external to the Citadel framework. The most obvious usage would be using XZ compression and PGP encryption. I plan on keeping them external in case of license incompatibility of future add-on filters for stronger encryption and compression algorithms but we can discuss that later.

The basic idea will be to use streamed data filters piped end-to-end to send the data out and receive it back over the 'net. Since tarballs are always formed by piping Tar streams through filters, this shouldn't be too difficult for a POSIX based server. I assume there is a global hook that can be piped into and out of for such an application. I may outgrow my little Cubox i4Pro server before too long though. ;-)

Adding hook types is reasonable as they are generalized so we can use them for lots of different applications.  It would probably also make sense to have them run either in an interpreted language or as a callout to a separate binary so that individual applications don't have to be directly compiled in to the server.  Think of it in terms of a plugin pack that would both extend both the front and back end components of the system.

If you want to get into details, when you're ready, let's discuss it in the development room.

Fri Feb 07 2025 19:55:42 UTC from IGnatius T Foobar Subject: Re: User metadata storage?

If you want to get into details, when you're ready, let's discuss it in the development room.

Ok. Sounds good. When I get that far, I will.

Dear all,

This happened now some times.

When I send an email it stalls the sending (it delays a lot in the sending box and it does not syncronize) and the recipient receives multiple copies of the email. Also when somebody sends to me I receive multiple copies. And the server do not sysncronize. I only see the multiple copies when I do "systemctl restart citadel" (and it solves the problem).

I have no screenshot of the command "systemctl status citadel" before the restart. I send the result of "systemctl status citadel" after restart. Despite I remember, before the restart, that citadel was up and running (with added reference to spamassassin in the status).

Thanks,

Luís Gonçalves.

Hi,

where can i define, wich room will displayed for a unregistered user ? 

Thu Feb 13 2025 14:21:50 UTC from IGnatius T Foobar Subject: Re: First Room

where can i define, wich room will displayed for a unregistered user ? 

Are you looking to select the landing page/room for a user who hasn't logged in yet? That is a command line option for WebCit (and it can be used in the command line options for the container as well, if you're using that). Basically you're going to supply the "-g" option to either the container or WebCit followed by the URL within the system that you want it to land on. So you could go to a room, or list users or show who is online or whatever.

For example:

webcit [other options] -g "/dotgoto?My Welcome Room"

You might have noticed we have that set up here. We used a Wiki room to have the most flexibility in creating the experience for unregistered users.

Yes, thats it :) 

A Landing Page, best in Wiki style like here. In Germany i need to place special Links on it for privacy policy and imprint. It must be reachable directly from the Landingpage. Stupid, but its the law here :)

Thanks for your help

Thu Feb 13 2025 14:48:57 UTC from Harvey Subject: Re: First Room

 

Thu Feb 13 2025 14:21:50 UTC from IGnatius T Foobar Subject: Re: First Room

where can i define, wich room will displayed for a unregistered user ? 

Are you looking to select the landing page/room for a user who hasn't logged in yet? That is a command line option for WebCit (and it can be used in the command line options for the container as well, if you're using that). Basically you're going to supply the "-g" option to either the container or WebCit followed by the URL within the system that you want it to land on. So you could go to a room, or list users or show who is online or whatever.

For example:

webcit [other options] -g "/dotgoto?My Welcome Room"

You might have noticed we have that set up here. We used a Wiki room to have the most flexibility in creating the experience for unregistered users.

Yes, thats it :) 

A Landing Page, best in Wiki style like here. In Germany i need to place special Links on it for privacy policy and imprint. It must be reachable directly from the Landingpage. Stupid, but its the law here :)

 

Thanks for your help

 

You mean on bash

/usr/local/webcit/ and then ./webcit ?? -g "/dotgoto?My Welcome Room" ?

What you mean with [other options] ?

Greetings

There is a problem when I try to make a rule for moving emails with the X-Spam-Status: True header to the spam folder, citadel crashes. Restarting the citadel.service service does not help, it crashes again. I have to restore everything from the backup again. Restoring only the database does not help, I have to rewrite the entire /usr/local/citadel directory. Has anyone had similar cases? How to fix it? I'm tired of constantly losing emails. My server is on Orange Pi 5 4GB, Debian 11 ARM64 OS

$ file /usr/local/citadel/citserver
/usr/local/citadel/citserver: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=116b0efcfd50697924880cf1089992ea8332ba57, for GNU/Linux 3.7.0, with debug_info, not stripped

Running: Citadel Server 1001 with WebCit 1009
Server Build: 1001

How to get a stack trace?

Today I was run "curl https://easyinstall.citadel.org/install | bash". After 5 minutes citadel.service failed, I try to restart and after few seconds fail again! (((((((((((((((((((((((( I restored again from morning backup

I found the problem, but did not find the solution. When updating citadel, only webcit is updated, but the database and the server itself are not updated. Accordingly, when restoring the server from a backup, the server and DB version are 1001, after starting the update, the server and DB are updated and an error occurs with the database files and the service crashes. Why is the database not converted during the update?

Also, for testing purposes, I installed citadel on a new server and loaded the DB dump, after which there is also a failure. I noticed that on the old server, the dump is made with the utility from the DB version 1001, and on the new server, the dump is loaded for the DB version 1009, and errors about damaged records pour in. After the dump is loaded, the citadel service starts and I can log in as a regular user, a user with administrative rights cannot log in, an incorrect password error occurs. And if you restart the reinstallation after this via easyinstall, everything crashes into an error.

I did another test. I installed a clean citadel installation on another server, then copied the database directory from the working server to the test server. After that, citadel started and worked successfully, and the database version was updated to version 1009. After that, I went again and set up a new mail sorting rule, citadel failed again. I deleted the database files again and copied them from the archive, citadel started and did not fail anymore. I decided to check again and went to the section for changing my contact information, made changes and citadel failed again after a few seconds. (((((((
I currently have the following version on my test server:
Running: Citadel Server 1009 with WebCit 1009
Server Build: 1009

Feb 19 20:28:24 deb-ams1-01 citserver[40754]: Existing database version on disk is 1001
Feb 19 20:28:24 deb-ams1-01 citserver[40754]: citserver[40754]: Existing database version on disk is 1001
Feb 19 20:28:24 deb-ams1-01 citserver[40754]: citserver[40754]: Running post-startup database upgrades.
Feb 19 20:28:24 deb-ams1-01 citserver[40754]: citserver[40754]: extensions: service DICT_TCP has been manually disabled, skipping
Feb 19 20:28:24 deb-ams1-01 citserver[40754]: citserver[40754]: extensions: TCP port *:5222: (XMPP) registered.
Feb 19 20:28:24 deb-ams1-01 citserver[40754]: Running post-startup database upgrades.

After restarting citadel:

Feb 19 20:28:49 deb-ams1-01 citserver[40763]: extensions: unix domain socket 'lmtp.socket': registered.
Feb 19 20:28:49 deb-ams1-01 citserver[40763]: extensions: unix domain socket 'lmtp-unfiltered.socket': registered.
Feb 19 20:28:49 deb-ams1-01 citserver[40763]: Existing database version on disk is 1009
Feb 19 20:28:49 deb-ams1-01 citserver[40763]: citserver[40763]: Existing database version on disk is 1009

Any change to the user settings causes a crash.