Language:
switch to room list switch to menu My folders
Go to page: [1] 2 3 4 5 ... Last
↑↑↑ Old messages ↑↑↑            ↓↓↓ New messages ↓↓↓
[#] Tue Jun 20 2017 18:35:41 EDT from athos-mn @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

The purge cleaned up the log files, but I'm still seeing a really large database; is there a way to clean it up?



[#] Wed Jun 21 2017 13:19:22 EDT from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

Unfortunately you are seeing the natural way that Berkeley DB allocates disk space. The files never shrink; instead, empty space is marked for re-use later on. As a result, the size of the file isn't actually the size of the data; it's more like a "high water mark."

See also: http://www.citadel.org/doku.php/faq:systemadmin:disk_space

If this is highly undesirable for some reason, you can (after taking a backup, of course) shut down Citadel server and run database_cleanup.sh to dump-and-load your database. Keep in mind, however, that database_cleanup.sh requires enough free space on your disk to hold both the dump and the database itself, so if you're doing this because of a disk space crunch, make sure there's an amount of free space on the filesystem containing /tmp that is *at least* double the size of your database.

Hmm. Now I'm seeing there's a function called db->compact()
https://docs.oracle.com/cd/E17275_01/html/api_reference/C/dbcompact.html

Maybe that can be worked into the code, but I don't see a command line utility to do that.

[#] Tue Jun 27 2017 11:38:04 EDT from bennabiy @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

That would be a worthy thing to look into....

Wed Jun 21 2017 01:19:22 PM EDT from IGnatius T Foobar @ Uncensored
Unfortunately you are seeing the natural way that Berkeley DB allocates disk space. The files never shrink; instead, empty space is marked for re-use later on. As a result, the size of the file isn't actually the size of the data; it's more like a "high water mark."

See also: http://www.citadel.org/doku.php/faq:systemadmin:disk_space

If this is highly undesirable for some reason, you can (after taking a backup, of course) shut down Citadel server and run database_cleanup.sh to dump-and-load your database. Keep in mind, however, that database_cleanup.sh requires enough free space on your disk to hold both the dump and the database itself, so if you're doing this because of a disk space crunch, make sure there's an amount of free space on the filesystem containing /tmp that is *at least* double the size of your database.

Hmm. Now I'm seeing there's a function called db->compact()
https://docs.oracle.com/cd/E17275_01/html/api_reference/C/dbcompact.html

Maybe that can be worked into the code, but I don't see a command line utility to do that.

 



[#] Tue Jun 27 2017 20:35:29 EDT from bennabiy @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

Specifically, this portion makes for a hopeful appearance...

 

flags

The flags parameter must be set to 0 or one of the following values:

 

  • DB_FREELIST_ONLY

    Do no page compaction, only returning pages to the filesystem that are already free and at the end of the file. This flag must be set if the database is a Hash access method database.

  • DB_FREE_SPACE

    Return pages to the filesystem when possible. If this flag is not specified, pages emptied as a result of compaction will be placed on the free list for re-use, but never returned to the filesystem.

    Note that only pages at the end of a file can be returned to the filesystem. Because of the one-pass nature of the compaction algorithm, any unemptied page near the end of the file inhibits returning pages to the file system. A repeated call to the DB->compact() method with a low compact_fillpercent may be used to return pages in this case.

 

 

Wed Jun 21 2017 01:19:22 PM EDT from IGnatius T Foobar @ Uncensored
Unfortunately you are seeing the natural way that Berkeley DB allocates disk space. The files never shrink; instead, empty space is marked for re-use later on. As a result, the size of the file isn't actually the size of the data; it's more like a "high water mark."

See also: http://www.citadel.org/doku.php/faq:systemadmin:disk_space

If this is highly undesirable for some reason, you can (after taking a backup, of course) shut down Citadel server and run database_cleanup.sh to dump-and-load your database. Keep in mind, however, that database_cleanup.sh requires enough free space on your disk to hold both the dump and the database itself, so if you're doing this because of a disk space crunch, make sure there's an amount of free space on the filesystem containing /tmp that is *at least* double the size of your database.

Hmm. Now I'm seeing there's a function called db->compact()
https://docs.oracle.com/cd/E17275_01/html/api_reference/C/dbcompact.html

Maybe that can be worked into the code, but I don't see a command line utility to do that.

 



[#] Thu Jun 29 2017 08:48:33 EDT from jp10558 @ Uncensored

Subject: Sync with Android phone

[Reply] [ReplyQuoted] [Headers] [Print]

So I'm trying to sync with my Android phone. Citadel ought to support one of the CalDav sync tools per the docs, but I don't have a lot of luck. Some tools work one way, others work another. Anywho, the one I'd like to use is DAVDroid, but it cannot list the calendar entries. Strangely, it seems to be able to sync up new entries from my phone. I've provided a lot of details to DavDroid here:

https://forums.bitfire.at/topic/1467/sync-errors-with-citadel-groupware-server-calendar

but the response from them per debug output is it's a Citadel / Server problem because:

The server sends an 500 Internal server error on a CalDAV request. Please contact server support.

In that thread I provide a lot of details so I'll copy the rest of my post here:

I have been running the Citadel groupware server from www.citadel.org for years. Recently I needed a calendar to share with others, and as it claims to support that as well as ical/caldav I figured I use the same system. The Calendar works fine from Thunderbird. DavDroid seems to be able to sync changes from my phone up to the calendar, but not entries from the server down to the phone. So currently I end up using 2 products, iCal Import/Export 3.1 to sync down entries (if I try and use it to upload entries it seems to wipe the server's calendar), and DavDroid to sync changes back up. I'd of course like to get down to one product, and DavDroid seems more simple in the UI if I could get it fully functional.

Citadel is a bit wonky on versioning now. I'm running on Scientific Linux 6.8 (RHEL derivative) and Citadel 907. I have DavDroid 1.6.2-ose from F-Droid repo on a Samsung Note 5 phone running Android 7 Security patch level May 1,2017 kernel version 3.10.61-10809541. Debug info here:
https://pastebin.com/R1NM0pAb

 

So on the Citadel side - any ideas?



[#] Sun Jul 02 2017 00:52:29 EDT from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

Hmm. Now I'm seeing there's a function called db->compact()
https://docs.oracle.com/cd/E17275_01/html/api_reference/C/dbcompact.html


A quick update:

I've tested db->compact() with Citadel Server, and it works as advertised.

So you can expect the next release of the server program to contain, at the very least, a site-configurable option to enable database compaction after the auto-purger is run. (I might make it a hidden setting, just to be able to get it out there faster.)

[#] Sun Jul 02 2017 00:54:02 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: Sync with Android phone

[Reply] [ReplyQuoted] [Headers] [Print]

So I'm trying to sync with my Android phone. Citadel ought to support
one of the CalDav sync tools per the docs, but I don't have a lot of
luck. Some tools work one way, others work another. Anywho, the one

Are you trying to make it use CalDAV? The current version doesn't support CalDAV yet. (It's being worked on as part of WebCit-NG though.)

The current version will support the idea of loading/saving the entire calendar as an .ics file over regular WebDAV. This has been tested with Android and it does work.

[#] Mon Jul 03 2017 06:28:05 EDT from Christian.Commarmond @ Uncensored

Subject: Big number of messages...

[Reply] [ReplyQuoted] [Headers] [Print]

Hi,

I installed Citadel and try to synch with my 1,500,000 messages gmail account. (OK, I receive more than 2000 per day, mostly automatized reports).

I try to do the synch with imapsync. It is super slow. I will try soon to do the initial loading from thunderbird.

 

My concern is that, when I go in a folder which contains nearly 200,000 messages, I get the good number at top of screen, but I can only see around 32000 messages (I guess, because I see only to 19th of may 2015). And worse, only the olders...

 

Questions:

Is this normal? Can I override this? Can I get to see the newest messages?

 

Thank you for your help.

 

Christian

 

 



[#] Mon Jul 03 2017 17:41:00 EDT from s3cr3to @ Uncensored to Citadel_Support <room_Citadel_Support@uncensored.citadel.org>

Subject: db->compact() on 8.17

[Reply] [ReplyQuoted] [Headers] [Print]

On 07/01/2017 10:52 PM, IGnatius T Foobar wrote:

A quick update:

I've tested db->compact() with Citadel Server, and it works as advertised.


So you can expect the next release of the server program to contain, at the
very least, a site-configurable option to enable database compaction after
the auto-purger is run. (I might make it a hidden setting, just to be able
to get it out there faster.)

I read that message, and I wonder if is possible for you to create a
tool to compact the db on Citadel 8.17.

My current version "Citadel 8.17" Debian Squeeze; currently using 67G
db.08. A few months ago, I understand and configure the "Expire by
message age/count" for the majority of my users, except the very
important ones.

I cross my fingers and hope you have a positive response :)

P.S. Or if you can point me in the right direction, so I can investigate
what libraries/language(python/bashscrip/pascal)/way to connect and run
the db.compact().

P.S.2. some months ago, I manage to read the users from the db using the
command line (bash+python?), but I don't remember how I do, need to dig
on my harddisk, maybe I can do it :O ?!

Regards

[#] Tue Jul 04 2017 16:12:15 EDT from bennabiy @ Uncensored

Subject: db->compact()

[Reply] [ReplyQuoted] [Headers] [Print]

Any noticeable performance hit?

Sun Jul 02 2017 12:52:29 AM EDT from IGnatius T Foobar @ Uncensored
Hmm. Now I'm seeing there's a function called db->compact()
https://docs.oracle.com/cd/E17275_01/html/api_reference/C/dbcompact.html


A quick update:

I've tested db->compact() with Citadel Server, and it works as advertised.

So you can expect the next release of the server program to contain, at the very least, a site-configurable option to enable database compaction after the auto-purger is run. (I might make it a hidden setting, just to be able to get it out there faster.)

 



[#] Wed Jul 05 2017 10:53:52 EDT from Freakdog @ Dog Pound BBS II

Subject: Re: db->compact() on 8.17

[Reply] [ReplyQuoted] [Headers] [Print]

 

Mon Jul 03 2017 05:41:00 PM EDT from s3cr3to @ Uncensored Subject: db->compact() on 8.17
On 07/01/2017 10:52 PM, IGnatius T Foobar wrote:

A quick update:

I've tested db->compact() with Citadel Server, and it works as advertised.


So you can expect the next release of the server program to contain, at the
very least, a site-configurable option to enable database compaction after
the auto-purger is run. (I might make it a hidden setting, just to be able
to get it out there faster.)
I read that message, and I wonder if is possible for you to create a
tool to compact the db on Citadel 8.17.

My current version "Citadel 8.17" Debian Squeeze; currently using 67G
db.08. A few months ago, I understand and configure the "Expire by
message age/count" for the majority of my users, except the very
important ones.

I cross my fingers and hope you have a positive response :)

P.S. Or if you can point me in the right direction, so I can investigate
what libraries/language(python/bashscrip/pascal)/way to connect and run
the db.compact().

P.S.2. some months ago, I manage to read the users from the db using the
command line (bash+python?), but I don't remember how I do, need to dig
on my harddisk, maybe I can do it :O ?!

Regards

It may be easier to run the easy_install option and upgrade your code...you can then move the DB and messages files over to the appropriate directories under the new directory structure, which will bring you up to current.



[#] Wed Jul 05 2017 11:42:57 EDT from jp10558 @ Uncensored

Subject: Re: Big number of messages...

[Reply] [ReplyQuoted] [Headers] [Print]

I have the same issue with a non synced, just citadel server. My messages in the Inbox stop around Jan 3rd, yet using IMAP I get messages through today... I'm not sure this is specific to a large number of messages or not, but I have 11417 messages according to webcit in my Inbox.



[#] Wed Jul 05 2017 12:30:42 EDT from Christian.Commarmond @ Uncensored

Subject: Re: Big number of messages...

[Reply] [ReplyQuoted] [Headers] [Print]

 

Wed Jul 05 2017 11:42:57 EDT from jp10558 @ Uncensored Subject: Re: Big number of messages...

I have the same issue with a non synced, just citadel server. My messages in the Inbox stop around Jan 3rd, yet using IMAP I get messages through today... I'm not sure this is specific to a large number of messages or not, but I have 11417 messages according to webcit in my Inbox.



Hi JP,

I think that may be a bug in webcit. Which cannot query all messages , but worse, read first the oldests...

Anyway, with imapsync, my synch will never finish. It seems that it tries to reread all the source each time... 

And it looks like only one people is still working on this project. He cannot do everything. I have a 12 hours plane tonight. I am downloading the C code to have a look. Who knows, maybe a miracle will show me the problem...

Don't hope too much... 

 

Christian



[#] Wed Jul 05 2017 13:55:18 EDT from s3cr3to @ Uncensored to Citadel_Support <room_Citadel_Support@uncensored.citadel.org>

Subject: Re: db->compact() on 8.17

[Reply] [ReplyQuoted] [Headers] [Print]

Easy_install/current I want to do that... but really I got a lot of work
in my hands. I don't lose my hope yet: to migrate to easy_install in
this life :)

Just that first I want to compact the DBs, that huge file I hope it has
a lot of unused space because my ignorance to manage correctly the
expired messages.

Thanks Freakdog

[#] Wed Jul 05 2017 16:16:39 EDT from s3cr3to @ Uncensored to IGnatius T Foobar <ajc@citadel.org>

Subject: Re: db->compact() on 8.17

[Reply] [ReplyQuoted] [Headers] [Print]

IG, trying to compile the source, I got this errors (solved I hope)

I only have this questions:
* The db_version_major can be 4?
* Can I run this tool on an backup directory to test it?

* In my system I got this db version:
i db4.6-util 4.6.21-16

Searching I found this:
https://stackoverflow.com/questions/2814686/what-is-the-difference-between-the-different-berkeleydb-versions-and-which-shou/2823053#2823053

"4.4: Adds Database compaction, in-memory databases, Peer-to-Peer HA"

I wonder if is ok just to change this lines:

#if DB_VERSION_MAJOR < 5
#error Citadel requires Berkeley DB v5.0 or newer. Please upgrade.
#endif

to:
#if DB_VERSION_MAJOR < 4
#error Citadel requires Berkeley DB v4.4 or newer. Please upgrade.
#endif

* using aptitude i found and install both:
Setting up libdb4.8-dev (4.8.30-2) ...
Setting up libdb-dev (4.8) ...
Setting up zlib1g-dev (1:1.2.3.4.dfsg-3) ...

* and now I can compile and got this:
# cc ctdlsqueeze.c -ldb -o ctdlsqueeze
# ls -l
total 16
-rwxr-xr-x 1 root root 9483 Jul 5 14:05 ctdlsqueeze
-rw-r--r-- 1 s3 s3 3823 Jul 5 13:50 ctdlsqueeze.c

I'm very rusted on C... I will try to create a "dummy" variant to answer
my previous questions working with backups.

Regards

[#] Mon Jul 10 2017 14:39:31 EDT from janik @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

Hey everyone,

 

i can send emails but not receive. error is 551 relaying denied..

what could i do?

 

 



[#] Mon Jul 10 2017 21:10:21 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: Big number of messages...

[Reply] [ReplyQuoted] [Headers] [Print]

My concern is that, when I go in a folder which contains nearly
200,000 messages, I get the good number at top of screen, but I can
only see around 32000 messages (I guess, because I see only to 19th
of may 2015). And worse, only the olders...

Are you viewing the folder with an IMAP client, or with WebCit?

IMAP does not, to the best of my knowledge, have a fixed limit of how many messages it can present. WebCit does, but if you have the source code online I can point out where to change it.

WebCit-NG will not have any such limits, but that doesn't help you right now :)

[#] Mon Jul 10 2017 21:11:38 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: db->compact()

[Reply] [ReplyQuoted] [Headers] [Print]

Any noticeable performance hit?

Very much so, which is why it will not be enabled by default, and even when enabled it will only run once per night, after the completion of an auto-purger run.

[#] Mon Jul 10 2017 21:13:12 EDT from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

i can send emails but not receive. error is 551 relaying denied..

what could i do?

First make sure it's actually Citadel that's listening on port 25 of your server. Telnet to port 25, and you should see something like

220 uncensored.citadel.org ESMTP Citadel server ready.

If it doesn't say "Citadel" somewhere in the greeting, you have another mail server running, and it needs to be shut off.

[#] Mon Jul 10 2017 21:55:14 EDT from Christian.Commarmond @ Uncensored

Subject: Re: Big number of messages...

[Reply] [ReplyQuoted] [Headers] [Print]

 

Mon Jul 10 2017 21:10:21 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Big number of messages...
My concern is that, when I go in a folder which contains nearly
200,000 messages, I get the good number at top of screen, but I can
only see around 32000 messages (I guess, because I see only to 19th
of may 2015). And worse, only the olders...

Are you viewing the folder with an IMAP client, or with WebCit?

IMAP does not, to the best of my knowledge, have a fixed limit of how many messages it can present. WebCit does, but if you have the source code online I can point out where to change it.

WebCit-NG will not have any such limits, but that doesn't help you right now :)

 

Hello Ignatius,

bad news... again.

In my Citadel display, under the View As:, I saw a small piece of white line... When I tried to click it, I discovered that it is a page number... For my 78000 messages list, I says 158 pages. But It does not show anything after the 23rd... After this one, only white pages.

My older message is 1/1/2015, and I just see january 2015 messages... 

I wonder if the choice of Berkeley is one of the problem? Do you need to reload all messages to memory at each time? And you are short of memory? I have a quite fast server, 2 Xeon in a Dell R720 with raid disks, and citadel is in a virtual server on it, with 8Gb of RAM, and it takes some time to get the display.

Other point of interest: most of these mails are automated reports coming from customers. Gmail groups them per source/title/day, so for some repeating emails, I see one line for 100 mails. Could you do the same thing with Citadel?

My 70000++ list would become less than 17000...

So, the problems I see now:

- Page selector is hidden because I have: "Make this my startpage", then "Logged in as...", then a search field, then "View As:" which fills the header space, So, the page selector is 90% hidden.

- Mails cannot display more than a few hundreds (or thousands), even if the count is good in the header, and the number of pages is good too.

- (Worse), we always see first the oldest mails, when we should (IMHO) see first the newest.

Point One seems easy to solve by moving a field around.

Point 2 (and 3) seems more tricky, if it is a memory problem, I still can put more memory to my server... But how much? My biggest 'floor' would contain more than 300000 messages.

Point 3? Is it easily solvable? This is a blocking point... I cannot send new reports to Citadel if I cannot see them... The full point for me is to NOT USE client software, since I move(travel) a lot and I cannot install software on most of the computers I use, and I cannot sync 15gb of emails on my phone... Reason why I need to use the web interface...

 

For your information, I am studying the source code now, I need time to know where are the things... I'd like to do an Oracle XE (free Oracle DB) interface. This would help to move some stuffs to the database. But this will take time, if I can do it, because I am very busy and it is a long time since I worked we the C (and the Pro*C Oracle interface).

Anyway, thank you for the interest you have for my problems... When would the NG version come out?

 

Best regards,

Christian COMMARMOND



Go to page: [1] 2 3 4 5 ... Last