Language:
switch to room list switch to menu My folders
Go to page: [1] 2 3 4 5 ... Last
↑↑↑ Old messages ↑↑↑            ↓↓↓ New messages ↓↓↓
[#] Fri Aug 29 2014 17:05:09 EDT from platonov @ Uncensored

Subject: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

My database grew up to huge size (>11 GB).

Is there a way to compress it and recover tons of wasted space?

 



[#] Fri Aug 29 2014 18:03:54 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

Is it the database that has grown, or the log files? You can turn on automatic log file removal, if that's what's causing all of the space consumption.

[#] Sat Aug 30 2014 04:22:12 EDT from platonov @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

 

Fri Aug 29 2014 18:03:54 EDT from IGnatius T Foobar @ Uncensored Subject: Re: How to compress the database?
Is it the database that has grown, or the log files? You can turn on automatic log file removal, if that's what's causing all of the space consumption.

It is the database itself that has grown. One of cdb.* files in particular.

I already turned on automatic log file removal.

 



[#] Sat Aug 30 2014 04:51:20 EDT from platonov @ Uncensored

Subject: Citserver can't start because database got screwed up

[Reply] [ReplyQuoted] [Headers] [Print]

Sorry to bother,

I ran out of disk space and found a lot of log.* files in about 20 log.* files in /usr/local/citadel/data (Ubuntu 14.04).

So, I removed all the log files and I guess it screwed up the database.

I am getting this in syslog:

Aug 30 11:43:56 preciseinfo citserver[3521]: DB: file cdb.00 has LSN 17296/3999708, past end of log at 1/28
Aug 30 11:43:56 preciseinfo citserver[3521]: DB: Commonly caused by moving a database from one database environment
Aug 30 11:43:56 preciseinfo citserver[3521]: DB: to another without clearing the database LSNs, or by removing all of
Aug 30 11:43:56 preciseinfo citserver[3521]: DB: the log files from a database environment
Aug 30 11:43:56 preciseinfo citserver[3521]: DB: the log files from a database environment
Aug 30 11:43:56 preciseinfo citserver[3521]: DB: __db_meta_setup: /usr/local/citadel/data/cdb.00: unexpected file type or format
Aug 30 11:43:56 preciseinfo citserver[3521]: DB: __db_meta_setup: /usr/local/citadel/data/cdb.00: unexpected file type or format
Aug 30 11:43:56 preciseinfo citserver[3521]: db_open[00]: Invalid argument
Aug 30 11:43:56 preciseinfo citserver[3521]: exit code 22

How can I recover?

 

 



[#] Mon Sep 01 2014 15:49:21 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: Citserver can't start because database got screwed up

[Reply] [ReplyQuoted] [Headers] [Print]

You can't. Deleting uncommitted log files will absolutely corrupt your database beyond repair. The automatic log file removal built in to Citadel is designed to only remove the ones that are fully committed and no longer needed by the system.

I'm sorry to say that you're going to have to restore from your latest complete backup.

[#] Mon Sep 01 2014 21:51:36 EDT from platonov @ Uncensored

Subject: Re: Citserver can't start because database got screwed up

[Reply] [ReplyQuoted] [Headers] [Print]

 

Mon Sep 01 2014 15:49:21 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Citserver can't start because database got screwed up
You can't. Deleting uncommitted log files will absolutely corrupt your database beyond repair. The automatic log file removal built in to Citadel is designed to only remove the ones that are fully committed and no longer needed by the system.

I'm sorry to say that you're going to have to restore from your latest complete backup.

Yep. I realized that pretty quickly and that is what we have done. But it was a mess in a situation where you run out of disk space.

So, now we have a backup of the database and in that backup we include only the last log.NNN file. Is that a correct assumption?

Also, since restoring from a backup, there appears to be a room with blank name and I tried to do all sorts of things to remove it, but no luck. I am curious about where did it come from and what can I do to clean it up or out.


Another thing is I view all my rooms in mbox format and I like it sorted in the latest date first order. It used to work fine. But now, even if I change that order it only works for as long as I don't leave that room. If I access that room again via Rooms->Room Name, it falls back on oldest day first order. I looked for some setting in Webcit but do not seem to see anything anywhere? How can I fix this?

And the last thing, I notice a lot of "Not found" messages especially when I try to move messages from one room to another. I have a room where I store all the messages I'd like to keep and so I have to move messages around all the time.

There are variations on "Not found" message. Don't recall exactly at the moment, but something like "Object does not exist" or "Invalid object" or something along those lines.

Are there any plans to fix these "Not found" situations?

 



[#] Mon Sep 01 2014 23:14:56 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: Citserver can't start because database got screwed up

[Reply] [ReplyQuoted] [Headers] [Print]

In a Berkeley DB environment (such as Citadel), you can use the db_archive utility to list the names of all log files which are no longer in use.

Man page for db_archive: http://docs.oracle.com/cd/E17275_01/html/api_reference/C/db_archive.html

In an Easy Install system you'll find db_archive in /usr/local/ctdlsupport/bin/db_archive , run it from /usr/local/citadel/data to hit your Citadel database.

Really though, it's easier to just let Citadel take care of this for you.
The only reason we make it an option not to automatically delete logs in the first place is because some people like to write scripts that delete them manually only after a successful backup has been run.

[#] Wed Sep 03 2014 12:33:03 EDT from platonov @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

 

Sat Aug 30 2014 04:22:12 EDT from platonov @ Uncensored Subject: Re: How to compress the database?

 

Fri Aug 29 2014 18:03:54 EDT from IGnatius T Foobar @ Uncensored Subject: Re: How to compress the database?
Is it the database that has grown, or the log files? You can turn on automatic log file removal, if that's what's causing all of the space consumption.

It is the database itself that has grown. One of cdb.* files in particular.

I already turned on automatic log file removal.

Somehow I doubt that I do have > 11 GB of actual data in the database.

Question: will database (cdb.* files) reduce in size if I try to set a shorter expiration interval on messages in some big rooms?

Or the cdb files will still remain the same size?

In that case, is is a good idea to try to dump the database to XML files and then try to restore it to see if database size decreases?



[#] Wed Sep 03 2014 14:41:09 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]


cdb.* files (as with any Berkeley DB files) never shrink. When you delete data from them, space within the files is marked as unused, and can be re-used for new data.

The effect of this is that when you delete data from Citadel (or anything that uses Berkeley DB), the files will not shrink, but new data will not make them grow any more.

[#] Wed Sep 03 2014 17:31:34 EDT from platonov @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

 

Wed Sep 03 2014 14:41:09 EDT from IGnatius T Foobar @ Uncensored Subject: Re: How to compress the database?

cdb.* files (as with any Berkeley DB files) never shrink. When you delete data from them, space within the files is marked as unused, and can be re-used for new data.

The effect of this is that when you delete data from Citadel (or anything that uses Berkeley DB), the files will not shrink, but new data will not make them grow any more.

Thanx IGnatius,

That's what I suspected. I'd like to try to dump the database to XML in that case and then restore it from the, if that is the right idea in your opinion.

I saw a couple of times lots of log files that filled the disk completely. So, I have removed all of them but the last one in sequence.

Not sure what it was, but it looked like some attack or something like that. I think I did already have the automatic removal of log flags set at that time.

If you happen to have the link to the instructions on how to dump/restore the database to XML, would be appreciated. I am really swamped here with all sorts of nasty lil things related to running out of disk space and upgrade of BTSync to 1.4.72.




[#] Thu Sep 04 2014 03:35:54 EDT from dothebart @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

 

Wed Sep 03 2014 17:31:34 EDT from platonov @ Uncensored Subject: Re: How to compress the database?

 

Wed Sep 03 2014 14:41:09 EDT from IGnatius T Foobar @ Uncensored Subject: Re: How to compress the database?

cdb.* files (as with any Berkeley DB files) never shrink. When you delete data from them, space within the files is marked as unused, and can be re-used for new data.

The effect of this is that when you delete data from Citadel (or anything that uses Berkeley DB), the files will not shrink, but new data will not make them grow any more.

Thanx IGnatius,

That's what I suspected. I'd like to try to dump the database to XML in that case and then restore it from the, if that is the right idea in your opinion.

I saw a couple of times lots of log files that filled the disk completely. So, I have removed all of them but the last one in sequence.

Not sure what it was, but it looked like some attack or something like that. I think I did already have the automatic removal of log flags set at that time.

If you happen to have the link to the instructions on how to dump/restore the database to XML, would be appreciated. I am really swamped here with all sorts of nasty lil things related to running out of disk space and upgrade of BTSync to 1.4.72.




you can use

sendcommand MIGR > /tmp/dump.xml

Please note that in git master are some serious performance boosts to the dump/import facility by switching i.e. the base64 de/encoder; you may consider using this.

One problem seems to still be there in the smtp auth; so you may want to revert the upgrade after your dump/import.



[#] Thu Sep 04 2014 08:13:03 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]


If you are dumping and loading on the same host, I would really recommend that you use database_cleanup.sh instead of exporting to XML and then loading back again.

You're going to need plenty of disk space to do this, of course.

database_cleanup.sh basically just dumps your databases to a binary flat file in /tmp and then deletes your db and loads back from that file.

[#] Thu Sep 04 2014 14:24:41 EDT from danielhz @ Uncensored

Subject: Charset tag in the Webcity interface

[Reply] [ReplyQuoted] [Headers] [Print]

Hello all,

I started using Citadel as an organizer for myself a few days ago. However, I miss the tag

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

in the web interface. As I write in Spanish with UTF-8, I change manually the browser default charset to see the non ASCII characters that are necessary  in Spanish. I suppose I could update some HTML files in /usr/share/citadel-webcit/static to include the charset tag, but I am not sure if it is the best way to customize Webcity. Maybe it is a customization option that I do not found yet in the documentation. Is there?

Best,

Daniel Hernández



[#] Thu Sep 04 2014 14:31:22 EDT from wduchene @ Uncensored

Subject: BinkD, fidonet?

[Reply] [ReplyQuoted] [Headers] [Print]

I've been Googling quite a bit, and can't seem to find any solution - but is there a work around, howto, or means of exchanging messages between something like binkd and citadel? I'd like to try out citadel with fidonet and am stopped short at the message conversion.



[#] Thu Sep 04 2014 17:16:49 EDT from dothebart @ Uncensored

Subject: Re: Charset tag in the Webcity interface

[Reply] [ReplyQuoted] [Headers] [Print]

 

Thu Sep 04 2014 14:24:41 EDT from danielhz @ Uncensored Subject: Charset tag in the Webcity interface

Hello all,

I started using Citadel as an organizer for myself a few days ago. However, I miss the tag

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

in the web interface. As I write in Spanish with UTF-8, I change manually the browser default charset to see the non ASCII characters that are necessary  in Spanish. I suppose I could update some HTML files in /usr/share/citadel-webcit/static to include the charset tag, but I am not sure if it is the best way to customize Webcity. Maybe it is a customization option that I do not found yet in the documentation. Is there?

Best,

Daniel Hernández



I can see your umlauts over here without problems ;-)

Seriously, the html thing is not the only place to look for encoding information.

Webcit puts it into the HTTP headers:

 

 
Content-Type  : text/html; charset=UTF-8

 



[#] Thu Sep 04 2014 17:33:05 EDT from dothebart @ Uncensored

Subject: Re: BinkD, fidonet?

[Reply] [ReplyQuoted] [Headers] [Print]

 

Thu Sep 04 2014 14:31:22 EDT from wduchene @ Uncensored Subject: BinkD, fidonet?

I've been Googling quite a bit, and can't seem to find any solution - but is there a work around, howto, or means of exchanging messages between something like binkd and citadel? I'd like to try out citadel with fidonet and am stopped short at the message conversion.



the probably easiest way would be to inject files into the spool directory of citadel. therefore you have to have one file per message, and use a binary format.

(if not an imap bridge is available on the fidonet side)



[#] Thu Sep 04 2014 22:08:40 EDT from platonov @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

 

Thu Sep 04 2014 08:13:03 EDT from IGnatius T Foobar @ Uncensored Subject: Re: How to compress the database?

If you are dumping and loading on the same host, I would really recommend that you use database_cleanup.sh instead of exporting to XML and then loading back again.

You're going to need plenty of disk space to do this, of course.

database_cleanup.sh basically just dumps your databases to a binary flat file in /tmp and then deletes your db and loads back from that file.

Can I do it on a running system or should I stop citadel sever?

Thanx

 



[#] Thu Sep 04 2014 22:53:47 EDT from IGnatius T Foobar @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

You definitely have to stop the Citadel server before running database_cleanup.sh

[#] Thu Sep 04 2014 23:00:45 EDT from platonov @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

 

Thu Sep 04 2014 22:53:47 EDT from IGnatius T Foobar @ Uncensored Subject: Re: How to compress the database?
You definitely have to stop the Citadel server before running database_cleanup.sh

Now I know!

I hope I can recover...

It would be great if your message would be displayed as the 1st line when you run database_cleanup.sh.

But, hey, NOTHING is perfect, right?

:)

 



[#] Fri Sep 05 2014 03:59:23 EDT from dothebart @ Uncensored

Subject: Re: How to compress the database?

[Reply] [ReplyQuoted] [Headers] [Print]

 

Thu Sep 04 2014 23:00:45 EDT from platonov @ Uncensored Subject: Re: How to compress the database?

 

Thu Sep 04 2014 22:53:47 EDT from IGnatius T Foobar @ Uncensored Subject: Re: How to compress the database?
You definitely have to stop the Citadel server before running database_cleanup.sh

Now I know!

I hope I can recover...

It would be great if your message would be displayed as the 1st line when you run database_cleanup.sh.

But, hey, NOTHING is perfect, right?

:)

its a very good side note. it should issue sendcommand down.



Go to page: [1] 2 3 4 5 ... Last