Language:
switch to room list switch to menu My folders
Go to page: First ... 15 16 17 18 [19] 20 21 22 23 ... Last
[#] Thu Oct 22 2020 16:56:40 EDT from platonov

Subject: Re: Ctdlmigrate "Slacker Method"

[Reply] [ReplyQuoted] [Headers] [Print]

I actually tried this method and it worked out pretty nicely. So, now I have my records "ported" from cit 8.24 to 9.29.

Thanx to warbaby for the info.

Thu Oct 22 2020 13:15:14 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate "Slacker Method"

By "Slacker Method" I mean:

Setting up two imap accounts., such as in Thunderbird or another email client.

[me@old-server.org]

  INBOX

  DRAFTS

  TRASH

 ... etc..

[me@new-server.net]

   INBOX

   DRAFTS

  TRASH

  etc..

[a very minimal example, supposed to look vaguely like account folders in Thunderbird..]

1) Select, highlight all messages in the preview pane of the old server.. [Edit..Select...All]

2) Drag all messages to the new target folder.  

Well, you can even drag the entire rooms and drop them on the same floor in destination and it will transfer all the messages.

[You might also consider doing this with "Local Folders" bypassing the new box at all. but then you'll have to copy them back.. .]

I've done this successfully many times.  You can also put out instructions to have your users migrate their own.

Afterward, just make the dns change and "new-server" becomes "orig-server"] ..

I completely understanding this is going to be case dependent and won't work for everybody.. BUT .. it does actually work. 

Pretty easy to scrape old room/list archives this way..if you are patient, and methodical.

Also consider the use of Add-ons like 'Remove Duplicate Messages".. that might be helpful for some of you.

Which program is this add-on for? Email client?

I use that one quite a bit.

Also be aware! MESSAGES may be LOST if the POWER GOES OUT while copying/moving!

Or due to a network outage.. so use with caution/intelligence.

 

Thu Oct 22 2020 12:27:06 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate

I'll insert my stuff in-place of your text.

Wed Oct 21 2020 17:28:50 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate

I would put the effort into your source database files first..

A). Just because you can't set c_db_shrink_files locally, doesn't mean BerkleyDB compact is out of reach.

The DB->compact() method can be reproduced with a fairly trivial c program. I am looking at some other utility packages to see if someone already has written it..

That would be great. I'd LOVE to run something to reduce database size by factor of 10000.
I happen to have citadel 8.24 source tree and was thinking of may be just adding the DB->compact call after each purge, even without using some test variable.
Can you tell me what do I need to do to build citadel?

B). "slacker method".. Drag/drop messages over IMAP. May not be terrible if both boxes are local to each other. I have done many systems this way.. admittedly nothing like 50G!

I am not quite following this "Drag/drop messages over IMAP". What exactly do I have to do? My 2 boxes are on the same lan, but it is at virtual server provider facilities.

C).  You may have the 'migrate' utility with your original version.  That's an XML exporter.  They at least you'd have a parse-able archive.

Unfortunately, I don't seem to have the 'migrate' utility, at least in /usr/local/ctdlsupport or /usr/local/citadel dirs. But I vaguely recall something about the XML export/import functionality of cit 824. Do you happen to know where I can find some info on it, if there is any? Looks like old docs, where it is described, are gone, and the new ones do not seem to mention it.

Wed Oct 21 2020 04:15:10 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

First of all, ctdlmigrate better have ssh port configurable. Not many run ssh on port 22 nowadays.

I tried to do db_recover on the database produced by Ctdlmigrate  and these are the results.

(as root)
cd /usr/local/citadel/data/
/usr/local/ctdlsupport/bin/db_recover -c

db_recover: BDB0110 Log sequence error: page LSN 1 4377; previous LSN 1 6649
db_recover: BDB1520 Recovery function for LSN 1 6761 failed on forward pass
db_recover: BDB0061 PANIC: Invalid argument
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 24
db_recover: BDB3027 cdb.02: unable to flush page: 24
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 25
db_recover: BDB3027 cdb.02: unable to flush page: 25
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 26
db_recover: BDB3027 cdb.02: unable to flush page: 26
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.03: write failed for page 2
db_recover: BDB3027 cdb.03: unable to flush page: 2
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.03: write failed for page 3
db_recover: BDB3027 cdb.03: unable to flush page: 3
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.0d: write failed for page 1
db_recover: BDB3027 cdb.0d: unable to flush page: 1
db_recover: BDB1544 process-private: unable to find environment
db_recover: DB_ENV->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

I hope they tell you something. So, what's the verdict? Is there any hope for recovering this baby?

Wed Oct 21 2020 15:42:41 EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

First of all, thanx for your help. Appreciated indeed.

I'll plug in some points inline of your text.

Wed Oct 21 2020 15:17:00 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

Just continue on this point ..

/usr/local/citadel/files is file attachments, I'm fairly certain, for those rooms marked as file upload rooms.  I have occasionally seen files in mine.  Whether a file is stored internally (such as an imap attachment) or put in this directory depends on things I'm too lazy to look up right now.. but I'm inferring, direct uploads through webcit to a 'file upload room', is the main criteria.

  • I have nothing in /usr/local/citadel/files

/usr/local/citadel/messages has nothing to do with actual messages.. it's probably not even used anymore.  Mine has nothing in it. It was probably more like what is in text client messages ^nodename login/out templates. It's listed as ^bbsdir .. even so, trivially easy to copy even if permissions need to be modified..

/usr/local/citadel/keys < - keys obviously.. and keys are usually very tightly clamped down.. But also, often particular to a specific host.  In my mind, keys don't specifically need to be copied, they might be, or they might be recreated on the new system.  ctlmigrate is just trying to make it easy for you.  But if it did work, it might not actually be correct, because keys can overlap hostnames, and the new system might not have all the same domains, subdomains as the old, so that is kind of moot anyway.  I think the goal is to move all your email messages, and accounts, not to predict every situation that anyone could ever be in.

  • I copied my original keys on the top of those produced by Ctdlmigrate

If I recall correctly, there was something of a catch 22 with citadel migrate..  You needed to be root, but couldn't be root in order to do one particular thing (sorry, it's been a few years since I've used it..)

  • Yes, I can tell you exactly what needs to be done in order to make ssh work:
  • Edit /etc/ssh/sshd_config
    Change:

    Port NN to:
    Port 22

    #PermitRootLogin no
    PermitRootLogin yes

    # Change to no to disable tunnelled clear text passwords
    PasswordAuthentication yes
    #PasswordAuthentication no

    Then restart sshd
    #sudo systemctl restart sshd
    sudo service ssh restart
Therefore, I'd say, that your migration was successful.   Sometime in the future we might add a little logic around "keys" or some verbiage about permissions..

"Your migration was successful but don't forget you must recreate your keys!"

  • Yep, it does look to me that I got all the data I need. But how come the cit server does not start with those "scheduled jobs" etc.?

or just bail out early on..

"You have to change permission on blah first, please run chmod ..."

  • To change permissions on what files?

or just dumb it down. "don't forget to examine the directories /files and /messages and /keys you may need to move those by hand"..

Sounds kind of odd for a "migration utility" to  become dumber, but may be the best choice considering many unknown factors out there.

Maybe the first step should be to chmod/chown all the files on the source system temporarily..

  • Chmod/chown to what values and for what purpose?

IN ANY CASE YOUR MIGRATION PROBABLY WORKED!

  • Huraaay!!! It worked so well that I just wanna cry like a baby!!!

   rv += create_dir(ctdl_message_dir   , S_IRUSR|S_IWUSR|S_IXUSR, UID, -1);

   help_subst(buffer, "^bbsdir", ctdl_message_dir);

  •  Well, if those instructions make you happy, then there is a hope I'll get it working one of these days, I hope not in too distant of a future...
  • Thanx again
Wed Oct 21 2020 02:38:53 PM EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

Because if rsync fails, then you'll get that message.

But it's right at the very end of the process..

So, you'll have to interpret the meaning.  [It would be good to have ctdlmigrate actually echo out the full commands..but even without that.. it's obvious from "receiving incremental file list"]

check messages/

check keys/

You should have your answer right there.

ctdlmigrate.c 247-275

 

      if (!strncasecmp(buf, "files|", 6)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[6], ctdl_file_dir);
      }
      else if (!strncasecmp(buf, "messages|", 9)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[9], ctdl_message_dir);
      }
      else if (!strncasecmp(buf, "keys|", 5)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[5], ctdl_key_dir);
      }
      else {
         strcpy(cmd, "false");   /* cheap and sleazy way to throw an error */
      }
      printf("%s\n", cmd);
      cmdexit = system(cmd);
      if (cmdexit != 0) {
         exitcode += cmdexit;
      }
   }
   pclose(sourcefp);

THEEND:  if (exitcode == 0) {
      printf("\n\n *** Citadel migration was successful! *** \n\n");
   }
   else {
      printf("\n\n *** Citadel migration was unsuccessful. *** \n\n");
   }


Wed Oct 21 2020 01:52:26 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

I am using letsencrypt and I do have a script that runs "sudo certbot renew" followed by mv-certs-to-citadel.sh below.

But I am not sure I follow why ctdlmigrate does not complete OK just because of a couple of key files.

Wed Oct 21 2020 12:20:43 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

If you want to break this out into another thread about Citadel Keys, if it is in fact the issue, it may not be a bad idea..

What I'm seeing is likely rsync failing on the two key files citadel.key and citadel.csr They are chmod 600 owned by root:staff or citadel:root

Your migration may be successful except for that.  Are you using letsencrypt?  you can just copy the files from live/ .. (can't link, wrong owner permissions)

If you're not using let's encrypt, try to recreate your self-signed keys, check the contents of everything in /usr/local/citadel/keys and (same on webcit/keys. for webcit I just use a link to citadel/keys)..

Also, if using letsencrypt, use fullchain.pem .. not sure if its still an issue but, there some browsers were throwing errors  if you didn't include the whole chain.

# mv-certs-to-citadel.sh

#!/bin/bash
if [[ $# -eq 0 ]] ; then
    echo 'I need a domain related to a cert in /etc/letsencrypt/live';
    echo "Like one of these";
    echo "`ls -alh /etc/letsencrypt/live`";
    exit 0
   fi

DOMAIN=$1;
CIT_CERT_DIR="/usr/local/citadel/keys"; # Citadel Server
WEB_CERT_DIR="/usr/local/webcit/keys"; # Webcit
cp "/etc/letsencrypt/live/$DOMAIN/fullchain.pem" "$CIT_CERT_DIR/citadel.cer"
cp "/etc/letsencrypt/live/$DOMAIN/privkey.pem" "$CIT_CERT_DIR/citadel.key"
cp "/etc/letsencrypt/live/$DOMAIN/fullchain.pem" "$WEB_CERT_DIR/citadel.cer"
cp "/etc/letsencrypt/live/$DOMAIN/privkey.pem" "$WEB_CERT_DIR/citadel.key"
chown -R citadel:root "$CIT_CERT_DIR"
chown -R citadel:root "$WEB_CERT_DIR"

Wed Oct 21 2020 08:41:59 AM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database from 8.24 to 9.29 node.

I was able to make it work and it did xfer the data to the new node. And, Subject: headers were magically working correctly. I did verify quite a bit of records and everything looked just fine. Does it mean it is a configuration issue?

Unfortunately, ctdlmigrate did not fully succeed and it bombed out with an error msg:

*** Citadel migration was unsuccessful. ***

Here are last few lines.

- - - - - - - - - - - - - - - - -

root@45.79.97.253:/usr/local/citadel/keys/ /usr/local/citadel/keys/
receiving incremental file list

sent 20 bytes  received 85 bytes  210.00 bytes/sec
total size is 5,286  speedup is 50.34
false
false

 *** Citadel migration was unsuccessful. ***

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

And the real bummer is that everything seemed to work just fine with imported data until you restart the citserver.

At that point, here's the status:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

$ sudo service citadel status
● citadel.service - Citadel Server
     Loaded: loaded (/etc/systemd/system/citadel.service; enabled; vendor preset                                                                                                                                   : enabled)
     Active: failed (Result: exit-code) since Tue 2020-10-20 20:42:41 UTC; 2s ag                                                                                                                                   o
    Process: 1507 ExecStart=/usr/local/citadel/citserver (code=exited, status=10                                                                                                                                   5)
   Main PID: 1507 (code=exited, status=105)

Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Scheduled restart job, resta                                                                                                                                   rt counter is at 5.
Oct 20 20:42:41 UbuToo systemd[1]: Stopped Citadel Server.
Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Start request repeated too q                                                                                                                                   uickly.
Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Failed with result 'exit-cod                                                                                                                                   e'.
Oct 20 20:42:41 UbuToo systemd[1]: Failed to start Citadel Server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

What can be done to fix this issue? I was thinking of doing a database recover kind of thing.

Tue Oct 20 2020 23:12:29 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
What I am seeing on 9.17 is incorrect displaying or not displaying at all
of
Subject: header in RSS feed rooms.

I have noticed this on occasion as well. It's got to be a buffer size issue somewhere but so far I haven't been able to figure out where it is. We'll keep looking.

 



 



 



 



 



 



 



 



 



 



 



[#] Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

I apologize if this isn't clear in the documentation.

[#] Thu Oct 22 2020 20:09:01 EDT from warbaby

Subject: Re: Ctdlmigrate "Slacker Method"

[Reply] [ReplyQuoted] [Headers] [Print]

I'm really glad to hear it!

Here's the Remove Duplicate Add-on for Thunderbird ..(I'm still using one of the older versions so, should be okay.. )

Also, I usually create the Local Folders

* Sent

* Archives

* Drafts

* Templates (not that I use them, but whatever)

and under 'Account' ... "Settings' .. 'Copies & Folders'

Select those instead of saving everything in Citadel.  Saves a little time/network overhead..

Also, the "Search Button Add-on" .. make 1000% times difference being able to actually find an email..

 

Thu Oct 22 2020 04:56:40 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate "Slacker Method"

I actually tried this method and it worked out pretty nicely. So, now I have my records "ported" from cit 8.24 to 9.29.

Thanx to warbaby for the info.

Thu Oct 22 2020 13:15:14 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate "Slacker Method"

By "Slacker Method" I mean:

Setting up two imap accounts., such as in Thunderbird or another email client.

[me@old-server.org]

  INBOX

  DRAFTS

  TRASH

 ... etc..

[me@new-server.net]

   INBOX

   DRAFTS

  TRASH

  etc..

[a very minimal example, supposed to look vaguely like account folders in Thunderbird..]

1) Select, highlight all messages in the preview pane of the old server.. [Edit..Select...All]

2) Drag all messages to the new target folder.  

Well, you can even drag the entire rooms and drop them on the same floor in destination and it will transfer all the messages.

[You might also consider doing this with "Local Folders" bypassing the new box at all. but then you'll have to copy them back.. .]

I've done this successfully many times.  You can also put out instructions to have your users migrate their own.

Afterward, just make the dns change and "new-server" becomes "orig-server"] ..

I completely understanding this is going to be case dependent and won't work for everybody.. BUT .. it does actually work. 

Pretty easy to scrape old room/list archives this way..if you are patient, and methodical.

Also consider the use of Add-ons like 'Remove Duplicate Messages".. that might be helpful for some of you.

Which program is this add-on for? Email client?

I use that one quite a bit.

Also be aware! MESSAGES may be LOST if the POWER GOES OUT while copying/moving!

Or due to a network outage.. so use with caution/intelligence.

 

Thu Oct 22 2020 12:27:06 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate

I'll insert my stuff in-place of your text.

Wed Oct 21 2020 17:28:50 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate

I would put the effort into your source database files first..

A). Just because you can't set c_db_shrink_files locally, doesn't mean BerkleyDB compact is out of reach.

The DB->compact() method can be reproduced with a fairly trivial c program. I am looking at some other utility packages to see if someone already has written it..

That would be great. I'd LOVE to run something to reduce database size by factor of 10000.
I happen to have citadel 8.24 source tree and was thinking of may be just adding the DB->compact call after each purge, even without using some test variable.
Can you tell me what do I need to do to build citadel?

B). "slacker method".. Drag/drop messages over IMAP. May not be terrible if both boxes are local to each other. I have done many systems this way.. admittedly nothing like 50G!

I am not quite following this "Drag/drop messages over IMAP". What exactly do I have to do? My 2 boxes are on the same lan, but it is at virtual server provider facilities.

C).  You may have the 'migrate' utility with your original version.  That's an XML exporter.  They at least you'd have a parse-able archive.

Unfortunately, I don't seem to have the 'migrate' utility, at least in /usr/local/ctdlsupport or /usr/local/citadel dirs. But I vaguely recall something about the XML export/import functionality of cit 824. Do you happen to know where I can find some info on it, if there is any? Looks like old docs, where it is described, are gone, and the new ones do not seem to mention it.

Wed Oct 21 2020 04:15:10 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

First of all, ctdlmigrate better have ssh port configurable. Not many run ssh on port 22 nowadays.

I tried to do db_recover on the database produced by Ctdlmigrate  and these are the results.

(as root)
cd /usr/local/citadel/data/
/usr/local/ctdlsupport/bin/db_recover -c

db_recover: BDB0110 Log sequence error: page LSN 1 4377; previous LSN 1 6649
db_recover: BDB1520 Recovery function for LSN 1 6761 failed on forward pass
db_recover: BDB0061 PANIC: Invalid argument
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 24
db_recover: BDB3027 cdb.02: unable to flush page: 24
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 25
db_recover: BDB3027 cdb.02: unable to flush page: 25
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 26
db_recover: BDB3027 cdb.02: unable to flush page: 26
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.03: write failed for page 2
db_recover: BDB3027 cdb.03: unable to flush page: 2
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.03: write failed for page 3
db_recover: BDB3027 cdb.03: unable to flush page: 3
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.0d: write failed for page 1
db_recover: BDB3027 cdb.0d: unable to flush page: 1
db_recover: BDB1544 process-private: unable to find environment
db_recover: DB_ENV->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

I hope they tell you something. So, what's the verdict? Is there any hope for recovering this baby?

Wed Oct 21 2020 15:42:41 EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

First of all, thanx for your help. Appreciated indeed.

I'll plug in some points inline of your text.

Wed Oct 21 2020 15:17:00 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

Just continue on this point ..

/usr/local/citadel/files is file attachments, I'm fairly certain, for those rooms marked as file upload rooms.  I have occasionally seen files in mine.  Whether a file is stored internally (such as an imap attachment) or put in this directory depends on things I'm too lazy to look up right now.. but I'm inferring, direct uploads through webcit to a 'file upload room', is the main criteria.

  • I have nothing in /usr/local/citadel/files

/usr/local/citadel/messages has nothing to do with actual messages.. it's probably not even used anymore.  Mine has nothing in it. It was probably more like what is in text client messages ^nodename login/out templates. It's listed as ^bbsdir .. even so, trivially easy to copy even if permissions need to be modified..

/usr/local/citadel/keys < - keys obviously.. and keys are usually very tightly clamped down.. But also, often particular to a specific host.  In my mind, keys don't specifically need to be copied, they might be, or they might be recreated on the new system.  ctlmigrate is just trying to make it easy for you.  But if it did work, it might not actually be correct, because keys can overlap hostnames, and the new system might not have all the same domains, subdomains as the old, so that is kind of moot anyway.  I think the goal is to move all your email messages, and accounts, not to predict every situation that anyone could ever be in.

  • I copied my original keys on the top of those produced by Ctdlmigrate

If I recall correctly, there was something of a catch 22 with citadel migrate..  You needed to be root, but couldn't be root in order to do one particular thing (sorry, it's been a few years since I've used it..)

  • Yes, I can tell you exactly what needs to be done in order to make ssh work:
  • Edit /etc/ssh/sshd_config
    Change:

    Port NN to:
    Port 22

    #PermitRootLogin no
    PermitRootLogin yes

    # Change to no to disable tunnelled clear text passwords
    PasswordAuthentication yes
    #PasswordAuthentication no

    Then restart sshd
    #sudo systemctl restart sshd
    sudo service ssh restart
Therefore, I'd say, that your migration was successful.   Sometime in the future we might add a little logic around "keys" or some verbiage about permissions..

"Your migration was successful but don't forget you must recreate your keys!"

  • Yep, it does look to me that I got all the data I need. But how come the cit server does not start with those "scheduled jobs" etc.?

or just bail out early on..

"You have to change permission on blah first, please run chmod ..."

  • To change permissions on what files?

or just dumb it down. "don't forget to examine the directories /files and /messages and /keys you may need to move those by hand"..

Sounds kind of odd for a "migration utility" to  become dumber, but may be the best choice considering many unknown factors out there.

Maybe the first step should be to chmod/chown all the files on the source system temporarily..

  • Chmod/chown to what values and for what purpose?

IN ANY CASE YOUR MIGRATION PROBABLY WORKED!

  • Huraaay!!! It worked so well that I just wanna cry like a baby!!!

   rv += create_dir(ctdl_message_dir   , S_IRUSR|S_IWUSR|S_IXUSR, UID, -1);

   help_subst(buffer, "^bbsdir", ctdl_message_dir);

  •  Well, if those instructions make you happy, then there is a hope I'll get it working one of these days, I hope not in too distant of a future...
  • Thanx again
Wed Oct 21 2020 02:38:53 PM EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

Because if rsync fails, then you'll get that message.

But it's right at the very end of the process..

So, you'll have to interpret the meaning.  [It would be good to have ctdlmigrate actually echo out the full commands..but even without that.. it's obvious from "receiving incremental file list"]

check messages/

check keys/

You should have your answer right there.

ctdlmigrate.c 247-275

 

      if (!strncasecmp(buf, "files|", 6)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[6], ctdl_file_dir);
      }
      else if (!strncasecmp(buf, "messages|", 9)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[9], ctdl_message_dir);
      }
      else if (!strncasecmp(buf, "keys|", 5)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[5], ctdl_key_dir);
      }
      else {
         strcpy(cmd, "false");   /* cheap and sleazy way to throw an error */
      }
      printf("%s\n", cmd);
      cmdexit = system(cmd);
      if (cmdexit != 0) {
         exitcode += cmdexit;
      }
   }
   pclose(sourcefp);

THEEND:  if (exitcode == 0) {
      printf("\n\n *** Citadel migration was successful! *** \n\n");
   }
   else {
      printf("\n\n *** Citadel migration was unsuccessful. *** \n\n");
   }


Wed Oct 21 2020 01:52:26 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

I am using letsencrypt and I do have a script that runs "sudo certbot renew" followed by mv-certs-to-citadel.sh below.

But I am not sure I follow why ctdlmigrate does not complete OK just because of a couple of key files.

Wed Oct 21 2020 12:20:43 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

If you want to break this out into another thread about Citadel Keys, if it is in fact the issue, it may not be a bad idea..

What I'm seeing is likely rsync failing on the two key files citadel.key and citadel.csr They are chmod 600 owned by root:staff or citadel:root

Your migration may be successful except for that.  Are you using letsencrypt?  you can just copy the files from live/ .. (can't link, wrong owner permissions)

If you're not using let's encrypt, try to recreate your self-signed keys, check the contents of everything in /usr/local/citadel/keys and (same on webcit/keys. for webcit I just use a link to citadel/keys)..

Also, if using letsencrypt, use fullchain.pem .. not sure if its still an issue but, there some browsers were throwing errors  if you didn't include the whole chain.

# mv-certs-to-citadel.sh

#!/bin/bash
if [[ $# -eq 0 ]] ; then
    echo 'I need a domain related to a cert in /etc/letsencrypt/live';
    echo "Like one of these";
    echo "`ls -alh /etc/letsencrypt/live`";
    exit 0
   fi

DOMAIN=$1;
CIT_CERT_DIR="/usr/local/citadel/keys"; # Citadel Server
WEB_CERT_DIR="/usr/local/webcit/keys"; # Webcit
cp "/etc/letsencrypt/live/$DOMAIN/fullchain.pem" "$CIT_CERT_DIR/citadel.cer"
cp "/etc/letsencrypt/live/$DOMAIN/privkey.pem" "$CIT_CERT_DIR/citadel.key"
cp "/etc/letsencrypt/live/$DOMAIN/fullchain.pem" "$WEB_CERT_DIR/citadel.cer"
cp "/etc/letsencrypt/live/$DOMAIN/privkey.pem" "$WEB_CERT_DIR/citadel.key"
chown -R citadel:root "$CIT_CERT_DIR"
chown -R citadel:root "$WEB_CERT_DIR"

Wed Oct 21 2020 08:41:59 AM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database from 8.24 to 9.29 node.

I was able to make it work and it did xfer the data to the new node. And, Subject: headers were magically working correctly. I did verify quite a bit of records and everything looked just fine. Does it mean it is a configuration issue?

Unfortunately, ctdlmigrate did not fully succeed and it bombed out with an error msg:

*** Citadel migration was unsuccessful. ***

Here are last few lines.

- - - - - - - - - - - - - - - - -

root@45.79.97.253:/usr/local/citadel/keys/ /usr/local/citadel/keys/
receiving incremental file list

sent 20 bytes  received 85 bytes  210.00 bytes/sec
total size is 5,286  speedup is 50.34
false
false

 *** Citadel migration was unsuccessful. ***

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

And the real bummer is that everything seemed to work just fine with imported data until you restart the citserver.

At that point, here's the status:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

$ sudo service citadel status
● citadel.service - Citadel Server
     Loaded: loaded (/etc/systemd/system/citadel.service; enabled; vendor preset                                                                                                                                   : enabled)
     Active: failed (Result: exit-code) since Tue 2020-10-20 20:42:41 UTC; 2s ag                                                                                                                                   o
    Process: 1507 ExecStart=/usr/local/citadel/citserver (code=exited, status=10                                                                                                                                   5)
   Main PID: 1507 (code=exited, status=105)

Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Scheduled restart job, resta                                                                                                                                   rt counter is at 5.
Oct 20 20:42:41 UbuToo systemd[1]: Stopped Citadel Server.
Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Start request repeated too q                                                                                                                                   uickly.
Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Failed with result 'exit-cod                                                                                                                                   e'.
Oct 20 20:42:41 UbuToo systemd[1]: Failed to start Citadel Server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

What can be done to fix this issue? I was thinking of doing a database recover kind of thing.

Tue Oct 20 2020 23:12:29 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
What I am seeing on 9.17 is incorrect displaying or not displaying at all
of
Subject: header in RSS feed rooms.

I have noticed this on occasion as well. It's got to be a buffer size issue somewhere but so far I haven't been able to figure out where it is. We'll keep looking.

 



 



 



 



 



 



 



 



 



 



 



 



[#] Fri Oct 23 2020 09:34:55 EDT from platonov

Subject: Re: Ctdlmigrate "Slacker Method"

[Reply] [ReplyQuoted] [Headers] [Print]

 

Thu Oct 22 2020 20:09:01 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate "Slacker Method"

I'm really glad to hear it!

Here's the Remove Duplicate Add-on for Thunderbird ..(I'm still using one of the older versions so, should be okay.. )

Is there a reason you are not using the most popular one: Remove Duplicate Messages?

Also, I usually create the Local Folders

* Sent

* Archives

* Drafts

* Templates (not that I use them, but whatever)

and under 'Account' ... "Settings' .. 'Copies & Folders'

Select those instead of saving everything in Citadel.  Saves a little time/network overhead..

Also, the "Search Button Add-on" .. make 1000% times difference being able to actually find an email..

 

Thu Oct 22 2020 04:56:40 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate "Slacker Method"

I actually tried this method and it worked out pretty nicely. So, now I have my records "ported" from cit 8.24 to 9.29.

Thanx to warbaby for the info.

Thu Oct 22 2020 13:15:14 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate "Slacker Method"

By "Slacker Method" I mean:

Setting up two imap accounts., such as in Thunderbird or another email client.

[me@old-server.org]

  INBOX

  DRAFTS

  TRASH

 ... etc..

[me@new-server.net]

   INBOX

   DRAFTS

  TRASH

  etc..

[a very minimal example, supposed to look vaguely like account folders in Thunderbird..]

1) Select, highlight all messages in the preview pane of the old server.. [Edit..Select...All]

2) Drag all messages to the new target folder.  

Well, you can even drag the entire rooms and drop them on the same floor in destination and it will transfer all the messages.

[You might also consider doing this with "Local Folders" bypassing the new box at all. but then you'll have to copy them back.. .]

I've done this successfully many times.  You can also put out instructions to have your users migrate their own.

Afterward, just make the dns change and "new-server" becomes "orig-server"] ..

I completely understanding this is going to be case dependent and won't work for everybody.. BUT .. it does actually work. 

Pretty easy to scrape old room/list archives this way..if you are patient, and methodical.

Also consider the use of Add-ons like 'Remove Duplicate Messages".. that might be helpful for some of you.

Which program is this add-on for? Email client?

I use that one quite a bit.

Also be aware! MESSAGES may be LOST if the POWER GOES OUT while copying/moving!

Or due to a network outage.. so use with caution/intelligence.

 

Thu Oct 22 2020 12:27:06 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate

I'll insert my stuff in-place of your text.

Wed Oct 21 2020 17:28:50 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate

I would put the effort into your source database files first..

A). Just because you can't set c_db_shrink_files locally, doesn't mean BerkleyDB compact is out of reach.

The DB->compact() method can be reproduced with a fairly trivial c program. I am looking at some other utility packages to see if someone already has written it..

That would be great. I'd LOVE to run something to reduce database size by factor of 10000.
I happen to have citadel 8.24 source tree and was thinking of may be just adding the DB->compact call after each purge, even without using some test variable.
Can you tell me what do I need to do to build citadel?

B). "slacker method".. Drag/drop messages over IMAP. May not be terrible if both boxes are local to each other. I have done many systems this way.. admittedly nothing like 50G!

I am not quite following this "Drag/drop messages over IMAP". What exactly do I have to do? My 2 boxes are on the same lan, but it is at virtual server provider facilities.

C).  You may have the 'migrate' utility with your original version.  That's an XML exporter.  They at least you'd have a parse-able archive.

Unfortunately, I don't seem to have the 'migrate' utility, at least in /usr/local/ctdlsupport or /usr/local/citadel dirs. But I vaguely recall something about the XML export/import functionality of cit 824. Do you happen to know where I can find some info on it, if there is any? Looks like old docs, where it is described, are gone, and the new ones do not seem to mention it.

Wed Oct 21 2020 04:15:10 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

First of all, ctdlmigrate better have ssh port configurable. Not many run ssh on port 22 nowadays.

I tried to do db_recover on the database produced by Ctdlmigrate  and these are the results.

(as root)
cd /usr/local/citadel/data/
/usr/local/ctdlsupport/bin/db_recover -c

db_recover: BDB0110 Log sequence error: page LSN 1 4377; previous LSN 1 6649
db_recover: BDB1520 Recovery function for LSN 1 6761 failed on forward pass
db_recover: BDB0061 PANIC: Invalid argument
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 24
db_recover: BDB3027 cdb.02: unable to flush page: 24
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 25
db_recover: BDB3027 cdb.02: unable to flush page: 25
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.02: write failed for page 26
db_recover: BDB3027 cdb.02: unable to flush page: 26
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.03: write failed for page 2
db_recover: BDB3027 cdb.03: unable to flush page: 2
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.03: write failed for page 3
db_recover: BDB3027 cdb.03: unable to flush page: 3
db_recover: BDB0060 PANIC: fatal region error detected; run recovery
db_recover: BDB3015 cdb.0d: write failed for page 1
db_recover: BDB3027 cdb.0d: unable to flush page: 1
db_recover: BDB1544 process-private: unable to find environment
db_recover: DB_ENV->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

I hope they tell you something. So, what's the verdict? Is there any hope for recovering this baby?

Wed Oct 21 2020 15:42:41 EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

First of all, thanx for your help. Appreciated indeed.

I'll plug in some points inline of your text.

Wed Oct 21 2020 15:17:00 EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

Just continue on this point ..

/usr/local/citadel/files is file attachments, I'm fairly certain, for those rooms marked as file upload rooms.  I have occasionally seen files in mine.  Whether a file is stored internally (such as an imap attachment) or put in this directory depends on things I'm too lazy to look up right now.. but I'm inferring, direct uploads through webcit to a 'file upload room', is the main criteria.

  • I have nothing in /usr/local/citadel/files

/usr/local/citadel/messages has nothing to do with actual messages.. it's probably not even used anymore.  Mine has nothing in it. It was probably more like what is in text client messages ^nodename login/out templates. It's listed as ^bbsdir .. even so, trivially easy to copy even if permissions need to be modified..

/usr/local/citadel/keys < - keys obviously.. and keys are usually very tightly clamped down.. But also, often particular to a specific host.  In my mind, keys don't specifically need to be copied, they might be, or they might be recreated on the new system.  ctlmigrate is just trying to make it easy for you.  But if it did work, it might not actually be correct, because keys can overlap hostnames, and the new system might not have all the same domains, subdomains as the old, so that is kind of moot anyway.  I think the goal is to move all your email messages, and accounts, not to predict every situation that anyone could ever be in.

  • I copied my original keys on the top of those produced by Ctdlmigrate

If I recall correctly, there was something of a catch 22 with citadel migrate..  You needed to be root, but couldn't be root in order to do one particular thing (sorry, it's been a few years since I've used it..)

  • Yes, I can tell you exactly what needs to be done in order to make ssh work:
  • Edit /etc/ssh/sshd_config
    Change:

    Port NN to:
    Port 22

    #PermitRootLogin no
    PermitRootLogin yes

    # Change to no to disable tunnelled clear text passwords
    PasswordAuthentication yes
    #PasswordAuthentication no

    Then restart sshd
    #sudo systemctl restart sshd
    sudo service ssh restart
Therefore, I'd say, that your migration was successful.   Sometime in the future we might add a little logic around "keys" or some verbiage about permissions..

"Your migration was successful but don't forget you must recreate your keys!"

  • Yep, it does look to me that I got all the data I need. But how come the cit server does not start with those "scheduled jobs" etc.?

or just bail out early on..

"You have to change permission on blah first, please run chmod ..."

  • To change permissions on what files?

or just dumb it down. "don't forget to examine the directories /files and /messages and /keys you may need to move those by hand"..

Sounds kind of odd for a "migration utility" to  become dumber, but may be the best choice considering many unknown factors out there.

Maybe the first step should be to chmod/chown all the files on the source system temporarily..

  • Chmod/chown to what values and for what purpose?

IN ANY CASE YOUR MIGRATION PROBABLY WORKED!

  • Huraaay!!! It worked so well that I just wanna cry like a baby!!!

   rv += create_dir(ctdl_message_dir   , S_IRUSR|S_IWUSR|S_IXUSR, UID, -1);

   help_subst(buffer, "^bbsdir", ctdl_message_dir);

  •  Well, if those instructions make you happy, then there is a hope I'll get it working one of these days, I hope not in too distant of a future...
  • Thanx again
Wed Oct 21 2020 02:38:53 PM EDT from warbaby @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

Because if rsync fails, then you'll get that message.

But it's right at the very end of the process..

So, you'll have to interpret the meaning.  [It would be good to have ctdlmigrate actually echo out the full commands..but even without that.. it's obvious from "receiving incremental file list"]

check messages/

check keys/

You should have your answer right there.

ctdlmigrate.c 247-275

 

      if (!strncasecmp(buf, "files|", 6)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[6], ctdl_file_dir);
      }
      else if (!strncasecmp(buf, "messages|", 9)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[9], ctdl_message_dir);
      }
      else if (!strncasecmp(buf, "keys|", 5)) {
         snprintf(cmd, sizeof cmd, "rsync -va --rsh='ssh -S %s' %s@%s:%s/ %s/",
            socket_path, remote_user, remote_host, &buf[5], ctdl_key_dir);
      }
      else {
         strcpy(cmd, "false");   /* cheap and sleazy way to throw an error */
      }
      printf("%s\n", cmd);
      cmdexit = system(cmd);
      if (cmdexit != 0) {
         exitcode += cmdexit;
      }
   }
   pclose(sourcefp);

THEEND:  if (exitcode == 0) {
      printf("\n\n *** Citadel migration was successful! *** \n\n");
   }
   else {
      printf("\n\n *** Citadel migration was unsuccessful. *** \n\n");
   }


Wed Oct 21 2020 01:52:26 PM EDT from platonov @ Uncensored Subject: Re: Ctdlmigrate and Citadel Key files

I am using letsencrypt and I do have a script that runs "sudo certbot renew" followed by mv-certs-to-citadel.sh below.

But I am not sure I follow why ctdlmigrate does not complete OK just because of a couple of key files.

Wed Oct 21 2020 12:20:43 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

If you want to break this out into another thread about Citadel Keys, if it is in fact the issue, it may not be a bad idea..

What I'm seeing is likely rsync failing on the two key files citadel.key and citadel.csr They are chmod 600 owned by root:staff or citadel:root

Your migration may be successful except for that.  Are you using letsencrypt?  you can just copy the files from live/ .. (can't link, wrong owner permissions)

If you're not using let's encrypt, try to recreate your self-signed keys, check the contents of everything in /usr/local/citadel/keys and (same on webcit/keys. for webcit I just use a link to citadel/keys)..

Also, if using letsencrypt, use fullchain.pem .. not sure if its still an issue but, there some browsers were throwing errors  if you didn't include the whole chain.

# mv-certs-to-citadel.sh

#!/bin/bash
if [[ $# -eq 0 ]] ; then
    echo 'I need a domain related to a cert in /etc/letsencrypt/live';
    echo "Like one of these";
    echo "`ls -alh /etc/letsencrypt/live`";
    exit 0
   fi

DOMAIN=$1;
CIT_CERT_DIR="/usr/local/citadel/keys"; # Citadel Server
WEB_CERT_DIR="/usr/local/webcit/keys"; # Webcit
cp "/etc/letsencrypt/live/$DOMAIN/fullchain.pem" "$CIT_CERT_DIR/citadel.cer"
cp "/etc/letsencrypt/live/$DOMAIN/privkey.pem" "$CIT_CERT_DIR/citadel.key"
cp "/etc/letsencrypt/live/$DOMAIN/fullchain.pem" "$WEB_CERT_DIR/citadel.cer"
cp "/etc/letsencrypt/live/$DOMAIN/privkey.pem" "$WEB_CERT_DIR/citadel.key"
chown -R citadel:root "$CIT_CERT_DIR"
chown -R citadel:root "$WEB_CERT_DIR"

Wed Oct 21 2020 08:41:59 AM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database from 8.24 to 9.29 node.

I was able to make it work and it did xfer the data to the new node. And, Subject: headers were magically working correctly. I did verify quite a bit of records and everything looked just fine. Does it mean it is a configuration issue?

Unfortunately, ctdlmigrate did not fully succeed and it bombed out with an error msg:

*** Citadel migration was unsuccessful. ***

Here are last few lines.

- - - - - - - - - - - - - - - - -

root@45.79.97.253:/usr/local/citadel/keys/ /usr/local/citadel/keys/
receiving incremental file list

sent 20 bytes  received 85 bytes  210.00 bytes/sec
total size is 5,286  speedup is 50.34
false
false

 *** Citadel migration was unsuccessful. ***

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

And the real bummer is that everything seemed to work just fine with imported data until you restart the citserver.

At that point, here's the status:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

$ sudo service citadel status
● citadel.service - Citadel Server
     Loaded: loaded (/etc/systemd/system/citadel.service; enabled; vendor preset                                                                                                                                   : enabled)
     Active: failed (Result: exit-code) since Tue 2020-10-20 20:42:41 UTC; 2s ag                                                                                                                                   o
    Process: 1507 ExecStart=/usr/local/citadel/citserver (code=exited, status=10                                                                                                                                   5)
   Main PID: 1507 (code=exited, status=105)

Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Scheduled restart job, resta                                                                                                                                   rt counter is at 5.
Oct 20 20:42:41 UbuToo systemd[1]: Stopped Citadel Server.
Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Start request repeated too q                                                                                                                                   uickly.
Oct 20 20:42:41 UbuToo systemd[1]: citadel.service: Failed with result 'exit-cod                                                                                                                                   e'.
Oct 20 20:42:41 UbuToo systemd[1]: Failed to start Citadel Server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

What can be done to fix this issue? I was thinking of doing a database recover kind of thing.

Tue Oct 20 2020 23:12:29 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
What I am seeing on 9.17 is incorrect displaying or not displaying at all
of
Subject: header in RSS feed rooms.

I have noticed this on occasion as well. It's got to be a buffer size issue somewhere but so far I haven't been able to figure out where it is. We'll keep looking.

 



 



 



 



 



 



 



 



 



 



 



 



 



[#] Fri Oct 23 2020 09:34:19 EDT from Martin Maurer <martin@proxmox.com> to room_Citadel_Support@citadel.org

Subject: unsubscribe

[Reply] [ReplyQuoted] [Headers] [Print]

pls unsubscribe my email address.

I cannot find a way to unsubscribe here, any hint?

Thanks,

Martin

[#] Fri Oct 23 2020 09:53:56 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation.

 



[#] Fri Oct 23 2020 15:10:29 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


[#] Fri Oct 23 2020 15:23:17 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



[#] Fri Oct 23 2020 16:23:52 EDT from warbaby

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
[#] Fri Oct 23 2020 16:25:22 EDT from warbaby

Subject: Re: unsubscribe

[Reply] [ReplyQuoted] [Headers] [Print]

Sorry man, you're here forever...

.. here Try this: http://uncensored.citadel.org/listsub

Fri Oct 23 2020 09:34:19 AM EDT from "Martin Maurer" <martin@proxmox.com> Subject: unsubscribe
pls unsubscribe my email address.

I cannot find a way to unsubscribe here, any hint?

Thanks,

Martin

 



[#] Fri Oct 23 2020 17:28:05 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
[#] Fri Oct 23 2020 17:32:34 EDT from warbaby

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

 I added your feed.  The subject is present.. it's just Mojibake.  Similar to what is happening to my From and Return-Path headers... although mysteriously, my list/header problem has cleared up for now.

I think we've got Gremlin.

see below:

Return-Path: rss
Date: Fri, 23 Oct 2020 20:26:18 -0000
Subject: Спортсмены салютуют Великой Победе. Праздничное шоу состоялось в "Мегаспорте"
Message-ID: 
From: "rss" 
Content-type: text/html
Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



[#] Fri Oct 23 2020 17:42:03 EDT from warbaby

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

No, not really, but am learning. I encourage you to hang in there & keep looking. 

We both really need to fix this and I am committed to finding it.

We need all the eyes we can get...

 

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



 



 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
[#] Sat Oct 24 2020 08:40:11 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

Commenting in-place.

Fri Oct 23 2020 17:32:34 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 I added your feed.  The subject is present.. it's just Mojibake.

I am not following. Are you saying that on your system you have the Subject: headers? That would be pretty wild. Are ALL Subject headers present or just some?

Similar to what is happening to my From and Return-Path headers... although mysteriously, my list/header problem has cleared up for now.

I am not aware of your From and Return-Path headers problem. What was it?

I think we've got Gremlin.

Wooo. That's bad news.

The thing is I originally did a fresh Citadel install (on a new node running Ubuntu 20.04) using easyinstall procedure. So, where did this problem come from in the first place and how come it wasn't showing up when I did ctdlmigrate from 8.24 to 9.29 even though this should not have been done.

This kinda "rings the bell" in terms of my theory regarding some configuration parameter. Btw, I wanted to ask about the place where configuration parms are stored. Is it in the database or some config files? Is it possible that some config parameter is not available via GUI, but can be configured by editing some file or doing a call to citserver?

This is getting bewildering!

see below:

Return-Path: rss
Date: Fri, 23 Oct 2020 20:26:18 -0000
Subject: Спортсмены салютуют Великой Победе. Праздничное шоу состоялось в "Мегаспорте"
Message-ID: 
From: "rss" 
Content-type: text/html

Again, are ALL Subject headers present in your case?

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



 



[#] Sat Oct 24 2020 09:09:44 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking. 

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



 



 



 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
[#] Sat Oct 24 2020 09:31:09 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking. 

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



 



 



 



 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
[#] Sun Oct 25 2020 11:04:26 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

Well, this Subject header issue is quite something. I'll try to be brief...

I removed /usr/local/ citadel, ctdlsupport and webcit dirs (rm -rf).

Then I did a fresh install of Citadel to make sure there are no side-effects or residual something that remained from my 8.24 version.

Then I created the accounts in Thunderbird to IMAP access citadel account. So, the RSS room would get automatically copied to Thunderbird.

Now...

Subject header problem, as viewed in that context:

The Subject headers, as viewing from Citadel, are absent on TASS RSS feed: http://tass.ru/rss/v2.xml
But they look fine as automatically copied via IMAP to Thunderbird. That means to me that headers do exist and the reason we see them as absent is because they are being trimmed down to pure English/Latin characters if present in the BEGINNING of the subject header.
Except in Thunderbird even though the Subject header looks fine, but article body is scrambled. (see below).

It is like KOI-8 encoded characters (in the body) are interpreted as UTF-8, which is a default for XML feeds.
So, the article body encoding does not recognize that it is not a UTF-8 char set, and we are reading KOI-8 or Windows-1251 as UTF-8.

Thunderbird version: (this is how it looks in Thunderbird)

Subject: В Ярославле разработали приложение, автоматически вызывающее скорую в случае опасности

Пока оно предназначено для сотрудников "Россетей", работающих с электрооборудованием, однако разработчики готовы модернизировать его и для других категорий граждан

https://tass.ru/obschestvo/9811873

Sat Oct 24 2020 09:31:09 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking. 

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



 



 



 



 



 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
[#] Sun Oct 25 2020 12:58:08 EDT from platonov

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

If we want to look and see how Thunderbird handles this RSS feed - http://tass.ru/rss/v2.xml we can add that feed directly to Thunderbird and then look at it.

Taking one article as an example and saving it in Thunderbird as .eml  (which is a standard SMTP article format):

X-Mozilla-Status: 0041
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
Received: by localhost; Sun, 25 Oct 2020 18:12:04 +0200
Date: Sun, 25 Oct 2020 18:25:28 +0300
Message-Id: <https://tass.ru/obschestvo/9812229@localhost.localdomain>
From: <ТАСС>
MIME-Version: 1.0
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Keywords: Общество, Спорт, Бой Нурмагомедов - Гэтжи
Content-Transfer-Encoding: 8bit
Content-Base: https://tass.ru/obschestvo/9812229
Content-Type: text/html; charset=UTF-8

<html>
<head>
<title>Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором</title>
<base href="https://tass.ru/obschestvo/9812229">
</head>
<body id="msgFeedSummaryBody" selected="false">
Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА
</body>
</html>

When we look at the same article in Citadel (doing view as email) and pushing headers menu choice in the article, what we get is this:

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: <5F959A4A-0000777E@preciseinfo.org>
From: "rss" <rss@preciseinfo.org>
Content-type: text/html

So, there is no charset specified. The question is "how come"?

Does anybody know?

Sun Oct 25 2020 16:56:40 EET from veeren Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Sun Oct 25 2020 11:04:26 EDT from platonov @ Uncensored
Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, this Subject header issue is quite something. I'll try to be brief...

I removed /usr/local/ citadel, ctdlsupport and webcit dirs (rm -rf).

Then I did a fresh install of Citadel to make sure there are no side-effects or residual something that remained from my 8.24 version.

Then I created the accounts in Thunderbird to IMAP access citadel account. So, the RSS room would get automatically copied to Thunderbird.

Now...

Subject header problem, as viewed in that context:

The Subject headers, as viewing from Citadel, are absent on TASS RSS feed: http://tass.ru/rss/v2.xml
But they look fine as automatically copied via IMAP to Thunderbird. That means to me that headers do exist and the reason we see them as absent is because they are being trimmed down to pure English/Latin characters if present in the BEGINNING of the subject header.
Except in Thunderbird even though the Subject header looks fine, but article body is scrambled. (see below).

It is like KOI-8 encoded characters (in the body) are interpreted as UTF-8, which is a default for XML feeds.
So, the article body encoding does not recognize that it is not a UTF-8 char set, and we are reading KOI-8 or Windows-1251 as UTF-8.

Thunderbird version: (this is how it looks in Thunderbird)

Subject: В Ярославле разработали приложение, автоматически вызывающее скорую в случае опасности

Пока оно предназначено для сотрудников "Россетей", работающих с электрооборудованием, однако разработчики готовы модернизировать его и для других категорий граждан

https://tass.ru/obschestvo/9811873

Sat Oct 24 2020 09:31:09 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking.

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation.


 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
[#] Sun Oct 25 2020 12:59:56 EDT from warbaby

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

http://tass.ru/rss/v2.xml says it's sending utf-8, but also uses a non-standard ISO language string.. ru-ru .. it should probably be ru_RU or just ru .. this might be confusing the XML Parser, see rss_parse_feed() in serv_rssclient.c 296. see below try

wget -q -S -O - http://tass.ru/rss/v2.xml
  HTTP/1.1 200 OK
  Server: nginx/1.19.0
  Date: Sun, 25 Oct 2020 15:50:26 GMT
  Content-Type: application/rss+xml, application/rdf+xml;q=0.8, application/atom+xml;q=0.6, application/xml;q=0.4, text/xml;q=0.4; charset=utf-8
  Transfer-Encoding: chunked
  Connection: keep-alive
  Set-Cookie: tass_uuid=4B9117AF-A95A-42C5-B5A5-ECDF06C230FB; Path=/; Expires=Mon, 25-Oct-21 15:50:26 GMT
  X-Frame-Options: SAMEORIGIN
  X-XSS-Protection: 1; mode=block
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  X-Content-Type-Options: nosniff
<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:atom="https://www.w3.org/2005/Atom"><channel><title>ТАСС</title>
<description>ИНФОРМАЦИОННОЕ АГЕНТСТВО РОССИИ ТАСС</description>
<language>ru-ru</language>
<link><![CDATA[https://tass.ru]]></link>

We should investigate XML_SetCharacterDataHandler() .. because it really does look like the utf-8 title is being converted back into KOI-8, which is the "mojibake" .. then it's probably stripped out somewhere else (but only apparantly in the title/subject).

This is pretty much what we're looking at. You might try to log the entire xml body in rss_pull_one_feed() at about 366.. then log it again after it is parsed, and find out what is actually going on.

Sun Oct 25 2020 11:04:26 AM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, this Subject header issue is quite something. I'll try to be brief...

I removed /usr/local/ citadel, ctdlsupport and webcit dirs (rm -rf).

Then I did a fresh install of Citadel to make sure there are no side-effects or residual something that remained from my 8.24 version.

Then I created the accounts in Thunderbird to IMAP access citadel account. So, the RSS room would get automatically copied to Thunderbird.

Now...

Subject header problem, as viewed in that context:

The Subject headers, as viewing from Citadel, are absent on TASS RSS feed: http://tass.ru/rss/v2.xml
But they look fine as automatically copied via IMAP to Thunderbird. That means to me that headers do exist and the reason we see them as absent is because they are being trimmed down to pure English/Latin characters if present in the BEGINNING of the subject header.
Except in Thunderbird even though the Subject header looks fine, but article body is scrambled. (see below).

It is like KOI-8 encoded characters (in the body) are interpreted as UTF-8, which is a default for XML feeds.
So, the article body encoding does not recognize that it is not a UTF-8 char set, and we are reading KOI-8 or Windows-1251 as UTF-8.

Thunderbird version: (this is how it looks in Thunderbird)

Subject: В Ярославле разработали приложение, автоматически вызывающее скорую в случае опасности

Пока оно предназначено для сотрудников "Россетей", работающих с электрооборудованием, однако разработчики готовы модернизировать его и для других категорий граждан

https://tass.ru/obschestvo/9811873

Sat Oct 24 2020 09:31:09 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking. 

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation. 


 



 



 



 



 



 



 



 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
[#] Sun Oct 25 2020 13:05:38 EDT from warbaby

Subject: Re: Is there a problem of incorrect displaying of Subject: header?

[Reply] [ReplyQuoted] [Headers] [Print]

You will probably want to read my last message, but a faster test would be to download the rss/xml file, then play around with editing the language string. 

Thunderbirds rss parser is probably advanced enough to figure out what the characters set should be, regardless of the language specified in the xml file..

Citadel probably giving you exactly what is asked for (XML_Parser) .. UTF-8, converted to KOI-8 ..

I'll bet if you play around with the ISO language code, you can probably get it to parse.. so the fastest fix might be to write a wrapper script which grabs the xml and fixes the lang code..

 

 

Sun Oct 25 2020 12:58:08 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

If we want to look and see how Thunderbird handles this RSS feed - http://tass.ru/rss/v2.xml we can add that feed directly to Thunderbird and then look at it.

Taking one article as an example and saving it in Thunderbird as .eml  (which is a standard SMTP article format):

X-Mozilla-Status: 0041
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
Received: by localhost; Sun, 25 Oct 2020 18:12:04 +0200
Date: Sun, 25 Oct 2020 18:25:28 +0300
Message-Id: <https://tass.ru/obschestvo/9812229@localhost.localdomain>
From: <ТАСС>
MIME-Version: 1.0
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Keywords: Общество, Спорт, Бой Нурмагомедов - Гэтжи
Content-Transfer-Encoding: 8bit
Content-Base: https://tass.ru/obschestvo/9812229
Content-Type: text/html; charset=UTF-8

<html>
<head>
<title>Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором</title>
<base href="https://tass.ru/obschestvo/9812229">
</head>
<body id="msgFeedSummaryBody" selected="false">
Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА
</body>
</html>

When we look at the same article in Citadel (doing view as email) and pushing headers menu choice in the article, what we get is this:

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: <5F959A4A-0000777E@preciseinfo.org>
From: "rss" <rss@preciseinfo.org>
Content-type: text/html

So, there is no charset specified. The question is "how come"?

Does anybody know?

Sun Oct 25 2020 16:56:40 EET from veeren Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Sun Oct 25 2020 11:04:26 EDT from platonov @ Uncensored
Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, this Subject header issue is quite something. I'll try to be brief...

I removed /usr/local/ citadel, ctdlsupport and webcit dirs (rm -rf).

Then I did a fresh install of Citadel to make sure there are no side-effects or residual something that remained from my 8.24 version.

Then I created the accounts in Thunderbird to IMAP access citadel account. So, the RSS room would get automatically copied to Thunderbird.

Now...

Subject header problem, as viewed in that context:

The Subject headers, as viewing from Citadel, are absent on TASS RSS feed: http://tass.ru/rss/v2.xml
But they look fine as automatically copied via IMAP to Thunderbird. That means to me that headers do exist and the reason we see them as absent is because they are being trimmed down to pure English/Latin characters if present in the BEGINNING of the subject header.
Except in Thunderbird even though the Subject header looks fine, but article body is scrambled. (see below).

It is like KOI-8 encoded characters (in the body) are interpreted as UTF-8, which is a default for XML feeds.
So, the article body encoding does not recognize that it is not a UTF-8 char set, and we are reading KOI-8 or Windows-1251 as UTF-8.

Thunderbird version: (this is how it looks in Thunderbird)

Subject: В Ярославле разработали приложение, автоматически вызывающее скорую в случае опасности

Пока оно предназначено для сотрудников "Россетей", работающих с электрооборудованием, однако разработчики готовы модернизировать его и для других категорий граждан

https://tass.ru/obschestvo/9811873

Sat Oct 24 2020 09:31:09 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking.

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?
Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation.


 



 



serv_rssclient.c (text/x-csrc, 15729 bytes) [ View | Download ]
Go to page: First ... 15 16 17 18 [19] 20 21 22 23 ... Last