The Subject: header is shown correctly while viewing the articles using textclient. You can view the same article from webcit and text client and see the difference.

Mon Oct 26 2020 09:10:35 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Just wanted to let you know that <p>VTimes</p> is in the XML source.

Sun Oct 25 2020 17:50:31 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

Sun Oct 25 2020 17:04:37 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I think you can take a break for now platonov..

Well, I'd like to try some things to get this issue resolved. I need to know if I can rely on cit929 with these RSS issues. Else, I'd have to go back to my old 8.24. The way it is right now is not something I'd be willing to swallow.

But I need to be able to build citserver. I did comment out the rm -rf in the easyinstall.sh and then tried to do ./configure. It complained about sieve2.h configure: error: sieve2.h was not found and is required. But I just did a fresh install on citadel yesterday, so... why is it getting stuck there and what do I do to be able to build citserver? Is there any documentation on building and dependencies?

although it might be interesting to still create a list of Russian language feeds, and see if we can find any that parse..

Somehow we're getting Header Injected.. which as I expected, appears to be related.. so it's  going to take more research.

It is in the <author> field in XML source. So, it isn't really "injected", it was there to begin with.

I'll have to get in touch with the "Big Guy".. [Not Joe Biden in this case, but the "Big Guy" knows who he is...]

Return-Path: <p>          VTimes       </p>
Date: Fri, 23 Oct 2020 12:27:00 -0000
Subject: В Москве введут скидки на проезд на двух линиях метро
 

It looks like we are running into a problem having both HTML Entities and the UTF-8 Russian in the subject line..

See this xml at https://www.vtimes.io/rss

 

Sun Oct 25 2020 04:35:43 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'm using grep (ag actually) + my brain.

I'll have a look at cscope.. thanks for the tip!

 

Sun Oct 25 2020 03:49:24 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Btw, are you using cscope?

- Highly recommended. Don't you even START thinking about finding things in citadel if you don't have its database in cscope!!!

Sun Oct 25 2020 14:29:17 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Okay, that did not work..trying something else..

Return-Path: rss
Date: Sun, 25 Oct 2020 13:30:17 -0000
Subject: Матыцин назвал Нурмагомедова выдающимся борцом современности
Message-ID: <5F95C063-0000163F@southernwork.org>
From: "rss" <rss@southernwork.org>
Content-type: text/html

Sun Oct 25 2020 02:27:23 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Whatever you sent probably got stripped away, because I don't see anything.

So, here's my test...

1) Log into a remote vps..

2) wget -O - http://tass.ru/rss/v2.xml

3) Edit <language>ru-ru</language> to be <language>ru_RU</language>

4) Add new test room in webcit.. add new feed under Remote Retrieval (link to the new edited file..)

5) Just waiting for the event to run to see what it looks like now..

 

Sun Oct 25 2020 02:08:23 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

Here's the complete article as viewed in Citadel: (header is obtained from Citadel [headers] article menu choice while viewing the article, and body is directly copied from Citadel article source)

Notice that the Subject header is all screwed up, but the article body is fine.

Now, when you look at a screwed up article header in Citadel, push the [headers] article menu choice and you will see that the Subject header is actually there, but its character encoding causes something in Citadel to trim it down to 0.

And, what is even more interesting and is proof is that we are dealing with character encoding problem is that the encoding is specified as the fist thing in the SMTP Subject header.

So, if you look at the article with trunkated Subject and Subject starts with English/Latin characters, the ARE displayed correctly and if you scroll that feed in citadel, you will notice that a few articles begin either with English chars or delimiters.

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: 
From: "rss" 
Content-type: text/html

Sun Oct 25 2020 17:25:28 EET from rss

Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА

https://tass.ru/obschestvo/9812229

Well, there are several Cyrillic encodings, including KOI, Windows-1251, 1252. So, which one we are dealing with is not quite clear and probably not very relevant.

Sun Oct 25 2020 13:39:59 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, you are way ahead of me on this and I am not quite following some things... see in-line.

Sun Oct 25 2020 13:12:51 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

"This is the probably the problem....." Got stripped out of my last message..

<?php

$s = 'Студентов университета &quot;Сириус&quot; в Сочи научили отражать кибератаки на банкоматы';
$s = html_entity_decode($s); 
$moji= mb_convert_encoding($s, 'UTF-8', 'KOI8-U');  # uses iconv
print $moji.PHP_EOL;

# prints п║я┌я┐пЄп╣пҐя┌п╬п╡ я┐пҐп╦п╡п╣я─я│п╦я┌п╣я┌п╟ "п║п╦я─п╦я┐я│" п╡ п║п╬я┤п╦ пҐп╟я┐я┤п╦п╩п╦ п╬я┌я─п╟пІп╟я┌я▄ п╨п╦п╠п╣я─п╟я┌п╟п╨п╦ пҐп╟ п╠п╟пҐп╨п╬п╪п╟я┌я▀

Sun Oct 25 2020 01:05:38 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

You will probably want to read my last message, but a faster test would be to download the rss/xml file, then play around with editing the language string. 

I am not following this "play around with editing the language string".

Thunderbirds rss parser is probably advanced enough to figure out what the characters set should be, regardless of the language specified in the xml file..

Citadel probably giving you exactly what is asked for (XML_Parser) .. UTF-8, converted to KOI-8 ..

I'll bet if you play around with the ISO language code, you can probably get it to parse.. so the fastest fix might be to write a wrapper script which grabs the xml and fixes the lang code..

Wooo! I have no intention to "play around with the ISO language code". I bet you can do a much better job at that :)

 

Sun Oct 25 2020 12:58:08 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

If we want to look and see how Thunderbird handles this RSS feed - http://tass.ru/rss/v2.xml we can add that feed directly to Thunderbird and then look at it.

Taking one article as an example and saving it in Thunderbird as .eml  (which is a standard SMTP article format):

X-Mozilla-Status: 0041
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
Received: by localhost; Sun, 25 Oct 2020 18:12:04 +0200
Date: Sun, 25 Oct 2020 18:25:28 +0300
Message-Id: <https://tass.ru/obschestvo/9812229@localhost.localdomain>
From: <ТАСС>
MIME-Version: 1.0
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Keywords: Общество, Спорт, Бой Нурмагомедов - Гэтжи
Content-Transfer-Encoding: 8bit
Content-Base: https://tass.ru/obschestvo/9812229
Content-Type: text/html; charset=UTF-8

<html>
<head>
<title>Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором</title>
<base href="https://tass.ru/obschestvo/9812229">
</head>
<body id="msgFeedSummaryBody" selected="false">
Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА
</body>
</html>

When we look at the same article in Citadel (doing view as email) and pushing headers menu choice in the article, what we get is this:

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: <5F959A4A-0000777E@preciseinfo.org>
From: "rss" <rss@preciseinfo.org>
Content-type: text/html

So, there is no charset specified. The question is "how come"?

Does anybody know?

Sun Oct 25 2020 16:56:40 EET from veeren Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Sun Oct 25 2020 11:04:26 EDT from platonov @ Uncensored
Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, this Subject header issue is quite something. I'll try to be brief...

I removed /usr/local/ citadel, ctdlsupport and webcit dirs (rm -rf).

Then I did a fresh install of Citadel to make sure there are no side-effects or residual something that remained from my 8.24 version.

Then I created the accounts in Thunderbird to IMAP access citadel account. So, the RSS room would get automatically copied to Thunderbird.

Now...

Subject header problem, as viewed in that context:

The Subject headers, as viewing from Citadel, are absent on TASS RSS feed: http://tass.ru/rss/v2.xml
But they look fine as automatically copied via IMAP to Thunderbird. That means to me that headers do exist and the reason we see them as absent is because they are being trimmed down to pure English/Latin characters if present in the BEGINNING of the subject header.
Except in Thunderbird even though the Subject header looks fine, but article body is scrambled. (see below).

It is like KOI-8 encoded characters (in the body) are interpreted as UTF-8, which is a default for XML feeds.
So, the article body encoding does not recognize that it is not a UTF-8 char set, and we are reading KOI-8 or Windows-1251 as UTF-8.

Thunderbird version: (this is how it looks in Thunderbird)

Subject: В Ярославле разработали приложение, автоматически вызывающее скорую в случае опасности

Пока оно предназначено для сотрудников "Россетей", работающих с электрооборудованием, однако разработчики готовы модернизировать его и для других категорий граждан

https://tass.ru/obschestvo/9811873

Sat Oct 24 2020 09:31:09 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking.

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation.

 

 

 

 

 

 

 

 

 

 

 

Wed Oct 28 2020 14:03:07 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

The Subject: header is shown correctly while viewing the articles using textclient. You can view the same article from webcit and text client and see the difference.

Mon Oct 26 2020 09:10:35 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Just wanted to let you know that <p>VTimes</p> is in the XML source.

Sun Oct 25 2020 17:50:31 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

Sun Oct 25 2020 17:04:37 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I think you can take a break for now platonov..

Well, I'd like to try some things to get this issue resolved. I need to know if I can rely on cit929 with these RSS issues. Else, I'd have to go back to my old 8.24. The way it is right now is not something I'd be willing to swallow.

But I need to be able to build citserver. I did comment out the rm -rf in the easyinstall.sh and then tried to do ./configure. It complained about sieve2.h configure: error: sieve2.h was not found and is required. But I just did a fresh install on citadel yesterday, so... why is it getting stuck there and what do I do to be able to build citserver? Is there any documentation on building and dependencies?

although it might be interesting to still create a list of Russian language feeds, and see if we can find any that parse..

Somehow we're getting Header Injected.. which as I expected, appears to be related.. so it's  going to take more research.

It is in the <author> field in XML source. So, it isn't really "injected", it was there to begin with.

I'll have to get in touch with the "Big Guy".. [Not Joe Biden in this case, but the "Big Guy" knows who he is...]

Return-Path: <p>          VTimes       </p>
Date: Fri, 23 Oct 2020 12:27:00 -0000
Subject: В Москве введут скидки на проезд на двух линиях метро
 

It looks like we are running into a problem having both HTML Entities and the UTF-8 Russian in the subject line..

See this xml at https://www.vtimes.io/rss

 

Sun Oct 25 2020 04:35:43 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'm using grep (ag actually) + my brain.

I'll have a look at cscope.. thanks for the tip!

 

Sun Oct 25 2020 03:49:24 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Btw, are you using cscope?

- Highly recommended. Don't you even START thinking about finding things in citadel if you don't have its database in cscope!!!

Sun Oct 25 2020 14:29:17 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Okay, that did not work..trying something else..

Return-Path: rss
Date: Sun, 25 Oct 2020 13:30:17 -0000
Subject: Матыцин назвал Нурмагомедова выдающимся борцом современности
Message-ID: <5F95C063-0000163F@southernwork.org>
From: "rss" <rss@southernwork.org>
Content-type: text/html

Sun Oct 25 2020 02:27:23 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Whatever you sent probably got stripped away, because I don't see anything.

So, here's my test...

1) Log into a remote vps..

2) wget -O - http://tass.ru/rss/v2.xml

3) Edit <language>ru-ru</language> to be <language>ru_RU</language>

4) Add new test room in webcit.. add new feed under Remote Retrieval (link to the new edited file..)

5) Just waiting for the event to run to see what it looks like now..

 

Sun Oct 25 2020 02:08:23 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

Here's the complete article as viewed in Citadel: (header is obtained from Citadel [headers] article menu choice while viewing the article, and body is directly copied from Citadel article source)

Notice that the Subject header is all screwed up, but the article body is fine.

Now, when you look at a screwed up article header in Citadel, push the [headers] article menu choice and you will see that the Subject header is actually there, but its character encoding causes something in Citadel to trim it down to 0.

And, what is even more interesting and is proof is that we are dealing with character encoding problem is that the encoding is specified as the fist thing in the SMTP Subject header.

So, if you look at the article with trunkated Subject and Subject starts with English/Latin characters, the ARE displayed correctly and if you scroll that feed in citadel, you will notice that a few articles begin either with English chars or delimiters.

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: 
From: "rss" 
Content-type: text/html

Sun Oct 25 2020 17:25:28 EET from rss

Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА

https://tass.ru/obschestvo/9812229

Well, there are several Cyrillic encodings, including KOI, Windows-1251, 1252. So, which one we are dealing with is not quite clear and probably not very relevant.

Sun Oct 25 2020 13:39:59 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, you are way ahead of me on this and I am not quite following some things... see in-line.

Sun Oct 25 2020 13:12:51 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

"This is the probably the problem....." Got stripped out of my last message..

<?php

$s = 'Студентов университета &quot;Сириус&quot; в Сочи научили отражать кибератаки на банкоматы';
$s = html_entity_decode($s); 
$moji= mb_convert_encoding($s, 'UTF-8', 'KOI8-U');  # uses iconv
print $moji.PHP_EOL;

# prints п║я┌я┐пЄп╣пҐя┌п╬п╡ я┐пҐп╦п╡п╣я─я│п╦я┌п╣я┌п╟ "п║п╦я─п╦я┐я│" п╡ п║п╬я┤п╦ пҐп╟я┐я┤п╦п╩п╦ п╬я┌я─п╟пІп╟я┌я▄ п╨п╦п╠п╣я─п╟я┌п╟п╨п╦ пҐп╟ п╠п╟пҐп╨п╬п╪п╟я┌я▀

Sun Oct 25 2020 01:05:38 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

You will probably want to read my last message, but a faster test would be to download the rss/xml file, then play around with editing the language string. 

I am not following this "play around with editing the language string".

Thunderbirds rss parser is probably advanced enough to figure out what the characters set should be, regardless of the language specified in the xml file..

Citadel probably giving you exactly what is asked for (XML_Parser) .. UTF-8, converted to KOI-8 ..

I'll bet if you play around with the ISO language code, you can probably get it to parse.. so the fastest fix might be to write a wrapper script which grabs the xml and fixes the lang code..

Wooo! I have no intention to "play around with the ISO language code". I bet you can do a much better job at that :)

 

Sun Oct 25 2020 12:58:08 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

If we want to look and see how Thunderbird handles this RSS feed - http://tass.ru/rss/v2.xml we can add that feed directly to Thunderbird and then look at it.

Taking one article as an example and saving it in Thunderbird as .eml  (which is a standard SMTP article format):

X-Mozilla-Status: 0041
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
Received: by localhost; Sun, 25 Oct 2020 18:12:04 +0200
Date: Sun, 25 Oct 2020 18:25:28 +0300
Message-Id: <https://tass.ru/obschestvo/9812229@localhost.localdomain>
From: <ТАСС>
MIME-Version: 1.0
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Keywords: Общество, Спорт, Бой Нурмагомедов - Гэтжи
Content-Transfer-Encoding: 8bit
Content-Base: https://tass.ru/obschestvo/9812229
Content-Type: text/html; charset=UTF-8

<html>
<head>
<title>Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором</title>
<base href="https://tass.ru/obschestvo/9812229">
</head>
<body id="msgFeedSummaryBody" selected="false">
Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА
</body>
</html>

When we look at the same article in Citadel (doing view as email) and pushing headers menu choice in the article, what we get is this:

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: <5F959A4A-0000777E@preciseinfo.org>
From: "rss" <rss@preciseinfo.org>
Content-type: text/html

So, there is no charset specified. The question is "how come"?

Does anybody know?

Sun Oct 25 2020 16:56:40 EET from veeren Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Sun Oct 25 2020 11:04:26 EDT from platonov @ Uncensored
Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, this Subject header issue is quite something. I'll try to be brief...

I removed /usr/local/ citadel, ctdlsupport and webcit dirs (rm -rf).

Then I did a fresh install of Citadel to make sure there are no side-effects or residual something that remained from my 8.24 version.

Then I created the accounts in Thunderbird to IMAP access citadel account. So, the RSS room would get automatically copied to Thunderbird.

Now...

Subject header problem, as viewed in that context:

The Subject headers, as viewing from Citadel, are absent on TASS RSS feed: http://tass.ru/rss/v2.xml
But they look fine as automatically copied via IMAP to Thunderbird. That means to me that headers do exist and the reason we see them as absent is because they are being trimmed down to pure English/Latin characters if present in the BEGINNING of the subject header.
Except in Thunderbird even though the Subject header looks fine, but article body is scrambled. (see below).

It is like KOI-8 encoded characters (in the body) are interpreted as UTF-8, which is a default for XML feeds.
So, the article body encoding does not recognize that it is not a UTF-8 char set, and we are reading KOI-8 or Windows-1251 as UTF-8.

Thunderbird version: (this is how it looks in Thunderbird)

Subject: В Ярославле разработали приложение, автоматически вызывающее скорую в случае опасности

Пока оно предназначено для сотрудников "Россетей", работающих с электрооборудованием, однако разработчики готовы модернизировать его и для других категорий граждан

https://tass.ru/obschestvo/9811873

Sat Oct 24 2020 09:31:09 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking.

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation.

 

 

 

 

 

 

 

 

 

 

 

 

Yes, it has been logged in project management, and goes along with some other things we're working on..

but having said that, please continue your research. Any documentation we can get is helpful and saves time.

There will be some other things that go along with this.. namely getting expat errors to bubble up and explain why certain feeds are failing. 

Wed Oct 28 2020 03:23:52 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'd like to ask if this subject header issue is being handled as an active issue, or it is being placed in the "backburner" or labeled as "non-issue"? In case it is being handled, here's some more info.

The message below with the Subject header correctly displayed was created with text client as a followup to an existing article and, even though the Subject header would not show up in webcit, it got "resurrected" when replying to the article.

The message header is a copy of [headers] article menu choice.
The Subject in the header below, even though it looks garbled, is actually displayed correctly as can be seen in the article during the article view (below).

Here is a copy of a complete article, preceeded by the article headers.
Notice the =?UTF-8 at the start of the Subject: header. That is the correct way of specifying the charset as far as I recall.

Return-Path: user
Date: Wed, 28 Oct 2020 20:22:30 +0200
References: 
Subject: =?UTF-8?B?UmU6INCi0LDQutGC0LDRgNC+0LI6INGD0LzQtdC90LjRjyDQk9GN0YLQttC4INCyINCx0L7RgNGM0LHQtSDQvtGH0LXQvdGMINC/0LXRgNC10L7RhtC10L3QuNC70LgsINC90L4g0Y3RgtC+INC90LUg0YPQvNCw0LvRj9C10YIg0LfQsNGB0LvRg9CzINCd0YPRgNC80LDQs9C+0LzQtdC00L7QstCw?=
Message-ID: 
From: "user" 

And this is the article itself. So, the Subject looks good indeed.

Wed Oct 28 2020 20:22:30 EET from adminski Subject: Re: Тактаров: умения Гэтжи в борьбе очень переоценили, но это не умаляет заслуг Нурмагомедова

Trying to type in Russian would garble the characters.

 

Wed Oct 28 2020 14:03:07 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

The Subject: header is shown correctly while viewing the articles using textclient. You can view the same article from webcit and text client and see the difference.

Mon Oct 26 2020 09:10:35 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Just wanted to let you know that <p>VTimes</p> is in the XML source.

Sun Oct 25 2020 17:50:31 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

Sun Oct 25 2020 17:04:37 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I think you can take a break for now platonov..

Well, I'd like to try some things to get this issue resolved. I need to know if I can rely on cit929 with these RSS issues. Else, I'd have to go back to my old 8.24. The way it is right now is not something I'd be willing to swallow.

But I need to be able to build citserver. I did comment out the rm -rf in the easyinstall.sh and then tried to do ./configure. It complained about sieve2.h configure: error: sieve2.h was not found and is required. But I just did a fresh install on citadel yesterday, so... why is it getting stuck there and what do I do to be able to build citserver? Is there any documentation on building and dependencies?

although it might be interesting to still create a list of Russian language feeds, and see if we can find any that parse..

Somehow we're getting Header Injected.. which as I expected, appears to be related.. so it's  going to take more research.

It is in the <author> field in XML source. So, it isn't really "injected", it was there to begin with.

I'll have to get in touch with the "Big Guy".. [Not Joe Biden in this case, but the "Big Guy" knows who he is...]

Return-Path: <p>          VTimes       </p>
Date: Fri, 23 Oct 2020 12:27:00 -0000
Subject: В Москве введут скидки на проезд на двух линиях метро
 

It looks like we are running into a problem having both HTML Entities and the UTF-8 Russian in the subject line..

See this xml at https://www.vtimes.io/rss

 

Sun Oct 25 2020 04:35:43 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'm using grep (ag actually) + my brain.

I'll have a look at cscope.. thanks for the tip!

 

Sun Oct 25 2020 03:49:24 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Btw, are you using cscope?

- Highly recommended. Don't you even START thinking about finding things in citadel if you don't have its database in cscope!!!

Sun Oct 25 2020 14:29:17 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Okay, that did not work..trying something else..

Return-Path: rss
Date: Sun, 25 Oct 2020 13:30:17 -0000
Subject: Матыцин назвал Нурмагомедова выдающимся борцом современности
Message-ID: <5F95C063-0000163F@southernwork.org>
From: "rss" <rss@southernwork.org>
Content-type: text/html

Sun Oct 25 2020 02:27:23 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Whatever you sent probably got stripped away, because I don't see anything.

So, here's my test...

1) Log into a remote vps..

2) wget -O - http://tass.ru/rss/v2.xml

3) Edit <language>ru-ru</language> to be <language>ru_RU</language>

4) Add new test room in webcit.. add new feed under Remote Retrieval (link to the new edited file..)

5) Just waiting for the event to run to see what it looks like now..

 

Sun Oct 25 2020 02:08:23 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

Here's the complete article as viewed in Citadel: (header is obtained from Citadel [headers] article menu choice while viewing the article, and body is directly copied from Citadel article source)

Notice that the Subject header is all screwed up, but the article body is fine.

Now, when you look at a screwed up article header in Citadel, push the [headers] article menu choice and you will see that the Subject header is actually there, but its character encoding causes something in Citadel to trim it down to 0.

And, what is even more interesting and is proof is that we are dealing with character encoding problem is that the encoding is specified as the fist thing in the SMTP Subject header.

So, if you look at the article with trunkated Subject and Subject starts with English/Latin characters, the ARE displayed correctly and if you scroll that feed in citadel, you will notice that a few articles begin either with English chars or delimiters.

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: 
From: "rss" 
Content-type: text/html

Sun Oct 25 2020 17:25:28 EET from rss

Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА

https://tass.ru/obschestvo/9812229

Well, there are several Cyrillic encodings, including KOI, Windows-1251, 1252. So, which one we are dealing with is not quite clear and probably not very relevant.

Sun Oct 25 2020 13:39:59 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, you are way ahead of me on this and I am not quite following some things... see in-line.

Sun Oct 25 2020 13:12:51 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

"This is the probably the problem....." Got stripped out of my last message..

<?php

$s = 'Студентов университета &quot;Сириус&quot; в Сочи научили отражать кибератаки на банкоматы';
$s = html_entity_decode($s); 
$moji= mb_convert_encoding($s, 'UTF-8', 'KOI8-U');  # uses iconv
print $moji.PHP_EOL;

# prints п║я┌я┐пЄп╣пҐя┌п╬п╡ я┐пҐп╦п╡п╣я─я│п╦я┌п╣я┌п╟ "п║п╦я─п╦я┐я│" п╡ п║п╬я┤п╦ пҐп╟я┐я┤п╦п╩п╦ п╬я┌я─п╟пІп╟я┌я▄ п╨п╦п╠п╣я─п╟я┌п╟п╨п╦ пҐп╟ п╠п╟пҐп╨п╬п╪п╟я┌я▀

Sun Oct 25 2020 01:05:38 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

You will probably want to read my last message, but a faster test would be to download the rss/xml file, then play around with editing the language string. 

I am not following this "play around with editing the language string".

Thunderbirds rss parser is probably advanced enough to figure out what the characters set should be, regardless of the language specified in the xml file..

Citadel probably giving you exactly what is asked for (XML_Parser) .. UTF-8, converted to KOI-8 ..

I'll bet if you play around with the ISO language code, you can probably get it to parse.. so the fastest fix might be to write a wrapper script which grabs the xml and fixes the lang code..

Wooo! I have no intention to "play around with the ISO language code". I bet you can do a much better job at that :)

 

Sun Oct 25 2020 12:58:08 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

If we want to look and see how Thunderbird handles this RSS feed - http://tass.ru/rss/v2.xml we can add that feed directly to Thunderbird and then look at it.

Taking one article as an example and saving it in Thunderbird as .eml  (which is a standard SMTP article format):

X-Mozilla-Status: 0041
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
Received: by localhost; Sun, 25 Oct 2020 18:12:04 +0200
Date: Sun, 25 Oct 2020 18:25:28 +0300
Message-Id: <https://tass.ru/obschestvo/9812229@localhost.localdomain>
From: <ТАСС>
MIME-Version: 1.0
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Keywords: Общество, Спорт, Бой Нурмагомедов - Гэтжи
Content-Transfer-Encoding: 8bit
Content-Base: https://tass.ru/obschestvo/9812229
Content-Type: text/html; charset=UTF-8

<html>
<head>
<title>Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором</title>
<base href="https://tass.ru/obschestvo/9812229">
</head>
<body id="msgFeedSummaryBody" selected="false">
Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА
</body>
</html>

When we look at the same article in Citadel (doing view as email) and pushing headers menu choice in the article, what we get is this:

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: <5F959A4A-0000777E@preciseinfo.org>
From: "rss" <rss@preciseinfo.org>
Content-type: text/html

So, there is no charset specified. The question is "how come"?

Does anybody know?

Sun Oct 25 2020 16:56:40 EET from veeren Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Sun Oct 25 2020 11:04:26 EDT from platonov @ Uncensored
Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, this Subject header issue is quite something. I'll try to be brief...

I removed /usr/local/ citadel, ctdlsupport and webcit dirs (rm -rf).

Then I did a fresh install of Citadel to make sure there are no side-effects or residual something that remained from my 8.24 version.

Then I created the accounts in Thunderbird to IMAP access citadel account. So, the RSS room would get automatically copied to Thunderbird.

Now...

Subject header problem, as viewed in that context:

The Subject headers, as viewing from Citadel, are absent on TASS RSS feed: http://tass.ru/rss/v2.xml
But they look fine as automatically copied via IMAP to Thunderbird. That means to me that headers do exist and the reason we see them as absent is because they are being trimmed down to pure English/Latin characters if present in the BEGINNING of the subject header.
Except in Thunderbird even though the Subject header looks fine, but article body is scrambled. (see below).

It is like KOI-8 encoded characters (in the body) are interpreted as UTF-8, which is a default for XML feeds.
So, the article body encoding does not recognize that it is not a UTF-8 char set, and we are reading KOI-8 or Windows-1251 as UTF-8.

Thunderbird version: (this is how it looks in Thunderbird)

Subject: В Ярославле разработали приложение, автоматически вызывающее скорую в случае опасности

Пока оно предназначено для сотрудников "Россетей", работающих с электрооборудованием, однако разработчики готовы модернизировать его и для других категорий граждан

https://tass.ru/obschestvo/9811873

Sat Oct 24 2020 09:31:09 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking.

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation.

 

 

 

 

 

 

 

 

 

 

 

 

 

platonov, I replied to this which should be in your inbox here, and would like to move the conversation out of Support for now.  

I think we're making progress, but probably not of interest to many people who are subscribed to Citadel Support. 

Please continue to correspond with me here on uncensored, by email.  We can post an update here after we have a patch or some other solution.

Wed Oct 28 2020 03:23:52 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

The message below with the Subject header correctly displayed was created with text client as a followup to an existing article and, even though the Subject header would not show up in webcit, it got "resurrected" when replying to the article.

The message header is a copy of [headers] article menu choice.
The Subject in the header below, even though it looks garbled, is actually displayed correctly as can be seen in the article during the article view (below).

Here is a copy of a complete article, preceeded by the article headers.
Notice the =?UTF-8 at the start of the Subject: header. That is the correct way of specifying the charset as far as I recall.

Return-Path: user
Date: Wed, 28 Oct 2020 20:22:30 +0200
References: 
Subject: =?UTF-8?B?UmU6INCi0LDQutGC0LDRgNC+0LI6INGD0LzQtdC90LjRjyDQk9GN0YLQttC4INCyINCx0L7RgNGM0LHQtSDQvtGH0LXQvdGMINC/0LXRgNC10L7RhtC10L3QuNC70LgsINC90L4g0Y3RgtC+INC90LUg0YPQvNCw0LvRj9C10YIg0LfQsNGB0LvRg9CzINCd0YPRgNC80LDQs9C+0LzQtdC00L7QstCw?=
Message-ID: 
From: "user" 

And this is the article itself. So, the Subject looks good indeed.

Wed Oct 28 2020 20:22:30 EET from adminski Subject: Re: Тактаров: умения Гэтжи в борьбе очень переоценили, но это не умаляет заслуг Нурмагомедова

Trying to type in Russian would garble the characters.

 

Wed Oct 28 2020 14:03:07 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

The Subject: header is shown correctly while viewing the articles using textclient. You can view the same article from webcit and text client and see the difference.

Mon Oct 26 2020 09:10:35 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Just wanted to let you know that <p>VTimes</p> is in the XML source.

Sun Oct 25 2020 17:50:31 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

Sun Oct 25 2020 17:04:37 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I think you can take a break for now platonov..

Well, I'd like to try some things to get this issue resolved. I need to know if I can rely on cit929 with these RSS issues. Else, I'd have to go back to my old 8.24. The way it is right now is not something I'd be willing to swallow.

But I need to be able to build citserver. I did comment out the rm -rf in the easyinstall.sh and then tried to do ./configure. It complained about sieve2.h configure: error: sieve2.h was not found and is required. But I just did a fresh install on citadel yesterday, so... why is it getting stuck there and what do I do to be able to build citserver? Is there any documentation on building and dependencies?

although it might be interesting to still create a list of Russian language feeds, and see if we can find any that parse..

Somehow we're getting Header Injected.. which as I expected, appears to be related.. so it's  going to take more research.

It is in the <author> field in XML source. So, it isn't really "injected", it was there to begin with.

I'll have to get in touch with the "Big Guy".. [Not Joe Biden in this case, but the "Big Guy" knows who he is...]

Return-Path: <p>          VTimes       </p>
Date: Fri, 23 Oct 2020 12:27:00 -0000
Subject: В Москве введут скидки на проезд на двух линиях метро
 

It looks like we are running into a problem having both HTML Entities and the UTF-8 Russian in the subject line..

See this xml at https://www.vtimes.io/rss

 

Sun Oct 25 2020 04:35:43 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'm using grep (ag actually) + my brain.

I'll have a look at cscope.. thanks for the tip!

 

Sun Oct 25 2020 03:49:24 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Btw, are you using cscope?

- Highly recommended. Don't you even START thinking about finding things in citadel if you don't have its database in cscope!!!

Sun Oct 25 2020 14:29:17 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Okay, that did not work..trying something else..

Return-Path: rss
Date: Sun, 25 Oct 2020 13:30:17 -0000
Subject: Матыцин назвал Нурмагомедова выдающимся борцом современности
Message-ID: <5F95C063-0000163F@southernwork.org>
From: "rss" <rss@southernwork.org>
Content-type: text/html

Sun Oct 25 2020 02:27:23 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Whatever you sent probably got stripped away, because I don't see anything.

So, here's my test...

1) Log into a remote vps..

2) wget -O - http://tass.ru/rss/v2.xml

3) Edit <language>ru-ru</language> to be <language>ru_RU</language>

4) Add new test room in webcit.. add new feed under Remote Retrieval (link to the new edited file..)

5) Just waiting for the event to run to see what it looks like now..

 

Sun Oct 25 2020 02:08:23 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

 

Here's the complete article as viewed in Citadel: (header is obtained from Citadel [headers] article menu choice while viewing the article, and body is directly copied from Citadel article source)

Notice that the Subject header is all screwed up, but the article body is fine.

Now, when you look at a screwed up article header in Citadel, push the [headers] article menu choice and you will see that the Subject header is actually there, but its character encoding causes something in Citadel to trim it down to 0.

And, what is even more interesting and is proof is that we are dealing with character encoding problem is that the encoding is specified as the fist thing in the SMTP Subject header.

So, if you look at the article with trunkated Subject and Subject starts with English/Latin characters, the ARE displayed correctly and if you scroll that feed in citadel, you will notice that a few articles begin either with English chars or delimiters.

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: 
From: "rss" 
Content-type: text/html

Sun Oct 25 2020 17:25:28 EET from rss

Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА

https://tass.ru/obschestvo/9812229

Well, there are several Cyrillic encodings, including KOI, Windows-1251, 1252. So, which one we are dealing with is not quite clear and probably not very relevant.

Sun Oct 25 2020 13:39:59 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, you are way ahead of me on this and I am not quite following some things... see in-line.

Sun Oct 25 2020 13:12:51 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

"This is the probably the problem....." Got stripped out of my last message..

<?php

$s = 'Студентов университета &quot;Сириус&quot; в Сочи научили отражать кибератаки на банкоматы';
$s = html_entity_decode($s); 
$moji= mb_convert_encoding($s, 'UTF-8', 'KOI8-U');  # uses iconv
print $moji.PHP_EOL;

# prints п║я┌я┐пЄп╣пҐя┌п╬п╡ я┐пҐп╦п╡п╣я─я│п╦я┌п╣я┌п╟ "п║п╦я─п╦я┐я│" п╡ п║п╬я┤п╦ пҐп╟я┐я┤п╦п╩п╦ п╬я┌я─п╟пІп╟я┌я▄ п╨п╦п╠п╣я─п╟я┌п╟п╨п╦ пҐп╟ п╠п╟пҐп╨п╬п╪п╟я┌я▀

Sun Oct 25 2020 01:05:38 PM EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

You will probably want to read my last message, but a faster test would be to download the rss/xml file, then play around with editing the language string. 

I am not following this "play around with editing the language string".

Thunderbirds rss parser is probably advanced enough to figure out what the characters set should be, regardless of the language specified in the xml file..

Citadel probably giving you exactly what is asked for (XML_Parser) .. UTF-8, converted to KOI-8 ..

I'll bet if you play around with the ISO language code, you can probably get it to parse.. so the fastest fix might be to write a wrapper script which grabs the xml and fixes the lang code..

Wooo! I have no intention to "play around with the ISO language code". I bet you can do a much better job at that :)

 

Sun Oct 25 2020 12:58:08 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

If we want to look and see how Thunderbird handles this RSS feed - http://tass.ru/rss/v2.xml we can add that feed directly to Thunderbird and then look at it.

Taking one article as an example and saving it in Thunderbird as .eml  (which is a standard SMTP article format):

X-Mozilla-Status: 0041
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
Received: by localhost; Sun, 25 Oct 2020 18:12:04 +0200
Date: Sun, 25 Oct 2020 18:25:28 +0300
Message-Id: <https://tass.ru/obschestvo/9812229@localhost.localdomain>
From: <ТАСС>
MIME-Version: 1.0
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Keywords: Общество, Спорт, Бой Нурмагомедов - Гэтжи
Content-Transfer-Encoding: 8bit
Content-Base: https://tass.ru/obschestvo/9812229
Content-Type: text/html; charset=UTF-8

<html>
<head>
<title>Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором</title>
<base href="https://tass.ru/obschestvo/9812229">
</head>
<body id="msgFeedSummaryBody" selected="false">
Всего на счету 32-летнего бойца смешанных единоборств 29 побед и ни одного поражения в ММА
</body>
</html>

When we look at the same article in Citadel (doing view as email) and pushing headers menu choice in the article, what we get is this:

Return-Path: rss
Date: Sun, 25 Oct 2020 17:25:28 +0200
Subject: Песков заявил, что решение Нурмагомедова завершить карьеру является его выбором
Message-ID: <5F959A4A-0000777E@preciseinfo.org>
From: "rss" <rss@preciseinfo.org>
Content-type: text/html

So, there is no charset specified. The question is "how come"?

Does anybody know?

Sun Oct 25 2020 16:56:40 EET from veeren Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Sun Oct 25 2020 11:04:26 EDT from platonov @ Uncensored
Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, this Subject header issue is quite something. I'll try to be brief...

I removed /usr/local/ citadel, ctdlsupport and webcit dirs (rm -rf).

Then I did a fresh install of Citadel to make sure there are no side-effects or residual something that remained from my 8.24 version.

Then I created the accounts in Thunderbird to IMAP access citadel account. So, the RSS room would get automatically copied to Thunderbird.

Now...

Subject header problem, as viewed in that context:

The Subject headers, as viewing from Citadel, are absent on TASS RSS feed: http://tass.ru/rss/v2.xml
But they look fine as automatically copied via IMAP to Thunderbird. That means to me that headers do exist and the reason we see them as absent is because they are being trimmed down to pure English/Latin characters if present in the BEGINNING of the subject header.
Except in Thunderbird even though the Subject header looks fine, but article body is scrambled. (see below).

It is like KOI-8 encoded characters (in the body) are interpreted as UTF-8, which is a default for XML feeds.
So, the article body encoding does not recognize that it is not a UTF-8 char set, and we are reading KOI-8 or Windows-1251 as UTF-8.

Thunderbird version: (this is how it looks in Thunderbird)

Subject: В Ярославле разработали приложение, автоматически вызывающее скорую в случае опасности

Пока оно предназначено для сотрудников "Россетей", работающих с электрооборудованием, однако разработчики готовы модернизировать его и для других категорий граждан

https://tass.ru/obschestvo/9811873

Sat Oct 24 2020 09:31:09 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Actually, now I have 2 citadel installations, one is the original 64 bit 929 and the other is the one that originally had cit 824 on it, running on Ubuntu 18.04 and was upgraded in-place to 929 via easyinstall.

Now, BOTH of these have the same exact problem with Subject: header. So, I am wondering what kind of a miracle had happened on your version that makes Subject headers work correctly...

Sat Oct 24 2020 09:09:44 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Yep, finding it would be great!

Btw, what version of Citadel are you running? When I did a fresh install it was 9.17, but I did apt-get install first and had some problems of starting it up. So, I removed those packages and did the easyinstall version and everything went fine and it was running OK from the very beginning.

I wonder if that had something to do with my Subject problem. May be something did not get completely removed when I deinstalled the package.

Since you are saying it works on your box, then what is the difference between your and mine.

Fri Oct 23 2020 17:42:03 EDT from warbaby Subject: Re: Is there a problem of incorrect displaying of Subject: header?

No, not really, but am learning. I encourage you to hang in there & keep looking.

We both really need to fix this and I am committed to finding it.

Well, that's good to hear. At this point I's be even willing to do something drastic to verify that it is not a problem of dpkg version of install vs. easyinstall.

We need all the eyes we can get...

Fri Oct 23 2020 05:28:05 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I am thinking about looking at diff between 8.24 and 9.28 of serv_rssclient.c. Looks like lots of changes...

Well, those trim and strip functions and the places where they are used is something to look at.

Btw, do you know that code well enough?

Fri Oct 23 2020 16:23:52 EDT from warbaby @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

it's in modules/rssclient/serv_rssclient.c [attached]

took a look also at striplt() in libcitadel/lib/tools.c ..

kind of a trim() function.. didn't see anything obvious..

 

Fri Oct 23 2020 03:23:17 PM EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

What is interesting about this Subject problem is that there is no such a problem in the article body. This RSS feed below is from the biggest Russian news agency and so the article bodies are in Russian.

Furthermore, the problem occurs only during article reception processing. Once the article is stored, from then on this problem does not occur. This is evident from the fact that these headers look fine on cit 824 and only the articles being received are affected.

Fri Oct 23 2020 15:10:29 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

This issue of Subject: header is getting more and more interesting. But there are several things involved.

First of all, I upgraded cit 8.24 to 9.29 via easyinstall. Everything went OK and after logging-in, the first thing I noticed is that the RSS feed room, which looked fine under 8.24 now is having problems with incorrect Subject header. Old messages that were received while it was still 8.24 looked correct, but the new ones that were received after cit upgrade did have the Subject problem.

When I looked at the xml source for the RSS feed http://tass.ru/rss/v2.xml all the <title> xxxx </title> fields were present, and those are the fields that would become the Subject header. So... Why such a simple thing as string copy of title to Subject: header would not work?

Furthermore, all the Subject headers that would start with English/Latin words would only show up the English/Latin characters, but everything else would get chopped off, starting with european languages delimiters and non-English/Latin characters.

So... Where is the problem in this case?

I really can't have this issue slide and it's a shame to loose some major news feed because of such an obvious issue. Loosing a Subject header is loosing about the only thing that people read in majority of cases, the title. In today's news, it is painful to read even the article titles.

This problem is present on all the RSS feeds. So, for all practical purposes the RSS feed functionality could be classified as "not very useful", putting it in the mildest of terms.

Fri Oct 23 2020 09:53:56 EDT from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

I'll comment in-line of your text..

Thu Oct 22 2020 17:47:55 EDT from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Well, yesterday I tried to do /usr/local/citadel/ctdlmigrate to xfer database

from 8.24 to 9.29 node.

I don't think that's even possible. If it is, then we need to put some code into the migrate utility to prevent that from being done. Citadel has a lot of code that runs when you upgrade, to convert old data formats into new data formats. For those interested, you can look at the source of serv_upgrade.c to see a fascinating timeline of things we've changed over the years and how it handles all of the conversion behind the scenes without bothering you about it.

What it *can't* do is perform that conversion and export/import at the same time.

The correct procedure is to upgrade to the latest version in-place, and *then* migrate it.

OK. Unfortunately, if I upgrade 824 in-place, then I'll probably get the original problem with Subject: headers getting trimmed down or absent for the RSS rooms.

I wonder if it would be helpful to you to track down that problem if I tell you exactly which RSS feed is guaranteed to have a problem with Subject headers.

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and there are very few headers that begin with Latin words.

I vaguely recall some Subject: header weirdness, such as character sets for the article are defined in the Subject header or something like that.

I wonder if you have a clue of where any manipulations involving the Subject header take place in the source code. Would be interesting to look at that code.

Thanx in advance.

I apologize if this isn't clear in the documentation.

 

 

 

 

 

 

 

 

 

 

 

 

 

Yep, I am seeing exactly the same problem in your room. Subject: header is not showing up, even though the header is not empty as you can see if you look at it from [headers] choice in the article menu. And there is no UTF-8 specified in the Subject header.

Fri Nov 06 2020 00:19:09 EST from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and

there are very few headers that begin with Latin words.

I have subscribed to this feed here at Uncensored in a hidden room called "tass.ru"

Can you please go to that room and tell me whether you are seeing the same problems here that you are on your own system, and if so, point them out?
Unfortunately I cannot read Russian.

Btw, I'd like to show you the diffs between 929 and 824 in serv_rssclient.c

It looks like it is pretty clear where this problem is, when/if you compare the 824 and 929 versions of serv_rssclient.c

It seems to me that the issue is missing the call to StrBufRFC2047encode().

Two lines, just before CM_SetAsFieldSB(), are missing in 929

serv_rssclient_824.c Lines 355 - 358

    StrBufTrim(Encoded);

    StrBufRFC2047encode(&QPEncoded, Encoded);

    CM_SetAsFieldSB(&SaveMsg->Msg, eMsgSubject, &QPEncoded);

Compare it to:
serv_rssclient_929.c (around line 202 CM_SetField()

Lines 200 - 204

	else if (!strcasecmp(el, "title")) {			// item subject (rss and atom)
		if ((r->msg != NULL) && (CM_IsEmpty(r->msg, eMsgSubject))) {
			CM_SetField(r->msg, eMsgSubject, ChrPtr(r->CData), StrLength(r->CData));
			striplt(r->msg->cm_fields[eMsgSubject]);
		}
	}

I'll include the c sources in attachments here.

Fri Nov 06 2020 00:19:09 EST from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and

there are very few headers that begin with Latin words.

I have subscribed to this feed here at Uncensored in a hidden room called "tass.ru"

Can you please go to that room and tell me whether you are seeing the same problems here that you are on your own system, and if so, point them out?
Unfortunately I cannot read Russian.

Please forgive me if I am missing something obvious. I  have looked through the docs and forums. I was able to get my citadel install to successfully serve up a certificate issued by Letsencrypt. 

Would someone please tell me how to make webcit redirect plaintext requests for port 80 over to 443 so it may use the https connection? 

Thanks in advance

I have tried to install Citadel through EasyInstall, but got an error message: "multiple definition of `ThreadKey'" and LD failed to link the code.

Easy Install can't install the dependencies automatically in Arch Linux, so I have tried to make it by hand (relating to the citadel.org description) and it seems that the dependencies are here.

So, how to proceed further?

Regards.

Well, I've got it working on my box with Subject headers. But there are 2 more headers that need to be fixed as well.

I did it mostly on intuition and with very lil understanding of code. So...

Fri Nov 06 2020 09:16:14 EST from platonov @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

Btw, I'd like to show you the diffs between 929 and 824 in serv_rssclient.c

It looks like it is pretty clear where this problem is, when/if you compare the 824 and 929 versions of serv_rssclient.c

It seems to me that the issue is missing the call to StrBufRFC2047encode().

Two lines, just before CM_SetAsFieldSB(), are missing in 929

serv_rssclient_824.c Lines 355 - 358

    StrBufTrim(Encoded);

    StrBufRFC2047encode(&QPEncoded, Encoded);

    CM_SetAsFieldSB(&SaveMsg->Msg, eMsgSubject, &QPEncoded);

---

Compare it to:
serv_rssclient_929.c (around line 202 CM_SetField()

Lines 200 - 204

	else if (!strcasecmp(el, "title")) {			// item subject (rss and atom)
		if ((r->msg != NULL) && (CM_IsEmpty(r->msg, eMsgSubject))) {
			CM_SetField(r->msg, eMsgSubject, ChrPtr(r->CData), StrLength(r->CData));
			striplt(r->msg->cm_fields[eMsgSubject]);
		}
	}

I'll include the c sources in attachments here.

Fri Nov 06 2020 00:19:09 EST from IGnatius T Foobar @ Uncensored Subject: Re: Is there a problem of incorrect displaying of Subject: header?

The feed is: http://tass.ru/rss/v2.xml
You'll notice there are no Subject headers in vast majority of cases and

there are very few headers that begin with Latin words.

I have subscribed to this feed here at Uncensored in a hidden room called "tass.ru"

Can you please go to that room and tell me whether you are seeing the same problems here that you are on your own system, and if so, point them out?
Unfortunately I cannot read Russian.

 

The package versions have an init.d script which you can edit to specify the ports for webcit http and https. 

If this is the case, just edit the webcit script in init.d, and make sure it's not starting on port 80, by commenting out a few lines. 

Then, jump down to the nginx configuration (below).

Hopeful you have used easyinstall.  There are two services for webcit that easyinstall puts in /etc/systemd/system

webcit-http.service

and

webcit-https.service

you can mv or rm webcit-http.service out of there. Just run webcit-https

service start webcit-https

If you want to redirect your http traffic to https, install nginx with one simple vhost

(delete the default virtual host in /etc/nginx/site-enabled), and add a file in site-enabled like

my-citadel.com.conf (substitute your own domain).

### NON-SSL

  {
   listen 80;
   listen [::]:80;
   server_name my-citadel.com;
   return 301 https://my-citadel.com$request_uri;
   }

Sat Nov 07 2020 12:01:28 PM EST from Smashbot @ Uncensored Subject: https redirection

Please forgive me if I am missing something obvious. I  have looked through the docs and forums. I was able to get my citadel install to successfully serve up a certificate issued by Letsencrypt. 

Would someone please tell me how to make webcit redirect plaintext requests for port 80 over to 443 so it may use the https connection? 

Thanks in advance

This solution worked perfectly. Thank you.

Sat Nov 07 2020 20:22:18 EST from warbaby @ Uncensored Subject: Re: https redirection

The package versions have an init.d script which you can edit to specify the ports for webcit http and https. 

If this is the case, just edit the webcit script in init.d, and make sure it's not starting on port 80, by commenting out a few lines. 

Then, jump down to the nginx configuration (below).

Hopeful you have used easyinstall.  There are two services for webcit that easyinstall puts in /etc/systemd/system

webcit-http.service

and

webcit-https.service

you can mv or rm webcit-http.service out of there. Just run webcit-https

service start webcit-https

If you want to redirect your http traffic to https, install nginx with one simple vhost

(delete the default virtual host in /etc/nginx/site-enabled), and add a file in site-enabled like

my-citadel.com.conf (substitute your own domain).

### NON-SSL

  {
   listen 80;
   listen [::]:80;
   server_name my-citadel.com;
   return 301 https://my-citadel.com$request_uri;
   }

 

 

Sat Nov 07 2020 12:01:28 PM EST from Smashbot @ Uncensored Subject: https redirection

Please forgive me if I am missing something obvious. I  have looked through the docs and forums. I was able to get my citadel install to successfully serve up a certificate issued by Letsencrypt. 

Would someone please tell me how to make webcit redirect plaintext requests for port 80 over to 443 so it may use the https connection? 

Thanks in advance

 

 I'll comment in-line.

Sun Nov 08 2020 17:06:35 EST from IGnatius T Foobar @ Uncensored Subject: Re: Problem with incorrect displaying of Subject: header fixed

Interesting. I see that you are reporting that the problem is that the current version of the RSS reader does not RFC2047-encode the subject lines, I agree that it needs to do so, and I have confirmed that the code change belongs where you say it belongs ...

The current version of RSS reader does not handle the headers at all. Basically, that code has never been written or adopted from the earlier version. Current version of XML parser and related code is a major rewrite and the XML fields are simply stored into the SMTP headers without any processing or handling.

And the problem is not only the Subject: header. The worst part of the problem is with SMTP From: header, and proper processing/handling of that one is not as simple as one might think. There are several forms of From header. Furthermore, not all the characters or tokens are to be encoded and so on... So, I think you are probably the one who has the most expertise on that code.

but I can't get it to display improperly here.

Strange... May be you need to look at the sources of serv_rssclient.c for current version compared to 8.21. It is pretty obvious that this thing can not just magically work for you unless you have a pretty old version. I'll include both version as an attachment here. Just in case...

Btw, have you seen the fix I have sent to warbaby? I left some older code commented out just to make it easier to see the differences. It only handles the Subject: issue and From header handling has not been implemented. It would probably take you not so long to wire that code in.

What does your reading environment look like? Linux or Windows?

I am reading it via Webcit under Windows. Warbaby is also having the same problem.

Text client, web, or IMAP?

Actually, I am also using Thunderbird IMAP and everything looks correct.

Can you send a marked-up screenshot?

 Not sure what exactly do you want to see. May be warbaby would understand you better...

Well, there was a mail delivery problem with my message to warbaby that included the fix source code. So, I'll repost the attachment here to make sure you see the fix and/or try it for yourself and do a code review. I am not sure if there is some kind of problem with this post, in which case you can just remove it. I just want to make sure we're making progress on this and get the fix into the source tree.

[See serv_rssclient.c with fix in attachment]

Sun Nov 08 2020 17:06:35 EST from IGnatius T Foobar @ Uncensored Subject: Re: Problem with incorrect displaying of Subject: header fixed

Interesting. I see that you are reporting that the problem is that the current version of the RSS reader does not RFC2047-encode the subject lines, I agree that it needs to do so, and I have confirmed that the code change belongs where you say it belongs ... but I can't get it to display improperly here.

What does your reading environment look like? Linux or Windows? Text client, web, or IMAP? Can you send a marked-up screenshot?

Mon Nov 09 2020 09:54:19 EST from IGnatius T Foobar @ Uncensored Subject: Re: Problem with incorrect displaying of Subject: header fixed

Sure thing. I understand the problem but I want to make sure I'm getting the fix correct. I have code up in the editor but if yours works I'll review that and use it instead.

You're correct in observing that serv_rssclient.c is a substantial rewrite.
A few years ago we made a decision to simplify all of the pollers by letting libcurl do all the work. This allowed us to remove thousands of lines of code and two external dependencies. RFC2047-encoding of header fields is definitely something that was overlooked, so I'm glad you have brought it to our attention.

Great to hear that. But I forgot to mention in my last post that this problem did not exist in textclient and the Subject headers looked good on it. After I wrote my msg I tried to run textclient again just to verify things.

And boom! Now all the rss messages that were received after I installed my fix no longer have Subject displayed correctly. But old messages, before the fix, looked good.

Not sure if textclient has been upgraded since this major rewrite related to libcurl.

As far as textclient issue goes, even before this fix you could not write a message in non-ASCII. If you switch the keyboard language and then start typing, it is all displayed in ASCII. So...

Excuse me, by chance any news about Webcit-ng and CalDAV?

There are several things broken in current Webcit. Prominently, for instance, the Calendar room.
The day view always graphically displays the hour ranges wrongly.
The "Check attendee availability" button only adds a certain amount of hours to the starting hour of the event, even if user is actually free.

Also, either I don't know how to use it, or the "Chat" room has never been functional
(testing in my local Citadel installation using latest version).

Well, this is not about the Subject header directly, but it deals with the same source file, and once we are at it, why not clean up some other things we can...

There is one not very pleasant issue with rssclient requiring a single source line change, that I'd like to ask to take care of.

The issue is: when rssclient is having a problem with some feed, it logs the message, saying:

• "rssclient: failed to load feed: ..."

But it does not tell you WHAT is the feed's URI and which room is it in, which is a shame - to have an error that you can not take care of simply because you don't have ANY information on where and what it is.

Good logging practice is to precisely and sufficiently completely identify the key information. Simply saying: "rssclient: failed to load feed" is pretty meaningless and is a cause of unnecessary waste of time and energy.

So, I would really appreciate if we make this single source line change:

File serv_rssclient.c, Line 406:

    if (res != CURLE_OK) {
        syslog(LOG_WARNING, "rssclient: failed to load feed: %s, room %s, %s",
            url->url, url->rooms->room, curl_easy_strerror(res));
    }

Mon Nov 09 2020 09:54:19 EST from IGnatius T Foobar @ Uncensored Subject: Re: Problem with incorrect displaying of Subject: header fixed

Sure thing. I understand the problem but I want to make sure I'm getting the fix correct. I have code up in the editor but if yours works I'll review that and use it instead.

You're correct in observing that serv_rssclient.c is a substantial rewrite.
A few years ago we made a decision to simplify all of the pollers by letting libcurl do all the work. This allowed us to remove thousands of lines of code and two external dependencies. RFC2047-encoding of header fields is definitely something that was overlooked, so I'm glad you have brought it to our attention.

Is there documentation on how to set up an RSS feed as a Room on Citadel?