Problem with parsing caldav input. different formatting of same content -> different result

[daald]

Using tomsquest/docker-radicale docker image (storage cache version: "b'radicale=3.3.1;vobject=0.9.9;'"), I have a problem importing a vcalendar element which I received from a google calendar. The description of this event contains unicode characters, spaces and non-breakable spaces. Namely a line of more than 80 characters of whitespace seems to irritate radicale. Here is the original (anonymized) calendar object:

BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
DTSTART:20250929T160000Z
DTEND:20250929T190000Z
UID:xxxxxxxxxxxxxxxxxxxxxxxxxx@google.com
DESCRIPTION:<span><span><span><span><span>xxxxx xäxx<br><br>xxxxx xxxxx xxx
 x</span></span></span> xxxxxx xxxxxx\, xxxxxx xxxxxxxxxx\, xxxx xxxxxx xxx 
 xxxxx xxxxx xxx xxxxxxxxxxxxéxx xxx.<span><span><span> <span>🥳</span></xxx
 x></span><span> 🎉 </span><span>🎁</span>                                  
                                                </span><br><span>           
                                                                      <span>
 <span><br>xxxxx xxxxxx xxxx xxx xx xxxxxxxx\, xx. xxxxxxx xx xxxx xx.<br><x
 x>xxx xxxxxx xxx xxxxxxxx xxxxxxxxxxx! <span><span><span>🍻</span></span></
 xxxx><br><br>xxxxx xxüxxxxx</span></span></span></span></span><br><span><xx
 xx><span><span><span>xxxxxxx &xxx\; xxxxx </span></span></span></span></xxx
 x>\n\nÜxxx xxxxxx xxxx xxxxxxxxxx: xxxxx://xxxx.xxxxxx.xxx/xxx-xxxx-xxx\nxx
 xx xxxxxxxxxxx: (xx) +xx xx xxx xx xx xxx: xxxxxxxxx#\nxxxxxxx xxxxxxxxxxxx
 xx: xxxxx://xxx.xxxx/xxx-xxxx-xxx?xxx=xxxxxxxxxxxxx&xx=x\n\nxxxxxxx xxxxxxx
 xxxxxx xx xxxx: xxxxx://xxxxxxx.xxxxxx.xxx/x/xxxxx/xxxxxx/xxxxxxx
SUMMARY:xxxxxxxxxxxxxxxx 🥳 🎉 🎁  
END:VEVENT
END:VCALENDAR

Note that the whitespace is not only ascii 32 (hex 20). In hexdump, it looks like this:

00000000  c2 a0 c2 a0 c2 a0 20 c2  a0 20 c2 a0 20 c2 a0 20  |...... .. .. .. |
00000010  c2 a0 20 c2 a0 20 c2 a0  20 c2 a0 20 c2 a0 20 c2  |.. .. .. .. .. .|
00000020  a0 20 c2 a0 20 c2 a0 20  c2 a0 20 c2 a0 20 c2 a0  |. .. .. .. .. ..|
00000030  20 c2 a0 20 0a 20 c2 a0  20 c2 a0 20 c2 a0 20 c2  | .. . .. .. .. .|
00000040  a0 20 c2 a0 20 c2 a0 20  c2 a0 20 c2 a0 20 c2 a0  |. .. .. .. .. ..|
00000050  20 c2 a0 20 c2 a0 20 c2  a0 20 c2 a0 20 c2 a0 20  | .. .. .. .. .. |
00000060  c2 a0 20 c2 a0 20 c2 a0  20 c2 a0 20 c2 a0 20 c2  |.. .. .. .. .. .|
00000070  a0 20 c2 a0 20 c2 a0 20  c2 a0 c2 a0 20           |. .. .. .... |

(don't blame me about the contents. it was written by another person without any bad intention. it was an invitation about a social event)
see attached files for the (anonymized) originals

The above object imported well in radicale using the following command:

$ curl -u user:y -vX PUT http://192.168.144.3:5232/user/personal/test-abc --data-binary @src/cal-issue-orig.ics 
*   Trying 192.168.144.3:5232...
* Connected to 192.168.144.3 (192.168.144.3) port 5232
* Server auth using Basic with user 'user'
> PUT /user/personal/test-abc HTTP/1.1
> Host: 192.168.144.3:5232
> Authorization: Basic dXNlcjp5
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Length: 1360
> Content-Type: application/x-www-form-urlencoded
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 201 Created
< Date: Sat, 20 Dec 2025 19:20:41 GMT
< Server: WSGIServer/0.2 CPython/3.12.12
< ETag: "0ee423a53176782c8fd0e6c8af23aef209a533fd106eb4c89487754367d4bcad"
< Content-Length: 0
< 
* Closing connection

BUT: after reading the above using a custom program using vobject 0.9.6.1 and 0.9.9:

import vobject  # tested with v0.9.6.1 and v0.9.9
import unittest


class TestStringMethods(unittest.TestCase):
    def test_orig_google(self):
        with open("cal-issue-orig.ics", "r") as vo_stream:
            cal = vobject.readOne(vo_stream)
        cal.prettyPrint()
        self.assertEqual(cal.name, "VCALENDAR")
        eve = cal.contents["vevent"][0]
        description1 = eve.contents["description"]
        icalstr = cal.serialize()
        print(f"--------------------\n{icalstr}\n--------------------\n")
        with open("cal-issue-reformatted.ics", "w") as vo_stream:
            vo_stream.write(icalstr)

        cal = vobject.readOne(icalstr)
        cal.prettyPrint()

        eve = cal.contents["vevent"][0]
        description2 = eve.contents["description"]
        self.assertEqual(description1, description2)
        self.assertEqual(description1[0].value, description2[0].value)

if __name__ == "__main__":
    unittest.main()

I get the following text file:

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//PYVOBJECT//NONSGML Version 1//EN
BEGIN:VEVENT
UID:xxxxxxxxxxxxxxxxxxxxxxxxxx@google.com
DTSTART:20250929T160000Z
DTEND:20250929T190000Z
DESCRIPTION:<span><span><span><span><span>xxxxx xäxx<br><br>xxxxx xxxxx xx
 xx</span></span></span> xxxxxx xxxxxx\, xxxxxx xxxxxxxxxx\, xxxx xxxxxx xx
 x xxxxx xxxxx xxx xxxxxxxxxxxxéxx xxx.<span><span><span> <span>🥳</spa
 n></xxxx></span><span> 🎉 </span><span>🎁</span>              
                                                  
                   </span><br><span>                    
                                                  
            <span><span><br>xxxxx xxxxxx xxxx xxx xx xxxxxxxx\, xx. x
 xxxxxx xx xxxx xx.<br><xx>xxx xxxxxx xxx xxxxxxxx xxxxxxxxxxx! <span><span
 ><span>🍻</span></span></xxxx><br><br>xxxxx xxüxxxxx</span></span></spa
 n></span></span><br><span><xxxx><span><span><span>xxxxxxx &xxx\; xxxxx </
 span></span></span></span></xxxx>\n\nÜxxx xxxxxx xxxx xxxxxxxxxx: xxxxx:/
 /xxxx.xxxxxx.xxx/xxx-xxxx-xxx\nxxxx xxxxxxxxxxx: (xx) +xx xx xxx xx xx xxx
 : xxxxxxxxx#\nxxxxxxx xxxxxxxxxxxxxx: xxxxx://xxx.xxxx/xxx-xxxx-xxx?xxx=xx
 xxxxxxxxxxx&xx=x\n\nxxxxxxx xxxxxxxxxxxxx xx xxxx: xxxxx://xxxxxxx.xxxxxx.
 xxx/x/xxxxx/xxxxxx/xxxxxxx
DTSTAMP:20251220T193004Z
SUMMARY:xxxxxxxxxxxxxxxx 🥳 🎉 🎁  
END:VEVENT
END:VCALENDAR

Note that the wrapped lines are slightly shorter and thus the whitespace is wrapped to full-whitespace lines.

This time, radicale doesn't import the file:

$ curl -u user:y -vX PUT http://192.168.144.3:5232/user/personal/test-abc5 --data-binary @src/cal-issue-reformatted.ics
*   Trying 192.168.144.3:5232...
* Connected to 192.168.144.3 (192.168.144.3) port 5232
* Server auth using Basic with user 'user'
> PUT /user/personal/test-abc5 HTTP/1.1
> Host: 192.168.144.3:5232
> Authorization: Basic dXNlcjp5
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Length: 1460
> Content-Type: application/x-www-form-urlencoded
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
< Date: Sat, 20 Dec 2025 19:47:44 GMT
< Server: WSGIServer/0.2 CPython/3.12.12
< Content-Type: text/plain; charset=utf-8
< Content-Length: 11
< 
* Closing connection
Bad Request

and the radicale log:

radicale               | [2025-12-20 19:47:44 +0000] [7/Thread-111 (process_request_thread)] [WARNING] Bad PUT request on '/user/personal/test-abc5' (read_components): At line 12: Failed to parse line:                   </span><br><span>                    
radicale               | [2025-12-20 19:47:44 +0000] [7/Thread-111 (process_request_thread)] [DEBUG] Bad PUT request content: suppressed by config/option [logging] bad_put_request_content
radicale               | [2025-12-20 19:47:44 +0000] [7/Thread-111 (process_request_thread)] [DEBUG] Response content:
radicale               | Bad Request
radicale               | [2025-12-20 19:47:44 +0000] [7/Thread-111 (process_request_thread)] [INFO] PUT response status for '/user/personal/test-abc5' in 0.003 seconds plain 11 bytes: 400 Bad Request

As you can see in the conversion and test code above, the input and output description are 100% equal, so the second file should be imported in the same way as the first one did. I also tested it with thunderbird and it worked.

You also can see that I check if vobject itself is able to handle what it writes out, and it does (I know, radicale uses the same library). So I came to the conclusion the this must be a problem outside of vobject

The issues reminds me slightly to the smtp-smuggling issue in e-mails, although I didn't find any security-relevant problem here yet

[daald]

bad-content-demo_2.py

cal-issue-orig.ics.txt

cal-issue-reformatted.ics.txt

[pbiering]

Interesting..the orig one is not passing validator but accepted, the reformatted one is passing the validator but rejected here

  File "/opt/src/Radicale-pbiering/radicale/app/put.py", line 199, in do_PUT
    vobject_items = radicale_item.read_components(content or "")
  File "/opt/src/Radicale-pbiering/radicale/item/__init__.py", line 58, in read_components
    return list(vobject.readComponents(s, allowQP=True))
  File "/usr/lib/python3.14/site-packages/vobject/base.py", line 1166, in readComponents
    vline = textLineToContentLine(line, n)
  File "/usr/lib/python3.14/site-packages/vobject/base.py", line 984, in textLineToContentLine
    return ContentLine(*parseLine(text, n), **{"encoded": True, "lineNumber": n})
                        ~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.14/site-packages/vobject/base.py", line 864, in parseLine
    raise ParseError("Failed to parse line: {0!s}".format(line), lineNumber)
vobject.base.ParseError: At line 12: Failed to parse line:                   </span><br><span>                    

Need to implement some extended debug code to see what really happens before content is feeded into vobject.

[daald]

which validator? I saw that google breaks the lines after each 75 unicode characters, while vobject breaks them after 76 bytes. I did a short check if vobject breaks in the middle of a unicode (utf-8) sequences, producing invalid utf-8, but this wasn't the case at least in this example. just to let you know.

[pbiering]

which validator?

https://github.com/Kozea/Radicale/wiki/01-%E2%80%90-Reporting-Issues

I'm still working on additional debug code to trace down the issue.

[daald]

no problem, I don't need it. I found the issue during a test and found it's important.
my bad: the missing PRODID and DTSTAMP properties were removed be me, to make the file smaller and reduce it to the relevant part, obviously too much. and during this process, my linux-based editor also replaced all CRLF by LF.
here is a fixed orig file which still causes the radicale problem:

cal-issue-orig_win.ics.txt

[pbiering]

H here is a fixed orig file which still causes the radicale problem:

I can successfully import this file using PUT request, which problem do you have on your side? Validator only reports a warning related to line length

sha256sum cal-issue-orig_win.ics.txt 
e1d0b325044a7c6cd9eaed73177465fd376f490945ef4e2edc8403cbba6ad78d  cal-issue-orig_win.ics.txt

[pbiering]

I have no clue why cal-issue-reformatted.ics.txt is rejected in line #12, btw: the unicode extends string from 58 chars to 77 bytes (incl CRLF):

12   58/ 77: 20 20 c2 a0 20 c2 a0 20  c2 a0 20 c2 a0 20 c2 a0  20 c2 a0 20 c2 a0 20 c2  a0 c2 a0 20 3c 2f 73 70  61 6e 3e 3c 62 72 3e 3c  73 70 61 6e 3e 20 c2 a0  20 c2 a0 20 c2 a0 20 c2  a0 20 c2 a0 20 c2 a0 20  c2 a0 20 c2 a0 20 c2 a0  20 c2 a0 0d 0a |  .. .. .. .. .. .. .. .... </span><br><span> .. .. .. .. .. .. .. .. .. ....|..u..u..u..u..u..u..u..u.u....................u..u..u..u..u..u..u..u..u..u.CL|

-> vobject developers have to investigate if the content of the file should be still considered as valid.

Note: the output is from a local extension, have to review and create a PR then.

[daald]

H here is a fixed orig file which still causes the radicale problem:

I can successfully import this file using PUT request, which problem do you have on your side? Validator only reports a warning related to line length

Same here.
cal-issue-orig.ics.txt imports.
cal-issue-orig_win.ics.txt imports
cal-issue-reformatted.ics.txt (generated from cal-issue-orig.ics.txt by .py script) doesn't import
cal-issue-reformatted_win.ics.txt (generated from cal-issue-orig_win.ics.txt by .py script) doesn't import too

[pbiering]

at least I found now the trigger

--- a/radicale/item/__init__.py
+++ b/radicale/item/__init__.py
@@ -55,7 +55,7 @@ def read_components(s: str) -> List[vobject.base.Component]:
     #  * 0x0A Line Feed
     #  * 0x0D Carriage Return
     s = re.sub(r'[\x00-\x08\x0B\x0C\x0E-\x1F]', '', s)
-    return list(vobject.readComponents(s, allowQP=True))
+    return list(vobject.readComponents(s))

further analysis is needed

[pbiering]

filed ticket with demo code to vobject