ktor client sets "Accept-Charset: UTF-8", which results in 403 from EWS

[baumanno]

Hello! Thanks for maintaining this nifty piece of software.

TL;DR: ktor sets the Accept-Charset header, which causes the Exchange Web Services instance I'm interfacing with to refuse to respond. Any chance of not setting that header?

For several months now, I've been getting a notification that a subscribed calendar was producing 403 Forbidden errors. Today, I noticed the calendar hadn't updated in quite a while, and dug deeper.
The org where my calendar is hosted uses Microsoft Exchange Web Services, unfortunately I don't know which version.

This is the stack trace from ICSx5 upon newly subscribing to the calendar in question:

HTTP 403 Forbidden

java.io.IOException: HTTP 403 Forbidden
	at at.bitfire.icsdroid.CalendarFetcher.fetchNetwork$icsx5_92_2_4_3_standardRelease(SourceFile:181)
	at at.bitfire.icsdroid.CalendarFetcher$fetchNetwork$1.invokeSuspend(Unknown Source:11)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:34)
	at kotlinx.coroutines.DispatchedTask.run(SourceFile:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(SourceFile:124)
	at kotlinx.coroutines.scheduling.TaskImpl.run(SourceFile:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(SourceFile:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(SourceFile:820)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(SourceFile:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(SourceFile:704)

Using PCAPdroid, I captured the HTTPS session for the feed validation during subscription. This resulted in the following headers:

GET /owa/calendar/<REDACTED>/reachcalendar.ics HTTP/1.1
Accept: text/calendar, */*;q=0.9
User-Agent: ICSx5/2.4.0 (ical4j/3.2.19 okhttp/5.1.0 Android/11)
Accept-Charset: UTF-8
Host: mail.<REDACTED>
Connection: Keep-Alive
Accept-Encoding: gzip

After playing around with various user-agent strings I found in other discussions, I tried running plain curl within termux, and it worked; so the UA wasn't the issue. I could also access the .ics from various browsers in private mode. What didn't work was setting the above headers in a curl command, so I successively removed headers until it worked, narrowing it down to the Accept-Charset: UTF-8 header. This apparently bothers the Exchange server so much it refuses to cooperate.

It seems that ktor sets the header Accept-Charset: UTF-8 by default, if not overridden otherwise.

Is there any chance of not setting the Accept-Charset header at all? According to the note in RFC 9110 §12.5.2 it is deprecated.

Thanks for considering this!

[ArnyminerZ]

After further investigation, yes, the Accept-Charset is technically deprecated/not-recommended. It should be replaced by either the XML header:

<?xml version="1.0" encoding="utf-8" ?>

Or the Content-Type header, such as:

Content-Type: text/vcard; charset=utf-8

So I guess that if before, with the old http client we were not sending it, and we had no complaints. And now we are sending it, and we have complaints, we can safely disable the header and call it a day

[baumanno]

Thanks, that sounds encouraging! It is indeed the case that formerly, the calendar in question was updating just fine, and at least since 2.4.0 (can't seem to go back later than that on FDroid), this glitch happens.