fix twisted multi-value header handling

[bentsku]

Motivation

Follow-up from #34, we had the wrong behavior for multivalue header handling. We were joining the multivalues in one line, where we need a single line per header value, with duplicated keys.

This is due to how Twisted changed how it returns headers: previous versions were returning key, value from iterating over headers, where the same key could appear multiple times (similar to how Werkzeug Headers works).

The new implementation now returns a dictionary of lists, so we need to keep the same logic as before by creating a line for each value in the list.

Technically, this is correct to concatenate them if they can be concatenated, but some clients expects them differently (and for LocalStack usage, AWS does not concatenate them).

See also the following SO response: https://stackoverflow.com/a/3097052/21356283
And the RFC 2616:

Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded.

Changes

• add new regression tests for Multivalue header handling
• fix the logic to add a new line in the raw HTTP response for each multivalue header entry

[thrau]

thanks for verifying this behavior. when i wrote this, i checked the RFCs related to header handling and multi-value headers are typically expected to be returned as a CSV list, but good to know that twisted handles things differently internally (and makes sense).
should have written a test for it though!

changes LGTM!

[bentsku]

when i wrote this, i checked the RFCs related to header handling and multi-value headers are typically expected to be returned as a CSV list, but good to know that twisted handles things differently internally (and makes sense).

I mean, it's technically correct, it's just that AWS does not concatenate them, so for LocalStack it is an issue. There's also the Set-Cookie header which has sometimes special handling, because it cannot be returned like that as it holds comma in the values. So easier to always return them on a single line I guess?

Thanks for the merge, I'll do a release later today 👍