Change media_type of JSONResponse to "application/json; charset=utf-8"

[muxacarin]

I am using fastapi (which using starlette), and when I return data with Korean character (and possibly none-alphabet character), encoding is wrong.

It is double-encoded: it was encoded in UTF-8, then the resulting bytes were interpreted as ISO-8859-1 (or a similar encoding) and encoded in UTF-8 again. (answer from GitHub copilot)

example: "피노키오랩" expected to be encoded to %ED%94%BC%EB%85%B8%ED%82%A4%EC%98%A4%EB%9E%A9 (in url encoded form), but resulted in %C3%AD%C2%94%C2%BC%C3%AB%C2%85%C2%B8%C3%AD%C2%82%C2%A4%C3%AC%C2%98%C2%A4%C3%AB%C2%9E%C2%A9

And when I change media_type of JSONResponse to "application/json; charset=utf-8", this issue solved

[Kludex]

Can you share a MRE with an application and client code?

[muxacarin]

sorry, i do not know what MRE is. but i have sample code

server code

pip install fastapi

import json
from fastapi import FastAPI, Response

app = FastAPI()


@app.get("/without_encoding")
async def without_encoding():
    return {"key": "안녕, 세상!"}


@app.get("/with_encoding")
async def with_encoding():
    return Response(
        content=json.dumps({"key": "안녕, 세상!"}),
    )

client code (dart/flutter)

import 'dart:convert';

import 'package:flutter/material.dart';
import 'package:http/http.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
        useMaterial3: true,
      ),
      home: const MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key, required this.title});

  final String title;

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  final String host =
      '192.168.4.200:8001'; //change this to your ip address and port
  String _withoutEncoding = 'fetch first';
  String _withEncoding = 'fetch first';

  void _fetch() async {
    setState(() {
      _withoutEncoding = 'fetching...';
      _withEncoding = 'fetching...';
    });
    var withoutEncoding = await get(Uri.http(host, '/without_encoding'));

    var withEncoding = await get(Uri.http(host, '/with_encoding'));
    setState(() {
      _withoutEncoding = json.decode(withoutEncoding.body)['key'];
      _withEncoding = json.decode(withEncoding.body)['key'];
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        backgroundColor: Theme.of(context).colorScheme.inversePrimary,
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            const Text('without_encoding'),
            Text(_withoutEncoding),
            const Text('with_encoding'),
            Text(_withEncoding),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _fetch,
        child: const Icon(Icons.download),
      ),
    );
  }
}

change media_type of JSONResponse to application/json; charset=utf-8 and compare

before change:

after change:

[Kludex]

I did an investigation about this.

The charset=utf-8 is redundant, because application/json can only be charset=utf-8.

[Kludex]

The client libraries should adapt.

[FinemechanicPub]

The charset=utf-8 is redundant, because application/json can only be charset=utf-8.

Where does this come from?

RFC 4329 tells that charset is a valid option and insists on its use in the article 3.2

And Webhint complaints about the absence of charset option.