JSON: support custom encode/decode without intermediate objects

[kevmoo]

The current support for custom encoding (toEncodable param and supporting toJson on classes) and decoding (reviver) deal with JSON literal types (String, num, Map, List, etc) requiring intermediate representations be created for all complex objects.

Propose to add three members (see below) to dart:convert and then update the JsonCodec, JsonEncoder, JsonDecoder classes to support then as alternatives to toEncodable and reviver.

Note: I don't expect devs to hand-write implementations of these types. Instead, code generators – such as json_serializable – could be updated to target the new API.

typedef WriteJson = bool Function(Object source, JsonWriter writer);

// Inspired by SDK code
// https://github.com/dart-lang/sdk/blob/e7bd3edc9f/sdk/lib/convert/json.dart#L513
abstract class JsonWriter {
  void writeObject(Object object);

  void startMap();
  void writeKeyValue(String key, Object value);
  void endMap();

  void startList();
  void writeListValue(Object value);
  void endList();
}

// Note: already exists in the SDK sources
// https://github.com/dart-lang/sdk/blob/e7bd3edc/runtime/lib/convert_patch.dart#L79-L101
abstract class JsonReader<T> {
  void handleString(String value);
  void handleNumber(num value);
  void handleBool(bool value);
  void handleNull();
  void objectStart();
  void propertyName();
  void propertyValue();
  void objectEnd();
  void arrayStart();
  void arrayElement();
  void arrayEnd();
  T get result;
}

Related issues:

• better reviver API for JsonDecoder #29750
• JSON.encode w/ Maps with non-string keys fails badly. #32476

See https://github.com/kevmoo/json_alt

[kevmoo]

CC @rakudrama @lrhn – I've been able to get much faster encode/decode of custom types vs the VM – imagine if someone who knew what they were doing tried!

[zoechi]

cc @davidmorgan would that work well with built_value as well?

[davidmorgan]

Yeah, this could be a significant improvement for the VM.

IIUC for web it'd have to be a parser implemented in js instead of the native one, curious if the slowdown makes it worth it overall.

built_value by default uses lists instead of maps in the serialization format, on the assumption that it's cheaper (CPU/RAM) to allocate lists than maps. It's clearly better to allocate neither, if possible...

[mraleph]

And if we combine this with dart:mirrors and we get a nice generic JSON deserialization support that does not require build step:

https://mrale.ph/blog/2017/01/08/the-fear-of-dart-mirrors.html

[kevmoo]

And if we combine this with dart:mirrors and we get a nice generic JSON deserialization support that does not require build step:

Sure. It's great for both scenarios. All good.

[kevmoo]

Thoughts @lrhn ?

[lrhn]

Definitely possible. Likely something we can add in a package. I don't expect to change the platform JSON parser now (too many things depending on it). Since the platform JSON parser is written in Dart, there is no reason to think this can't be as efficient.

We might get some code sharing by adding it to the SDK libraries.

[kevmoo]

Since the platform JSON parser is written in Dart, there is no reason to think this can't be as efficient.

There are a number of patches that separate the JS and VM implementations. There is also a one-off implementation of parseDouble. It'd be nice if we could build on top of this...and let folks avoid adding a package import.