MessagePack for C# v4: Round7 - #2294

In this round, we finished building out the Source Generator and implemented all remaining features.
While the code polishing and final review are not complete yet, we consider the implementation of all planned features to be done.

First of all: v4 achieves extremely fast serialization/deserialization speed even in map mode!
It is even faster than other libraries' array mode.

Serialize

Deserialize

You may notice that v3's performance looks quite bad here. For the sake of benchmark fairness, we use Jil's Stackoverflow Answer as the large POCO, and all of its primitive fields are nullable. v3's Source Generator handles nullables poorly, which causes this performance degradation. Since this is not a particularly common scenario, we may switch to a benchmark with the nullables removed.

As for Nerdbank.MessagePack, it has been updated to the latest v1.3.85, which is why its array performance has improved significantly.

New Features

We added support for the new .NET 11 types and C# 15 unions.

[MessagePackObject]
[UnionTag<Cat>(0)]
[UnionTag<Dog>(1)]
[UnionTag<Bird>(2)]
public union Pet(Cat, Dog, Bird);

We also support the non-boxing access pattern, so when the case types are structs, this finally delivers the long-requested unions over value types.

Note that as a breaking change to the annotations, MessagePack.UnionAttribute is renamed to MessagePack.UnionTag. The reason is that it collides with System.Runtime.CompilerServices.UnionAttribute, which makes it awkward to use.

We intend to take the utmost care with annotation compatibility, but there is one more break: the Type of MessagePackFormatterAttribute is now a factoryType instead of a formatterType, which is not compatible. Due to the architectural change, formatters can no longer be instantiated directly, so this change is unavoidable.

In addition, the generator-related attributes (MessagePackAssumedFormattableAttribute, MessagePackKnownFormatterAttribute) are retired. In their place, MessagePackSerializableAttribute and MessagePackKnownTypeAttribute are introduced.

// The Source Generator implements MessagePackFormatterFactory and registers it to SourceGeneratedFormatterFactory automatically
[MessagePackSerializable<SandboxPerson[]>]
public partial class RootFactory;

By passing a KeyNamingPolicy at declaration time, you can now apply name conversion in the automatic map mode.

[MessagePackObject(KeyNamingPolicy.SnakeCaseLower)]
public class SandboxPerson
{
    public required int Id { get; init; } // id
    public string? Name { get; set; } // name
    public string[]? MyTags { get; set; } // my_tags
}

We also added the options ValidateRequiredMembers and ValidateNullableAnnotations, which validate required and nullable respectively. The defaults are true for required and false for nullable. Because of the nature of the language, nullable validation can never be 100% complete, so we judged that opting in deliberately, with an understanding of its limitations, is the better default.

Circular references and dedup are now supported. Wrapping every formatter has real problems (versioning resilience, efficiency, and the implementation bleeding into every formatter), so we chose a per-type opt-in approach for the types you want to support. I believe this is a well-balanced approach that poses no problems in practice.

[MessagePackObject(AllowCircularReferences = true)]
public class Node
{
    [Key(0)]
    public Node? Parent { get; set; }
    [Key(1)]
    public List<Node> Children { get; set; } = [];
    [Key(2)]
    public string? Name { get; set; }
}

As the equivalent of protobuf's Unknown Fields, we introduce MessagePackUnknownMembers.

[MessagePackObject]
public struct Foo
{
    [Key(0)]
    public int Id { get; set; }
    [Key(1)]
    public string? Name { get; set; }

    public MessagePackUnknownMembers? Extra { get; set; }
}

We added IMessagePackSurrogate<TTarget, TSurrogate>, which makes external types without annotations serializable without writing a custom formatter. Once defined, the Source Generator automatically adds it to the standard runtime factory as well, so it automatically becomes serializable in AOT environments too. Note that since it uses static abstract members, it targets modern .NET only.

[MessagePackObject]
public struct UserIdSurrogate : IMessagePackSurrogate<UserId, UserIdSurrogate>
{
    [Key(0)]
    public int Value { get; }

    [Key(1)]
    public string Realm { get; }

    public UserIdSurrogate(int value, string realm)
    {
        this.Value = value;
        this.Realm = realm;
    }

    public static UserIdSurrogate ToSurrogate(UserId value)
    {
        return new UserIdSurrogate(value.Value, value.Realm);
    }

    public static UserId FromSurrogate(UserIdSurrogate surrogate)
    {
        return new UserId(surrogate.Value, surrogate.Realm);
    }
}