Serializers: Add System.Text.Json benchmarks for comparison. #701

NickCraver · 2019-07-25T15:58:55Z

This is an attempt to add System.Text.Json's JSON serialization to the comparison microbenchmarks. We're trying to get an apples to apples comparison of serializers we can use in .NET Core for ASP.NET Core and async streams. We figured adding them here for everyone is the best course of action.

Note: this test series fails with one of the generic type payloads. There is an exception in the serialization of Dictionary<int, string> isn't a valid thing for System.Text.Json to handle as of 3.0 preview 7. The rest seems to work well and...I hope I added these correctly

Here's an error trace from the failing tests on System.Text.Json with the CollectionsOfPrimitives type specifically:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.NotSupportedException: The collection type 'System.Collections.Generic.Dictionary`2[System.Int32,System.String]' on 'MicroBenchmarks.Serializers.CollectionsOfPrimitives.Dictionary' is not supported.
   at System.Text.Json.JsonClassInfo.GetElementType(Type propertyType, Type parentType, MemberInfo memberInfo, JsonSerializerOptions options)
   at System.Text.Json.JsonClassInfo.CreateProperty(Type declaredPropertyType, Type runtimePropertyType, PropertyInfo propertyInfo, Type parentClassType, JsonSerializerOptions options)
   at System.Text.Json.JsonClassInfo.AddProperty(Type propertyType, PropertyInfo propertyInfo, Type classType, JsonSerializerOptions options)
   at System.Text.Json.JsonClassInfo..ctor(Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializerOptions.GetOrAddClass(Type classType)
   at System.Text.Json.WriteStackFrame.Initialize(Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.WriteCore(Utf8JsonWriter writer, PooledByteBufferWriter output, Object value, Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.WriteCore(PooledByteBufferWriter output, Object value, Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.WriteCoreString(Object value, Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Serialize[TValue](TValue value, JsonSerializerOptions options)
   at MicroBenchmarks.Serializers.Json_ToString`1.SystemTextJson_() in C:\git\NickCraver\performance\src\benchmarks\micro\Serializers\Json_ToString.cs:line 38
   at BenchmarkDotNet.Autogenerated.Runnable_89.WorkloadActionNoUnroll(Int64 invokeCount) in C:\git\NickCraver\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\4645516e-6e5e-4e7e-8b18-05ede5ec7e66\4645516e-6e5e-4e7e-8b18-05ede5ec7e66.notcs:line 58161
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
   at BenchmarkDotNet.Autogenerated.Runnable_89.Run(IHost host, String benchmarkName) in C:\git\NickCraver\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\4645516e-6e5e-4e7e-8b18-05ede5ec7e66\4645516e-6e5e-4e7e-8b18-05ede5ec7e66.notcs:line 57838
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\git\NickCraver\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\4645516e-6e5e-4e7e-8b18-05ede5ec7e66\4645516e-6e5e-4e7e-8b18-05ede5ec7e66.notcs:line 137
// AfterAll

Open questions:

Is the usage right, or can it be more efficient?
Is the #if NETCOREAPP3_0 correct? (I used Perf.Enumerable.cs as a reference for closest-to-this)
What do we want to do for the breaking Dictionary<int, string> runs?

cc @deanward81 @mgravell @benaadams @stevejgordon

Note: this doesn't full work. There is an exception in the runtime of these tests where the serialization of Dictionary<int, string> isn't a valid thing for System.Text.Json to handle as of 3.0 preview 7. The rest seems to work well and...I hope I added these correctly. We're trying to get an apples to apples comparison of serializers we can use in .NET Core for ASP.NET Core and async streams. We figured adding them here for everyone is the best course of action.

glennc · 2019-07-25T16:01:49Z

@ahsonkhan @steveharter @rynowak

NickCraver · 2019-07-25T16:19:46Z

If anyone's curious - Json_ToStream<T> is the one we're mainly interested in (async to sync comparison for System.Text.Json though), here are the current runs on my machine. Definitely some interesting memory vs. performance trade-offs across the board:

BenchmarkDotNet=v0.11.3.1003-nightly, OS=Windows 10.0.17763.615 (1809/October2018Update/Redstone5)
Intel Core i7-6900K CPU 3.20GHz (Skylake), 1 CPU, 16 logical and 8 physical cores
.NET Core SDK=3.0.100-preview7-012821
  [Host]     : .NET Core 3.0.0-preview7-27912-14 (CoreCLR 4.700.19.32702, CoreFX 4.700.19.36209), 64bit RyuJIT
  Job-VYJPVT : .NET Core 3.0.0-preview7-27912-14 (CoreCLR 4.700.19.32702, CoreFX 4.700.19.36209), 64bit RyuJIT

`Json_ToStream<LoginViewModel>`

Method	Mean	Error	StdDev	Median	Min	Max	Gen 0/1k Op	Gen 1/1k Op	Gen 2/1k Op	Allocated Memory/Op
Jil	2,361.1 ns	222.413 ns	237.979 ns	2,400.0 ns	1,900.0 ns	2,700.0 ns	-	-	-	-
JSON.NET	750.8 ns	12.671 ns	11.853 ns	753.9 ns	728.2 ns	763.5 ns	0.0419	-	-	448 B
Utf8Json	177.4 ns	3.510 ns	3.448 ns	176.1 ns	173.3 ns	184.8 ns	-	-	-	-
DataContractJsonSerializer	1,258.0 ns	24.658 ns	24.218 ns	1,262.6 ns	1,217.3 ns	1,287.3 ns	0.0917	-	-	1008 B
System.Text.Json	781.0 ns	14.737 ns	13.785 ns	775.6 ns	769.1 ns	813.6 ns	0.0256	-	-	296 B

`Json_ToStream<Location>`

Method	Mean	Error	StdDev	Median	Min	Max	Gen 0/1k Op	Gen 1/1k Op	Gen 2/1k Op	Allocated Memory/Op
Jil	4,215.8 ns	1,205.291 ns	1,339.678 ns	3,500.0 ns	3,200.0 ns	7,400.0 ns	-	-	-	96 B
JSON.NET	1,746.8 ns	34.909 ns	38.801 ns	1,722.9 ns	1,708.6 ns	1,802.9 ns	0.0358	-	-	448 B
Utf8Json	412.7 ns	8.169 ns	9.080 ns	408.1 ns	402.3 ns	425.0 ns	-	-	-	-
DataContractJsonSerializer	2,716.1 ns	49.102 ns	45.930 ns	2,696.7 ns	2,674.4 ns	2,808.7 ns	0.0896	-	-	1008 B
System.Text.Json	1,924.2 ns	31.464 ns	29.431 ns	1,922.8 ns	1,869.0 ns	1,971.2 ns	0.0460	-	-	488 B

`Json_ToStream<IndexViewModel>`

Method	Mean	Error	StdDev	Median	Min	Max	Gen 0/1k Op	Gen 1/1k Op	Gen 2/1k Op	Allocated Memory/Op
Jil	51.18 us	1.4612 us	1.5005 us	50.70 us	48.50 us	55.10 us	-	-	-	96 B
JSON.NET	46.68 us	0.9201 us	0.9449 us	46.81 us	45.29 us	48.18 us	0.1915	-	-	2448 B
Utf8Json	33.22 us	0.6490 us	0.6071 us	33.12 us	32.44 us	34.31 us	-	-	-	-
DataContractJsonSerializer	98.09 us	1.9564 us	1.8300 us	98.06 us	95.99 us	101.54 us	-	-	-	2432 B
System.Text.Json	52.63 us	0.6729 us	0.5965 us	52.86 us	51.15 us	53.27 us	0.6333	-	-	7776 B

`Json_ToStream<MyEventsListerViewModel>`

Method	Mean	Error	StdDev	Median	Min	Max	Gen 0/1k Op	Gen 1/1k Op	Gen 2/1k Op	Allocated Memory/Op
Jil	694.0 us	15.19 us	17.50 us	692.5 us	665.1 us	730.3 us	-	-	-	236.34 KB
JSON.NET	923.8 us	18.91 us	21.77 us	913.7 us	895.4 us	965.3 us	22.9008	-	-	258.95 KB
Utf8Json	789.2 us	15.41 us	14.42 us	791.8 us	767.3 us	811.9 us	41.2698	41.2698	41.2698	357.52 KB
DataContractJsonSerializer	932.6 us	17.19 us	16.07 us	938.3 us	903.5 us	963.6 us	-	-	-	23.63 KB
System.Text.Json	872.6 us	13.81 us	12.92 us	864.4 us	858.6 us	892.5 us	28.7770	-	-	320.18 KB

For anyone wanting to run all of these:

dotnet run -c Release -f netcoreapp3.0 --filter MicroBenchmarks.Serializers.Json*

...though there might be a way to combine the results in a single output file that I can't seem to find. Does anyone know if that exists?

adamsitnik · 2019-07-25T16:58:38Z

though there might be a way to combine the results in a single output file that I can't seem to find. Does anyone know if that exists?

--join

ahsonkhan · 2019-07-25T23:24:28Z

We already added similar perf tests here:

performance/src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadJson.cs

Line 14 in 6af2114

namespace System.Text.Json.Serialization.Tests

performance/src/benchmarks/micro/corefx/System.Text.Json/Serializer/WriteJson.cs

Line 14 in 6af2114

namespace System.Text.Json.Serialization.Tests

But, I think it is still valuable to add them to the list of serializers being benchmarked. It makes comparisons easier, when run on-demand. Thanks!
https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/Serializers/README.md

However, last I checked, we don't run these set of benchmarks in our regularly scheduled runs, which is why the S.T.Json specific benchmarks were added separately (which do get run to track regression/etc).

Funnily enough, I had provided a similar benchmark report here:
#456

ahsonkhan

Otherwise, LGTM.

ahsonkhan · 2019-07-25T23:30:30Z

src/benchmarks/micro/Serializers/Json_FromStream.cs

+        {
+            memoryStream.Position = 0;
+
+            using (var writer = new System.Text.Json.Utf8JsonWriter(memoryStream))


nit: Consider moving the global setup next to the rest of the setup.

I realize that would result in more if/defs but maybe that's ok. I am also fine with it as is.

I was 50/50 on this too - was just trying to minimize the #if noise as a default, but happy to do it either way you prefer!

I'll let @adamsitnik, @billwert, @jorive chime in and decide which way they prefer.

ahsonkhan · 2019-07-25T23:35:26Z

What do we want to do for the breaking Dictionary<int, string> runs?

CollectionOfPrimitives contains an unsupported collection (Dictionary<int,string>).

@adamsitnik Is there a way to exclude that test case as part of the GenericTypeArguments set?

@steveharter, @layomia - is there a workaround (maybe via converters), to make this collection type work?

…t-json-benchmarks

NickCraver · 2019-07-25T23:49:32Z

But, I think it is still valuable to add them to the list of serializers being benchmarked. It makes comparisons easier, when run on-demand. Thanks!

Oh yes indeed, good spot. I had #700 touching that file earlier and was waiting for the merge...then forgot :) Added now!

Martin1994 · 2019-09-13T02:52:16Z

What do we want to do for the breaking Dictionary<int, string> runs?

CollectionOfPrimitives contains an unsupported collection (Dictionary<int,string>).

@adamsitnik Is there a way to exclude that test case as part of the GenericTypeArguments set?

@steveharter, @layomia - is there a workaround (maybe via converters), to make this collection type work?

Is there any strong reason we need to test Dictionary<int, string> instead of Dictionary<string, int>? This is actually a very valuable test case for enumerable serializing, so I don't want it to be dropped from System.Test.Json.

steveharter · 2019-10-02T20:45:22Z

Is there any strong reason we need to test Dictionary<int, string> instead of Dictionary<string, int>?

@steveharter, @layomia - is there a workaround (maybe via converters), to make this collection type work?

This is actually a very valuable test case for enumerable serializing, so I don't want it to be dropped from System.Test.Json.

Work-around for 3.0 and 3.1 is to create a converter. This issue (in progress) will provide examples to do that: https://github.com/dotnet/corefx/issues/40120

ahsonkhan previously approved these changes Jul 25, 2019

View reviewed changes

Nick Craver added 2 commits July 25, 2019 19:46

Merge remote-tracking branch 'upstream/master' into craver/system-tex…

195588b

…t-json-benchmarks

Add System.Text.Json to the serializer README list

8c5453b

neuecc mentioned this pull request Jul 27, 2019

How configure MessagePackSerializer.DefaultOptions? MessagePack-CSharp/MessagePack-CSharp#529

Closed

Base automatically changed from master to main March 18, 2021 17:12

LoopedBard3 dismissed ahsonkhan’s stale review via 8c5453b May 12, 2023 16:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Serializers: Add System.Text.Json benchmarks for comparison. #701

Serializers: Add System.Text.Json benchmarks for comparison. #701

NickCraver commented Jul 25, 2019

glennc commented Jul 25, 2019

NickCraver commented Jul 25, 2019

adamsitnik commented Jul 25, 2019 •

edited

Loading

ahsonkhan commented Jul 25, 2019 •

edited

Loading

ahsonkhan left a comment

ahsonkhan Jul 25, 2019

NickCraver Jul 25, 2019

ahsonkhan Jul 25, 2019

ahsonkhan commented Jul 25, 2019

NickCraver commented Jul 25, 2019

Martin1994 commented Sep 13, 2019

steveharter commented Oct 2, 2019

Serializers: Add System.Text.Json benchmarks for comparison. #701

Are you sure you want to change the base?

Serializers: Add System.Text.Json benchmarks for comparison. #701

Conversation

NickCraver commented Jul 25, 2019

glennc commented Jul 25, 2019

NickCraver commented Jul 25, 2019

Json_ToStream<LoginViewModel>

Json_ToStream<Location>

Json_ToStream<IndexViewModel>

Json_ToStream<MyEventsListerViewModel>

adamsitnik commented Jul 25, 2019 • edited Loading

ahsonkhan commented Jul 25, 2019 • edited Loading

ahsonkhan left a comment

Choose a reason for hiding this comment

ahsonkhan Jul 25, 2019

Choose a reason for hiding this comment

NickCraver Jul 25, 2019

Choose a reason for hiding this comment

ahsonkhan Jul 25, 2019

Choose a reason for hiding this comment

ahsonkhan commented Jul 25, 2019

NickCraver commented Jul 25, 2019

Martin1994 commented Sep 13, 2019

steveharter commented Oct 2, 2019

`Json_ToStream<LoginViewModel>`

`Json_ToStream<Location>`

`Json_ToStream<IndexViewModel>`

`Json_ToStream<MyEventsListerViewModel>`

adamsitnik commented Jul 25, 2019 •

edited

Loading

ahsonkhan commented Jul 25, 2019 •

edited

Loading