[C#]如何使用Newton.Json从流中反序列化json数据
有个程序需要从大文件反序列化json，但是发现读取字符串很慢，于是想从stream反序列化这样至少比读字符串快，于是找到这个文章。
Can Json.NET serialize / deserialize to / from a stream?
Ask Question
Asked 11 years, 1 month ago
Modified 4 months ago
Viewed 237k times
186
I have heard that Json.NET is faster than DataContractJsonSerializer, and wanted to give it a try...
But I couldn't find any methods on JsonConvert that take a stream rather than a string.
For deserializing a file containing JSON on WinPhone, for example, I use the following code to read the file contents into a string, and then deserialize into JSON. It appears to be about 4 times slower in my (very ad-hoc) testing than using DataContractJsonSerializer to deserialize straight from the stream...
// DCJS
DataContractJsonSerializer dc = new DataContractJsonSerializer(typeof(Constants));
Constants constants = (Constants)dc.ReadObject(stream);
// JSON.NET
string json = new StreamReader(stream).ReadToEnd();
Constants constants = JsonConvert.DeserializeObject(json);
Am I doing it wrong?
- .net
- serialization
- json.net
Share
Improve this question
Follow
edited Dec 28, 2018 at 16:35
Tim Cooper
155k3737 gold badges322322 silver badges275275 bronze badges
asked Nov 16, 2011 at 19:41
Omri Gazitt
3,42822 gold badges2121 silver badges2727 bronze badges
Add a comment
6 Answers
Sorted by:
Highest score (default) Trending (recent votes count more) Date modified (newest first) Date created (oldest first)
339
The current version of Json.net does not allow you to use the accepted answer code. A current alternative is:
public static object DeserializeFromStream(Stream stream)
{
var serializer = new JsonSerializer();
using (var sr = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(sr))
{
return serializer.Deserialize(jsonTextReader);
}
}
Documentation: Deserialize JSON from a file stream
Share
Improve this answer
Follow
edited Jan 30, 2015 at 20:35
James Newton-King
47.3k2424 gold badges109109 silver badges129129 bronze badges
answered Jul 22, 2013 at 12:52
user2535425
-
5
JsonTextReader will close its StreamReader by default, so this example could be simplified a bit by constructing the StreamReader in the call to the JsonTextReader constructor.– Oliver Bock
Jun 29, 2016 at 6:31 -
3
Actually, I have an OutOfMemory exception and I already use this code, pretty much exactly. Which, I believe, goes to say, this is not a guarantee - if the deserialized object is large enough, and you're stuck in a 32-bit process, you may still get memory errors with this code– PandaWood
Oct 21, 2016 at 3:03 -
1
i am getting an error "The type or namespace name 'JsonTextReader' could not be found" ...any suggestions?– hnvasa
Sep 22, 2017 at 0:24 -
6
I needed to addstream.Position = 0;to correctly deserialize my json.– hybrid2102
Mar 21, 2019 at 14:54 - The documentation in the link does not mention anything about
JsonTextReader– John Henckel
Nov 17 at 20:16
Add a comment
114
public static void Serialize(object value, Stream s)
{
using (StreamWriter writer = new StreamWriter(s))
using (JsonTextWriter jsonWriter = new JsonTextWriter(writer))
{
JsonSerializer ser = new JsonSerializer();
ser.Serialize(jsonWriter, value);
jsonWriter.Flush();
}
}
public static T Deserialize(Stream s)
{
using (StreamReader reader = new StreamReader(s))
using (JsonTextReader jsonReader = new JsonTextReader(reader))
{
JsonSerializer ser = new JsonSerializer();
return ser.Deserialize(jsonReader);
}
}
Share
Improve this answer
Follow
edited May 4, 2016 at 17:37
Jim Buck
2,3552525 silver badges4141 bronze badges
answered Mar 27, 2014 at 14:02
ygaradon
2,16822 gold badges2020 silver badges2727 bronze badges
-
4
Thanks! This helped me avoid an OutOfMemoryException I was getting when I was serializing a very large object collection to a string, and then writing that string into my stream (instead of just serializing directly to the stream).– Jon Schneider
Apr 14, 2016 at 19:50 -
9
Why flush? Doesn't the Dispose call caused by the using block already do that?– Şafak Gür
Nov 17, 2017 at 8:45 -
9
Side note, because it might help others: if you useJsonSerializer ser = JsonSerializer.Create(settings);you can define which settings to use during de/serialization.– mike
May 15, 2018 at 14:12 -
6
One potential issue with thisSerializeimplementation is that it closes theStreampassed as an argument, which depending on the application can be a problem. With .NET 4.5+ you can avoid this problem by using aStreamWriterconstructor overload with a parameterleaveOpenthat lets you leave the stream open.– Joe
Aug 29, 2019 at 13:23 - @ŞafakGür It may not. Seen that happening to many times, so as a reflex I just flush, like the sample.
– Erdogan Kurtur
Sep 7, 2021 at 11:52
Add a comment
63
UPDATE: This no longer works in the current version, see below for correct answer (no need to vote down, this is correct on older versions).
Use the JsonTextReader class with a StreamReader or use the JsonSerializer overload that takes a StreamReader directly:
var serializer = new JsonSerializer();
serializer.Deserialize(streamReader);
Share
Improve this answer
Follow
edited May 23, 2017 at 12:34
CommunityBot
111 silver badge
answered Nov 16, 2011 at 20:29
Paul Tyng
7,87811 gold badge3333 silver badges5757 bronze badges
-
27
Pretty sure this no longer works. You have to use a JsonReader or TextReader– BradLaney
Jul 7, 2012 at 0:08 -
12
You may want to include the version number this is still working on so people know when to scroll down.– PoeHaH
Oct 24, 2017 at 5:00 - @BradLaney yup JsonTextReader(givenStreamReader) is the way to go now
– Antoine Meltzheim
Apr 18, 2018 at 12:28 - Thank you for taking the time to edit your answer re it's working status and answer recommendation
– Nick Bull
Mar 7, 2019 at 10:43 -
4
"no need to vote down, this is correct on older versions" - It's no longer relevant to most people, so it should be voted down as a better answer exists.– Ed S.
Oct 7, 2020 at 17:15
Add a comment
32
I've written an extension class to help me deserializing from JSON sources (string, stream, file).
public static class JsonHelpers
{
public static T CreateFromJsonStream(this Stream stream)
{
JsonSerializer serializer = new JsonSerializer();
T data;
using (StreamReader streamReader = new StreamReader(stream))
{
data = (T)serializer.Deserialize(streamReader, typeof(T));
}
return data;
}
public static T CreateFromJsonString(this String json)
{
T data;
using (MemoryStream stream = new MemoryStream(System.Text.Encoding.Default.GetBytes(json)))
{
data = CreateFromJsonStream(stream);
}
return data;
}
public static T CreateFromJsonFile(this String fileName)
{
T data;
using (FileStream fileStream = new FileStream(fileName, FileMode.Open))
{
data = CreateFromJsonStream(fileStream);
}
return data;
}
}
Deserializing is now as easy as writing:
MyType obj1 = aStream.CreateFromJsonStream();
MyType obj2 = "{\"key\":\"value\"}".CreateFromJsonString();
MyType obj3 = "data.json".CreateFromJsonFile();
Hope it will help someone else.
Share
Improve this answer
Follow
answered Jul 29, 2013 at 13:42
Tok'
56588 silver badges1111 bronze badges
-
2
Against: it will pollute all strings with the extension methods. Workarounds: Only declareUsing SomeJsonHelpersNamespacewhere needed or remove thethiskeyword and useJsonHelpers.CreateFromJsonString(someJsonString)Pro: it's so easier to use :)– Tok'
Jul 17, 2017 at 10:24 -
2
Although it could be seen as "polluting", almost half the extensions in String object could be seen the same way. This extends an object in a manner seen as useful to anyone that would consistently change from string(json) to JSON.– vipersassassin
Apr 3, 2018 at 21:44 - Also using Encoding.Default is bad as it will behave differently on different machines (see the big warning at the Microsoft docu). JSON is expected to be UTF-8 and this is what JsonSerializer expects. Thus it should be
Encoding.UTF8. The code as is will produce garbled strings or fail to deserialize if non-ASCII characters are used.– ckuri
Oct 25, 2019 at 7:11
Add a comment
17
I arrived at this question looking for a way to stream an open ended list of objects onto a System.IO.Stream and read them off the other end, without buffering the entire list before sending. (Specifically I'm streaming persisted objects from MongoDB over Web API.)
@Paul Tyng and @Rivers did an excellent job answering the original question, and I used their answers to build a proof of concept for my problem. I decided to post my test console app here in case anyone else is facing the same issue.
using System;
using System.Diagnostics;
using System.IO;
using System.IO.Pipes;
using System.Threading;
using System.Threading.Tasks;
using Newtonsoft.Json;
namespace TestJsonStream {
class Program {
static void Main(string[] args) {
using(var writeStream = new AnonymousPipeServerStream(PipeDirection.Out, HandleInheritability.None)) {
string pipeHandle = writeStream.GetClientHandleAsString();
var writeTask = Task.Run(() => {
using(var sw = new StreamWriter(writeStream))
using(var writer = new JsonTextWriter(sw)) {
var ser = new JsonSerializer();
writer.WriteStartArray();
for(int i = 0; i < 25; i++) {
ser.Serialize(writer, new DataItem { Item = i });
writer.Flush();
Thread.Sleep(500);
}
writer.WriteEnd();
writer.Flush();
}
});
var readTask = Task.Run(() => {
var sw = new Stopwatch();
sw.Start();
using(var readStream = new AnonymousPipeClientStream(pipeHandle))
using(var sr = new StreamReader(readStream))
using(var reader = new JsonTextReader(sr)) {
var ser = new JsonSerializer();
if(!reader.Read() || reader.TokenType != JsonToken.StartArray) {
throw new Exception("Expected start of array");
}
while(reader.Read()) {
if(reader.TokenType == JsonToken.EndArray) break;
var item = ser.Deserialize(reader);
Console.WriteLine("[{0}] Received item: {1}", sw.Elapsed, item);
}
}
});
Task.WaitAll(writeTask, readTask);
writeStream.DisposeLocalCopyOfClientHandle();
}
}
class DataItem {
public int Item { get; set; }
public override string ToString() {
return string.Format("{{ Item = {0} }}", Item);
}
}
}
}
Note that you may receive an exception when the AnonymousPipeServerStream is disposed, I ignored this as it isn't relevant to the problem at hand.
Share
Improve this answer
Follow
answered Feb 24, 2014 at 22:39
Blake Mitchell
2,56711 gold badge2525 silver badges2222 bronze badges
-
1
I need to modify this so that I can get any complete JSON object. My server and client communicate by sending snippets of JSON so the client could send{"sign in":{"username":"nick"}}{"buy item":{"_id":"32321123"}}and it needs to see this as two fragments of JSON signaling an event each time it reads a fragment. In nodejs this can be done in 3 lines of code.– Nick Sotiros
Jun 26, 2016 at 6:35
Add a comment
-1
another option that is handy when you are running out of memory is to periodically flush
/// serialize the value in the stream.
/// the type to serialize
/// The stream.
/// The value.
/// The json settings to use.
///
///
public static void JsonSerialize(this Stream stream,[DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, int bufferSize=1024, bool leaveOpen=false)
{
using (var writer = new StreamWriter(stream,encoding: System.Text.Encoding.UTF32,bufferSize,leaveOpen))
using (var jsonWriter = new JsonTextWriter(writer))
{
var ser = JsonSerializer.Create( settings );
ser.Serialize(jsonWriter, value);
jsonWriter.Flush();
}
}
/// serialize the value in the stream asynchronously.
/// (this Stream stream,[DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, int bufferSize=1024, bool leaveOpen=false, CancellationToken cancellationToken=default)
{
using (var writer = new StreamWriter(stream,encoding: System.Text.Encoding.UTF32,bufferSize: bufferSize,leaveOpen: leaveOpen))
using (var jsonWriter = new JsonTextWriter(writer))
{
var ser = JsonSerializer.Create( settings );
ser.Serialize(jsonWriter, value);
return jsonWriter.Flush();
}
//jsonWriter.FlushAsnc with my version gives an error on the stream
return Task.CompletedTask;
}
You can test/ use it like so:
[TestMethod()]
public void WriteFileIntoJsonTest()
{
var file = new FileInfo(Path.GetTempFileName());
try
{
var list = new HashSet();
for (int i = 0; i < 100; i++)
{
list.Add(Guid.NewGuid());
}
file.JsonSerialize(list);
var sr = file.IsValidJson>(out var result);
Assert.IsTrue(sr);
Assert.AreEqual(list.Count, result.Count);
foreach (var item in result)
{
Assert.IsFalse(list.Add(item), $"The GUID {item} should already exist in the hash set");
}
}
finally
{
file.Refresh();
file.Delete();
}
}
you'd need to create the extension methods, here is the whole set:
public static class JsonStreamReaderExt
{
static JsonSerializerSettings _settings ;
static JsonStreamReaderExt()
{
_settings = JsonConvert.DefaultSettings?.Invoke() ?? new JsonSerializerSettings();
_settings.ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor;
_settings.DateTimeZoneHandling = DateTimeZoneHandling.Utc;
_settings.DateFormatHandling = DateFormatHandling.IsoDateFormat ;
}
///
/// serialize the value in the stream.
///
/// (this Stream stream,[DisallowNull] T value)
{
stream.JsonSerialize(value,_settings);
}
///
/// serialize the value in the file .
///
/// (this FileInfo file,[DisallowNull] T value)
{
if (string.IsNullOrEmpty(file.DirectoryName)==true && Directory.Exists(file.DirectoryName) == false)
{
Directory.CreateDirectory(file.FullName);
}