So you’re making yet another Scala app and you need to parse or write some data to file/database/webservice.
What do you do? If you’re not already inside an (awesome) framework like Play Framework you will probably search “Scala JSON” and find something like Spray JSON. That’s what I did. So then you write:
import scalax.io._ import spray.json._ import DefaultJsonProtocol._ val jsonString = largeMapObject.toJson.compactPrint Resource.fromFile("someFile").write(jsonString)(Codec.UTF8)
… and you get an exception upon either writing, or parsing this string after it was only partially written. You have several megabytes of this so just writing to a String will not work.
You will need a JSON printer that can output streams or handle writers. After searching I found Scala Stuff’s json-parser that has support for several JSON AST libraries like Spray. With json-parser writing large JSON objects is just a matter of:
import org.scalastuff.json.spray._ val writer = new PrintWriter(new File(file)) new SprayJsonPrinter(writer, 0)(largeMapObject.toJson) writer.close()
and reading is just as easy:
val reader = Source.fromFile(file).bufferedReader() val jsValue = SprayJsonParser.parse(reader)
Although this basic usage of Spray and Scala Stuff’s json-parser allow you to parse and write large JSON object, your machines memory is still a bottleneck as they keep the resulting JSON AST in memory. When dealing with huge JSON streams I recommend that you take a look at json-parser’s JsonHandler trait or Play Frameworks JSON Transformers.