Using ZON with zig to save/load data

I needed a way to save and load a series of structs for zigkov to store the markov chains that were generated. The easiest way is always to use some kind of existing framework, and luckily, zig offers their own serialization format ZON as well as JSON. I never used ZON before so I decided this would be a pretty good way to test it.

The solution I ended up with was to define two different set of structs, with the first one being the dynamic one for processing the data, and the other for the static loading of the said data, as I did not need to modify anything after the data was processed.

The speed of the parser is not the greatest and does take a bit for a large amount of data, which was the same I found using JSON in testing. If speed is a concern with large amounts of data I would explore other potential options.

// For Generating zon
const processWord = struct {
    word: std.ArrayList(u8),
    next: std.ArrayList(processWordStat),
    hash: u64,
    end_count: f64,
    end_normalized: f64,
    start_count: f64,
    start_normalized: f64,
};

const processWordStat = struct {
    word: std.ArrayList(u8),
    hash: u64,
    count: f64,
    normalized: f64,
};

// For Reading zon and using + Slices instead of ArrayLists
const Word = struct {
    word: []const u8,
    next: []const WordStat,
    hash: u64,
    end_count: f64,
    end_normalized: f64,
    start_count: f64,
    start_normalized: f64,
};

const WordStat = struct {
    word: []const u8,
    hash: u64,
    count: f64,
    normalized: f64,
};

var processMarkovChain: std.ArrayList(processWord) = .empty;
var MarkovChain: []Word = undefined;

Now, the converting of the dynamic struct to the static version was pretty straight forward. It is interesting to note that std.ArrayList will serialize, but not in a desirable way.

// Converts processMarkovChain -> MarkovChain array with slices instead of ArrayList()s
fn processMC(allocator: std.mem.Allocator, processMarkovChain: *std.ArrayList(processWord)) ![]Word {
    var MarkovChain: std.ArrayList(Word) = .empty;
    for (processMarkovChain.items) |*entry| {
        var tmp: std.ArrayList(WordStat) = .empty;
        for (entry.next.items) |*next| { // To Slices
            try tmp.append(allocator, .{
                .word = try next.word.toOwnedSlice(allocator),
                .hash = next.hash,
                .count = next.count,
                .normalized = next.normalized,
            });
        }
        try MarkovChain.append(allocator, .{
            .word = try entry.word.toOwnedSlice(allocator),
            .next = try tmp.toOwnedSlice(allocator),
            .hash = entry.hash,
            .end_count = entry.end_count,
            .end_normalized = entry.end_normalized,
            .start_count = entry.start_count,
            .start_normalized = entry.start_normalized,
        });
    }
    return try MarkovChain.toOwnedSlice(allocator);
}

After that, dumping it to a file is relatively straight forward, so is loading the file.

fn dumpToFile(io: std.Io, MarkovChain: *[]Word, fileName: []const u8) !void {
    var buffer: [4096]u8 = undefined;
    var file = try std.Io.Dir.cwd().createFile(io, fileName, .{});
    defer file.close(io);

    var file_writer = file.writer(io, &buffer);
    const writer = &file_writer.interface;

    try zon.stringify.serialize(MarkovChain, .{}, writer);

    try writer.flush();
}

The only thing I do not like about this approach is needing a null terminated C string to pass into the parser.

fn readFromFile(io: std.Io, allocator: std.mem.Allocator, fileName: []const u8) ![]Word {
    const file_contents = try std.Io.Dir.cwd().readFileAlloc(io, fileName, allocator, .unlimited);
    defer allocator.free(file_contents);
    const t = try allocator.dupeSentinel(u8, file_contents, 0);
    defer allocator.free(t);
    const ret = try std.zon.parse.fromSliceAlloc([]Word, allocator, t , null, .{ .free_on_error = true });
    return ret;
}

That concludes this exciting adventure of coding in zig.

Zig Stuff Online

Using ZON with zig to save/load data