One of the really neat ideas that I picked up lurking in the Moonworks community was building a “static audio atlas” for sfx that works by loading a single wav file and initializing in-game audio buffers using regions of the loaded wav data.

While I can imagine there is a list of many benefits to this practice it would be dishonest to suggest that I pursued it for any reason other than personal aesthetics and fun. I know that FMOD does something similar to this with it’s bank files so it’s not entirely unusual. :D

But just for the heck of it, I can imagine the following might be true:

  • For a project with many short sfx you’re saving 44 bytes per file by packing them. (I have no idea how many sfx you’d need before that savings is meaningful though 😅).
  • You only have to alloc/free a single buffer for all sfx. (This is arguably also possible with multiple wav files depending on how you manage memory)
  • Compiling content into nicely packed distribution formats works nicely with mmap.

A possible disadvantage might be that it’s now more complicated to hot-reload asset data since you’re always re-compiling the atlas. You’d have to either ensure stable pointers to data, or use handles, but a lot of how messy or complicated that would be depends on the audio library you’re using.

Regardless of any of this, I did the thing and now present it to you!

Wavpacker Link to heading

The goal was simple: create a simple but useful utility to build an “audio atlas” (a ‘wpak’) using only what Jai provides in the distribution. Turns out this is pretty easy to do because Jai provides.

The essential process:

  • Given a folder with many wav files:
    • Group them into lists by their common formats:
  • For each group:
    • Write out an index
    • Write out a concatenated wav file.

Jai more or less provides everything out of the box to achieve this:

  • Command line args are just a [..]string and this tool isn’t complex enough to warrant any sort of elegant procedural arg parser. Instead we just iterate the list and match whatever we support. The Basic module provides access to args, and we just use a standard if-case sequence in a while loop to match strings.
  • Given a path to a folder, we can walk the contents and find files with a *.wav extension. This is provided by the File_Utilities module.
  • Wav files must be loaded and their headers read in order to place them into groups. This is provided by the Wav_File module.
  • All user facing output is handled by print and log_error.
  • Our output files are written using the functions provided by the File module.
  • Path parsing and filename formatting handled by String and Basic modules.
  • And our groups are managed using the Hash_Table module.

An additional goal of this tool was to adhere to the notion of being “maximally useful for its size”, and to provide sufficient documentation of the goals and behaviors of the program.

To that end I include an “+info” (alias “+man”, “+manual”) flag that will print extended information about how the utility works and describes the format of the index with an example. The idea that perhaps all future utilities might adopt a common convetion allowing usage such as wavpacker +info | less in order to always be certain that all functions and options of a program can be adequately described for others (including my future self).

Info

A quick note on the index format

Folks in the beta will recognize the index format but it might be useful for anyone peering in from the outside and provide some context.

The gist of the format is:

[1]

DATA pak_0.wav

BUFFER Foo
OFFSET 0
LEN    10

BUFFER Bar
OFFSET 10
LEN    20

The original C# version of this utility used json, and while there is the jaison module, it is not part of the standard Jai distribution.

What Jai does provide is a module called Text_File_Handler which provides some common scaffolding for parsing text based file formats. This is used by several other modules and even the skeletal-animation example. So I went with it, and I’m pretty happy with the choice so far. When I look at the format I believe it’s easily consumed by whatever language I’m working with and doesn’t require me to reach for anything other than standard language features.

You can make the case that conforming to proper json would be more robust and I wouldn’t argue with you. But this is intended to be tightly bound to your game or application as an index and not represent serialized data. Your content sources are expected to conform to project requirements, and the index should be as simple as possible.

A note about future plans Link to heading

As you might notice in the code below, I indicate plans for a binary format that combines the wav and the index. Before I do this, I’d like to build up some examples of using this and make a small tool to audition and inspect a wpak. Without having that little tool in-place the only way to really verify the output of the tool would be to use ImHex which works but doesn’t answer the question of usability.

Finally, I feel like want to extend the pak concept to include compressed audio formats as well. To me that’s the sort of final game of the idea. Allowing a content builder to package up all audio into banks and not just wav files for sfx.

Before reaching that point I will most likely create another *packer tool. I’m fond of qoa and ogg is pretty much standard, so perhaps a qoapacker and *.qpak or an oggpacker and *.opak will preceed any superpak format. Jai is extremely low friction and it will be interesting to see what patterns emerge in terms of organizing any common function between several tools.

As these things become more complete I’d like to think I could make a more appropriate home on itch for builds of things.

For now the idea was to share the outcome and experience using Jai rather than build some permanent home for releases of the tool itself. If you are not in the beta and would like a build of this then just let me know.

So without spending more of your time, here is the full source of the utility at the time of writing this post.


USAGE :: #string END

-------------------------------------------------------------------------------
Usage: wavpacker -i <input folder> -o <output folder>
-------------------------------------------------------------------------------

Given a folder containing a set of wav files, this program will load each and
create a single new "atlas" or "pack" wav containing the data of all input
files.

-------------------------------------------------------------------------------

Options:
    -h    | -help       : Show this help.
    +info | +man        : Display extended information ("the manual").
    -i    | -input      : Path to an input folder.
    -o    | -output     : Path to the output folder.
    -w    | -overwrite  : Overwrite any existing data in output folder.
    -c    | -mkdir      : Create output folder if it doesn't exist.
    -f    | -force      : Overwrite existing data *and/or* create output folders.
    -R                  : Recursively scan input folder.

END

RIFF_CHUNK_HDR :: "RIFF";
WAVE_FMT_CHUNK_HDR :: "WAVEfmt ";
DATA_CHUNK_HDR :: "data";

EXT_WPAK_DATA :: "wav"; // A regular *.wav file, but composed of all inputs concatenated.
EXT_WPAK_INDEX :: "wpak"; // An ascii index of the names and byte ranges of the wavs in the associated pak.

EXT_BWPAK :: "bwpak"; // A combined index+wav in binary format. @TODO


#add_context input_directory : string;
#add_context output_directory : string;

#add_context overwrite_existing_outputs : bool = false;
#add_context create_missing_output_directory : bool = false;

#add_context recursive : bool = false;


Pak_Item :: struct {
    path: string;
    samples: string;
};

Wav_Pak :: struct {
    format: Waveformatex;
    data_length: u32;
    items: [..]Pak_Item;
}

wav_table : Table(s64, Wav_Pak);


ingest_input_folder :: () -> [] string {
    files : [..] string;

    visitor :: (info: *File_Visit_Info, user_data: *[..] string) {
        if ends_with(info.short_name, "wav") {
            array_add(user_data, copy_string(info.full_name));
        }
    }

    visit_files(context.input_directory, context.recursive, *files, visitor, follow_directory_symlinks=false);

    return files;

}


main :: () {
    args :=  get_command_line_arguments();
    defer  array_reset(*args);

    // Exit early with usage information when run with no arguments.
    if (args.count == 0) {
        print(USAGE);
        exit(1);
    }

    ///
    /// Process command line flags.
    ///

    idx := 1; // 0 is always the path to the program
    while idx < args.count {
        if args[idx] == {
            case "+info"; #through;
            case "+manual"; #through;
            case "+man"; {
                print(USAGE);
                print(MANUAL);
                exit(0);
            }

            case "-h"; #through;
            case "-help"; {
                print(USAGE);
                exit(0);
            }

            case "-R"; context.recursive = true;

            case "-f"; #through;
            case "-force"; {
                context.overwrite_existing_outputs = true;
                context.create_missing_output_directory = true;
            }

            case "-w"; #through;
            case "-overwrite"; context.overwrite_existing_outputs = true;

            case "-c"; #through;
            case "-mkdir"; context.create_missing_output_directory = true;

            case "-i"; #through;
            case "-input"; {
                idx += 1;
                context.input_directory = args[idx];
            }

            case "-o"; #through;
            case "-output"; {
                idx += 1;
                context.output_directory = args[idx];
            }

            // Unknown argument
            case; {
                log_error("Unknown argument: %\n", args[idx]);
                print(USAGE);
                exit(1);
            }
        }
        idx += 1;
    }

    ///
    /// Validate the path parameters.
    ///

    // @TODO: Consider ensuring that paths are resolved to their absolute versions?

    valid_paths := true;

    if !file_exists(context.input_directory) {
        valid_paths = false;
        if !context.input_directory {
            log_error("Input directory was not specified.\n");
        }
        else {
            log_error("Input directory does not exist.\n");
        }
    }

    if !file_exists(context.output_directory) {

        // Create the directory and any intermediate directories.
        if context.create_missing_output_directory {
            print("Creating: %\n", context.output_directory);
            if !make_directory_if_it_does_not_exist(context.output_directory, true) {
                log_error("Failed to create output directory!\n");
                exit(1);
            }
        }
        else {
            valid_paths = false;
            if !context.output_directory {
                log_error("Output directory was not specified.\n");
            }
            else {
                log_error("Output directory does not exist.\n");
            }
        }

    }

    if !valid_paths {
        print(USAGE);
        exit(1);
    }

    ///
    /// Collect all input files to process, grouping them by their format
    ///

    print("Scanning input folder:\n");
    wav_files := ingest_input_folder();

    if wav_files.count == 0 {
        print("No wav files found.");
        exit(0);
    }

    for wav_files {
        print("  Loading: %\n", it);
        file_data, file_read := read_entire_file(it);
        if !file_read {
            log_error("Failed to load %\n", it);
        }

        format, samples, wav_header_parsed, extra := get_wav_header(file_data);
        if !wav_header_parsed {
            log_error("Unable to parse '%' as wav.\n", it);
            exit(1);
        }

        // Using an integer for the key instead of defining a hasher for the struct.
        // this assumes that these sum up to a unique value for each format that
        // might be present in a folder.
        // TODO: verify this!
        key := format.wFormatTag + format.nChannels + format.nSamplesPerSec + format.wBitsPerSample;

        pak, just_added := find_or_add(*wav_table, key);
        if just_added {
            pak.format = format;
            pak.data_length = xx samples.count;
        }
        else {
            pak.data_length += xx samples.count;
        }

        array_add(*pak.items, Pak_Item.{
            path = it,
            samples = samples,
        });
    }

    ///
    /// Write out our packed data and index
    ///

    print("\n\nWriting paks to output folder:\n");

    pak_id := 0;
    for pak: wav_table {
        index_filename := tprint("pak_%.%", pak_id, EXT_WPAK_INDEX);
        wav_filename   := tprint("pak_%.%", pak_id, EXT_WPAK_DATA);

        index_path := tprint("%/%", context.output_directory, index_filename);
        wav_path   := tprint("%/%", context.output_directory, wav_filename);

        //
        // Make sure we aren't overwriting data (unless the user instructs us to)
        //

        if file_exists(index_path) || file_exists(wav_path) {
            if !context.overwrite_existing_outputs {
                log_error("\nERROR: The following outputs already exist:\n");
                log_error("  %\n  %\n\nRerun with the '-f' or '-w' flag to overwrite them.\n\n",
                    index_path, wav_path);
                exit(1);
            }
        }

        //
        // Open the index and wav files
        //

        print("  %: % %-bit\n", index_path, pak.format.nSamplesPerSec, pak.format.wBitsPerSample);

        index_file, index_opened := file_open(index_path, for_writing=true, keep_existing_content=false);
        if !index_opened {
            log_error("Failed to open % for writing!", index_filename);
            exit(1);
        }
        defer file_close(*index_file);

        wav_file, wav_opened := file_open(wav_path, for_writing=true, keep_existing_content=false);
        if !wav_opened {
            log_error("Failed to open % for writing!", wav_filename);
            exit(1);
        }
        defer file_close(*wav_file);

        //
        // Write the wav and index headers
        //

        ///////////////
        // wav header
        file_write(*wav_file, RIFF_CHUNK_HDR);

        total_file_size : u32 = 36 + pak.data_length;
        wav_chunk_size : u32 = 16;

        file_write(*wav_file, *total_file_size, 4);
        file_write(*wav_file, WAVE_FMT_CHUNK_HDR);
        file_write(*wav_file, *wav_chunk_size, 4);
        file_write(*wav_file, *pak.format.wFormatTag, 2);
        file_write(*wav_file, *pak.format.nChannels, 2);
        file_write(*wav_file, *pak.format.nSamplesPerSec, 4);
        file_write(*wav_file, *pak.format.nAvgBytesPerSec, 4);
        file_write(*wav_file, *pak.format.nBlockAlign, 2);
        file_write(*wav_file, *pak.format.wBitsPerSample, 2);
        file_write(*wav_file, DATA_CHUNK_HDR);
        file_write(*wav_file, *pak.data_length, 4);

        //////////////////
        // Index header
        file_write(*index_file, "[1]\n\n");

        data_line := tprint("DATA %\n\n", wav_filename);
        file_write(*index_file, data_line);

        //
        // For each item in this pack, write the samples and it's index entry
        //

        offset := 0;

        for item: pak.items {
            path, filename, ext, filename_with_ext := path_decomp(item.path);

            // Write the sample buffer
            file_write(*wav_file, item.samples);

            // Write the index entry
            entry := tprint("BUFFER %\nOFFSET %\nLEN    %\n\n", filename, offset, item.samples.count);
            file_write(*index_file, entry);

            offset += item.samples.count;
        }

        pak_id += 1;

        reset_temporary_storage();
    }
}


#import "Basic";
#import "System";
#import "File";
#import "File_Utilities";
#import "String";
#import "Wav_File";
#import "Hash_Table";


MANUAL :: #string END

-------------------------------------------------------------------------------

This utility is a simple way to compile a collection of uncompressed audio
files into a single "packed" version that can be loaded from disk in one
shot.

The inspiration comes from Evan Hemsley (https://moonside.games/pages/about),
and you can find a C# version here: https://github.com/thatcosmonaut/GGJ2024

-------------------------------------------------------------------------------

Given a path to an input folder and a path to an output folder:

  - For each *.wav file found in the input folder:
    - Group by format (tag + samplerate + bit-depth + channel-count)

  - For each group:
    - Create an index pak_<N>.wpak file
    - Create an pak_<N>.wav with the concatenated data from the files.
    - Where <N> is the "id", the enumeration, of the group

-------------------------------------------------------------------------------

The output created by wavpacker consists of an index and concatenated audio
data. By default the index is an ascii file containing the names of each buffer,
their offset in the sample data, and their length. The concatenated audio is
just a regular wav file for easy analysis and audition in standard tools.


The index format is as follows:

- The extension shall be .wpak

- The first line of the file is the revision: "[1]".

- Next is a line that specifies the name of the wav data: "DATA <filename>"

- Following that will be a series of lines:
  - The name of the buffer entry:       "BUFFER <buffer name>"
  - The byte offset of the buffer:      "OFFSET <int>"
  - The length in bytes of the buffer:  "LENGTH <int>"

Example:

[1]

DATA    pak_0.wav

BUFFER Foo
OFFSET 0
LEN    10

BUFFER Bar
OFFSET 10
LEN    20

BUFFER Biz
OFFSET 30
LEN    10

END