A nom parser for the Starcraft 2 Protocol Replay format. Additionally transduces through the replays to provide SC2ReplayState that keeps track of units, players, buildings, movement, units initialized and died.
The goal is to learn how to parse binary files format with nom and to learn
how the Starcraft 2 Replay file is so incredibly small for the amount of
information it packs.
From the available data, analytics, visualizations and generative art can be created, for example by using
- rerun : See the repo swarmy
- lyon (PoC in progress in cooper)
- yew cooper
- eframe/egui: See repo eframes-c2
- bevyengine/bevy can be used to see:
- An Enhanced Replay Minimap
- Additional statistics.
To consume events, they are currently loaded in memory in a HashMap:
GameLoop -> Vec<EventTypes>
let include_stats = false;
let mut replay = SC2ReplayState::new(file_path, SC2ReplayFilters::default(), include_stats).unwrap();
// at this point, all events frcom the MPQ file at `file_path` have been loaded to memory.
// To progress through the game loop, the `replay` state machine transduces from one gameloop to the next one.
// This means it recycles variables, sets position, maintains active units, etc.
// For each transduce step, an SC2EventType is returned and the unit IDs that have been changed.
// These "units" properties can be looked up in the `replay` state machine further.
// In this example, the `add_tracker_event` and the `add_game_event` also are sent a reference to the SC2ReplayState
// For a working example, see the swarmy repo referenced above.
while let Some((event, updated_units)) = replay.transduce() {
match event {
SC2EventType::Tracker {
tracker_loop,
event,
} => add_tracker_event(&self, tracker_loop, &event, updated_units)?, // Some code accessing the Tracker Events
SC2EventType::Game {
game_loop,
user_id,
event,
} => add_game_event(&self, game_loop, user_id, &event, updated_units)?,
}
}$ cargo install polars-cli
# Generate a file stats.ipc that contains all the Statistic Evenst in Arrow IPC format, This is running in parallel mode, but for 3600 files needs 1.5 GBs RAM
$ cargo run -r -- --source "/mnt/windows/Users/sebos/Documents/StarCraft II/Accounts/51504154/2-S2-1-8459957/Replays/Multiplayer/" --output stats.ipc write-arrow-ipc stats
Found 3600 files
Loaded 1579177 files
________________________________________________________
Executed in 6.21 secs fish external
usr time 64.09 secs 439.00 micros 64.09 secs
sys time 1.53 secs 133.00 micros 1.53 secs
$ # List the max number of minerals that were lost in per map when the army was killed.
❯ echo "SELECT ext_fs_replay_file_name, MAX(minerals_lost_army) FROM read_ipc('/home/seb/git/s2protocol-rs/stats.ipc') GROUP BY ext_fs_replay_file_name ORDER BY minerals_lost_army DESC;"|polars
┌───────────────────────────────────┬────────────────────┐
│ ext_fs_replay_file_name ┆ minerals_lost_army │
│ --- ┆ --- │
│ str ┆ i32 │
╞═══════════════════════════════════╪════════════════════╡
│ Heavy Artillery LE (349).SC2Repl… ┆ 71362 │
│ Arctic Dream LE (398).SC2Replay ┆ 59375 │
│ Nightscape LE (52).SC2Replay ┆ 54846 │
│ … ┆ … │
│ Emerald City LE (223).SC2Replay ┆ 43450 │
│ Rhoskallian LE (101).SC2Replay ┆ 41614 │
│ Fields of Death (345).SC2Replay ┆ 41529 │
│ Rhoskallian LE (346).SC2Replay ┆ 41425 │
└───────────────────────────────────┴────────────────────┘- Replay Tracker, Game Events and Chat Message Events for protocol75689
- Replay Tracker, Game Events and Chat Message Events for protocol84643
- Replay Tracker, Game Events and Chat Message Events for protocol87702
- Replay Tracker, Game Events and Chat Message Events for protocol88500
- Replay Tracker, Game Events and Chat Message Events for protocol89634
- Replay Tracker, Game Events and Chat Message Events for protocol89720
- Replay Tracker, Game Events and Chat Message Events for protocol90136
- Replay Tracker, Game Events and Chat Message Events for protocol90779
- Replay Tracker, Game Events and Chat Message Events for protocol90870
- Parsing unit movement is done.
- Decoding the tag/recycle done to match Game Events.
- Game Events are parsed (tho some that seem irrelevant are skipped).
- Read the version and from the version call the correct module so that we can support multiple modules.
- Support for MPQ embedded file:
replay.initData - Add the remaining Tracker/Game event types.
- Support for MPQ embedded file:
replay.gamemetadata.json - Support for MPQ embedded file:
replay.attributes.events
Currently we load all events in memory, Perhaps we can try to read batches on events by keeping MPQ nom parser &[u8] reference. For example, we could read different sections, and return events in different sections in a batch of evenst through a game loop.
After a bit of testing, it seems most of the types are compatible between versions, so only when they differ would they make part of the protocol version. Since I started this exercise on protocol87702, all types would be relative to it. That is, most modules would re-use protocol87702 as much as possible. This explains why old-er versions such as 75689 would still reference 87702 as much as possible.
The generator above thus would show example code S2ProtoResult created in favour of unwrapping/panic'ing.
The rust code for the protocols versions available were generated using: This would now be compared with ./src/versions/protocol99999.template file and from there we can analyze what has changed. Notably, the number of bits used for the Chat Message is still miscalculated to 3 so it needs to be dismissed.
mkdir src/versions/protocol89720/
RUST_LOG_SPAN_EVENTS=full RUST_LOG=debug cargo watch -i src/versions/protocol89720/mod.rs -x 'run -- --source ../s2protocol/json/protocol89720.json generate --output src/versions/protocol89720/mod.rs'
# Add the new module to src/versions/mod.rs
# Run rust format on the new src/versions/protocol87702/mod.rs file
# cargo check, cargo build, etc
# Additionally some code to transform from Protocol-Specific to Protocol-Agnostic was added, TODO: Add to generator.rs