emotes/tools/capture

Emote capture

Measures an emote from a store page's 3D preview and writes it as an Emotecraft emote our loader reads. It works from pixels only: it renders our own player model in the same pose from the same cameras and adjusts the pose until outlines and colours match. It never reads the page's animation data.

Setup

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/playwright install firefox   # Playwright's own Firefox, in its cache folder

Steps

.venv/bin/python capture.py <store url> <name> --skin Steve   # frames/<name>/<view>/<tick>.png + views.json
.venv/bin/python measure.py <name> --ticks 0:13               # frames/<name>/rig.json (one loop: 13 ticks)
.venv/bin/python fit.py <name> --symmetric --title "Twerk" --out <path>.json

• capture.py opens the page in headless Firefox with viewer.js injected, switches the preview to 3D and puts a skin on it. --skin takes one of the dialog's presets or any Minecraft username. viewer.js replaces the page's clock, so frames step one tick (50 ms) at a time however fast the emote moves. It also finds the camera through three.js's devtools hook. Each tick is taken from 10 fixed cameras with a transparent background. It also measures the loop length from the frames.
• skin.py fetches that username's skin from Mojang, so our model can wear the same texture.
• measure.py fits our rig to every tick. It renders the model with model.py (a port of LimbBend and PoseApplier) and render.py, and scores it by outline distance and colour. It searches with CMA-ES on all cores. The first tick is searched in stages: torso and head, then each limb from several starts, then everything, then its back-to-front twins (flip.py). It also finds the viewer's scale then. For a whole loop, later rounds refit every tick held near the loop's smoothed curve, so drift can't build up. Leaning further forward while arching further back looks almost the same from every camera, so a small cost on the torso's bend picks the straightest torso.
• fit.py smooths each channel over the loop (Fourier harmonics) and stands the emote on the ground. It keys each channel wherever straight-line interpolation misses by more than 1° or 0.1 px. --symmetric removes left/right lopsidedness (symmetry.py): each tick is averaged with the mirror of the pose half a loop on, which keeps an even sway and drops any lean to one side.

Checking

• check_port.py compares model.py's bend maths with the Java classes (after ./gradlew :core:compileJava).
• synth_test.py renders our own twerk test animation as a fake capture, so measure.py can be scored against a known answer.
• symmetry.py <name> lists the most lopsided channels.
• compare.py <name> <tick> puts the captured frames next to our render of the fit.
• overlay.py <name> draws both outlines on top of each other.
• sheet.py tiles captured frames.

To see it in game, add the JSON to the testmod's player_animations and EMOTECRAFT_EMOTES, and add shots to EmotesGameTest.

Limits

• A loop takes about 45 minutes on 7 cores.
• The emote comes out with a keyframe on almost every tick for every channel.
• Other rigs differ from ours (a knee made of two rigid boxes, a torso that doesn't bend), so a fit is our closest match, not an exact copy.
• Extreme poses can still land in a wrong answer on the first tick. synth_test.py's 80° lean does.