Add a tool that measures emotes from a store's 3D preview

tools/capture drives a store page's skin viewer in headless Firefox, stepping its clock one tick at
a time and taking every tick from ten fixed cameras. It then fits our own rig to the frames by
rendering the model (a Python port of LimbBend and PoseApplier) and matching outlines and colours,
and writes the result as Emotecraft JSON. It measures pixels only and never reads the page's
animation data. The README covers the steps, the checks and the limits.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-09-29 16:55:08 +02:00
co-authored by claude
parent 7f8de39253
commit df9a2587e6
19 changed files with 1785 additions and 0 deletions
+63
View File
@@ -0,0 +1,63 @@
# Emote capture
Measures an emote from a store page's 3D preview and writes it as an Emotecraft emote our loader
reads. It works from pixels only: it renders our own player model in the same pose from the same
cameras and adjusts the pose until outlines and colours match. It never reads the page's animation
data.
## Setup
```sh
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/playwright install firefox # Playwright's own Firefox, in its cache folder
```
## Steps
```sh
.venv/bin/python capture.py <store url> <name> --skin Steve # frames/<name>/<view>/<tick>.png + views.json
.venv/bin/python measure.py <name> --ticks 0:13 # frames/<name>/rig.json (one loop: 13 ticks)
.venv/bin/python fit.py <name> --symmetric --title "Twerk" --out <path>.json
```
- **`capture.py`** opens the page in headless Firefox with `viewer.js` injected, switches the
preview to 3D and puts a skin on it. `--skin` takes one of the dialog's presets or any Minecraft
username. `viewer.js` replaces the page's clock, so frames step one tick (50 ms) at a time however
fast the emote moves. It also finds the camera through three.js's devtools hook. Each tick is taken
from 10 fixed cameras with a transparent background. It also measures the loop length from the
frames.
- **`skin.py`** fetches that username's skin from Mojang, so our model can wear the same texture.
- **`measure.py`** fits our rig to every tick. It renders the model with `model.py` (a port of
`LimbBend` and `PoseApplier`) and `render.py`, and scores it by outline distance and colour. It
searches with CMA-ES on all cores. The first tick is searched in stages: torso and head, then each
limb from several starts, then everything, then its back-to-front twins (`flip.py`). It also finds
the viewer's scale then. For a whole loop, later rounds refit every tick held near the loop's
smoothed curve, so drift can't build up. Leaning further forward while arching further back looks
almost the same from every camera, so a small cost on the torso's bend picks the straightest torso.
- **`fit.py`** smooths each channel over the loop (Fourier harmonics) and stands the emote on the
ground. It keys each channel wherever straight-line interpolation misses by more than 1° or 0.1 px.
`--symmetric` removes left/right lopsidedness (`symmetry.py`): each tick is averaged with the mirror
of the pose half a loop on, which keeps an even sway and drops any lean to one side.
## Checking
- `check_port.py` compares `model.py`'s bend maths with the Java classes (after
`./gradlew :core:compileJava`).
- `synth_test.py` renders our own twerk test animation as a fake capture, so `measure.py` can be
scored against a known answer.
- `symmetry.py <name>` lists the most lopsided channels.
- `compare.py <name> <tick>` puts the captured frames next to our render of the fit.
- `overlay.py <name>` draws both outlines on top of each other.
- `sheet.py` tiles captured frames.
To see it in game, add the JSON to the testmod's `player_animations` and `EMOTECRAFT_EMOTES`, and add
shots to `EmotesGameTest`.
## Limits
- A loop takes about 45 minutes on 7 cores.
- The emote comes out with a keyframe on almost every tick for every channel.
- Other rigs differ from ours (a knee made of two rigid boxes, a torso that doesn't bend), so a fit is
our closest match, not an exact copy.
- Extreme poses can still land in a wrong answer on the first tick. `synth_test.py`'s 80° lean does.