← @shoutplenty Twitter archive
fleegar
@shoutplenty
i wanna write down briefly what a correctly-designed autosplitter looks like, since i mentioned it. this is kinda taken off this sketch i wrote down in 2021, but more focused on the essentials after a couple years’ thinking [1/12] https://gist.github.com/pyorot/2a0d8dcda477acc66c65bfa9414ff1fa
12/21/2023, 12:07:07 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
(this applies to console games like sm64 and sms that aren’t deterministic like smb1)
there are two key features that make existing designs inadequate
1) they read in live capture feeds rather than encoded videos as they’re recorded
2) they don’t dump the frames they split on
12/21/2023, 12:07:08 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
we need #1 cos there is no relationship between forked capture feeds being polled by the splitter and by the video encoder. so the splitter has to use its own real timer to guess when a frame happened using when it first saw it
12/21/2023, 12:07:09 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
the encoded video otoh literally contains the timestamps of the finished speedrun product – the video itself (which is the standard of truth). hence, read those and u no longer need to retime anything, cos it’s the exact same data
12/21/2023, 12:07:09 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
#2 is for accountability – u need to know where it splits. if before-and-after still frames are saved then we can know w/o retiming that the split was correct, which cuts down the verification work and lets us rapidly improve on the image recognition algorithms too
12/21/2023, 12:07:10 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
with these 2 essentials done, and maybe a hacked version of obs, we have a program that can read both live and recorded video, and generate split timestamps that match the video, in real-time
next comes the image recognition algorithm
12/21/2023, 12:07:11 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
neural networks prolly work well enough out of the box, but even the l2-norm + image mask was effectively 100% reliable for me when running sms any%, off a really tricky visual cue as well (shine get animation)
the mask can be fixed for a type of split frame on a game version
12/21/2023, 12:07:11 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
then u calibrate it to a setup by rescaling it and then averaging out the colours it captures on the split frames. dumped frames mean u can even find the right frames using an opaque neural network, and the user can just fix the frames it chose if they’re wrong
12/21/2023, 12:07:12 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
the masks and visuals can be fixed for a game; so can literally the whole split pattern. the next step then is to do away with livesplit and throw out the linearity requirement, which is a pain when ppl are often doing levels out-of-order
12/21/2023, 12:07:13 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
rather, the splitting can just output a feed of raw timestamps and metadata about the split frame, then some heuristics can be used to interpret those according to a route
even just reading data as it comes in, the splitter can tell u, last load was 1.57, last IL was 30.20 etc.
12/21/2023, 12:07:13 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
make a basic splits editor that can assign level info to splits, then it’s order-independent while still containing breakdowns into loads and sublevels etc.
plus, u can use it to automatically retime literally anything, and know it’s correct using the frame dump
12/21/2023, 12:07:14 PM
Favs: 0
Retweets: 0
linkfleegar
@shoutplenty
i will go out on a limb and say none of this is hard to do; it’s just long. you would want sm64’s economy to get restructured a bit to financially incentivise this kinda thing, and it would get done [12/12]
12/21/2023, 12:07:15 PM
Favs: 0
Retweets: 0
link