Loading demo…
Selected data projections, sampled model-layer activity and recorded examples of typing and mouse movement. The model view is a visual excerpt, not the full architecture; training and generation do not run in the browser.
October 2026 · Generative ML · Input dynamics
Human Input: motion & typing
A generator of mouse movement and typing rhythm with natural variation. Turns a target and text into a ready-to-use sequence of actions, with an API and an interactive demo.
Author and developer
The challenge
Build a system that generates motion toward a target and types text with a varied rhythm. Its output should be usable in automated interface workflows.
01 · Approach
From data to ready-to-use actions
Data preparation, neural generation and an API form one system.
- Prepared a research corpus with data quality checks and clustering.
- Developed and compared generative models for mouse motion and typing rhythm in PyTorch.
- Built an API, EN/RU/Unicode support and an interactive demonstration of generated scenarios.
- 01
Data
Preparation and quality checks
- 02
Models
Training and comparison
- 03
Integration
API and input planning
- 04
Demonstration
Visual replay of the result
02 · Mouse movement
From conditions to trajectory
A conditional spectral flow generates the entire path and duration. Three scales are trained jointly; diagrams show layers and dimensions, not learned weights.
Conditions
A shared representation of the movement conditions.
Coarse path
The overall path and movement duration.
Corrections
Path corrections conditioned on the coarse motion.
Fine detail
Fine detail conditioned on the previous scales.
One model, three scales
Coarse motion and time use 17 coordinates, corrections 32, and detail 140. Each scale transforms its own part of a random vector, conditioned on earlier scales. The inverse spectral transform reconstructs a 96-point path.
Temperature 0–1 controls variation. Geometry ensures target acquisition and screen bounds; a separate network adds reaction time and a target-width timing adjustment.
How realism was evaluated
New-motion distributions were compared, rather than matching one recorded path. Wasserstein W1 distances were normalized by the training-set interquartile range.
Metrics covered path length and shape, speed, acceleration, jerk, duration, turns, pauses and backtracking. Further checks examined temporal dependencies, micro-corrections and geometry at multiple scales.
Checking generalization
Splits grouped participants; normalization and transforms were fitted on fold-train only. Checks covered the train–validation NLL gap, feature correlations and proximity of generated samples to training examples.
Repeated evaluations informed model selection. They are neither an independent final test nor proof of human realism.
03 · Keystrokes
Rhythm with memory of previous keystrokes
The network reads the text and remembers previous keystrokes to generate hold times and intervals. The first three diagrams show generation; the fourth shows the training process.
Text context
Turns the text into numerical features: each key receives information about the surrounding characters.
Rhythm memory
Remembers previous hold times and intervals to carry the typing rhythm through the phrase.
Timing distribution
Estimates how long to hold each key and when to press the next one, with variation between generated sequences.
Learning from typing
A separate training block analyzes recorded typing times to teach the model variation in rhythm.
From one key to a sequence
The model generates a whole phrase with a connected rhythm: each new hold and interval depends on the timings already generated. A variation setting controls how predictable the sequence is.
Recorded timings are used for learning; new sequences are generated without looking at them. The planner assembles press and release events, allowing one key to be pressed before another is released.
How the model was selected
Experiments compared different ways to represent rhythm: short and long timing patterns, ordinary typing and pauses, and the influence of previous keystrokes. A change had to improve the output without degrading other typing properties.
Compared 14 characteristics, including holds, intervals, pauses, overlapping keys and dependencies between keystrokes. The selected model was also checked on recordings from 248 new participants.
Languages and scope
The rhythm was learned from English-layout typing. Russian letters map to the same physical keys; the planner adds Shift, layout changes and text composition for other Unicode characters.
Plan checks verify that the text is preserved and events are ordered correctly. Natural timing for each language still needs separate data and evaluation.
04 · Research results
The research in numbers
The complete workflow: from open data and model training to an API, multilingual input and an interactive demonstration.
- 308,191
- data episodes
- 19
- data clusters
- 12,855
- mouse-movement examples
- 4,980
- typing sequences
- 14
- mouse-research iterations
- 7
- typing-research iterations
- 585,545
- mouse-generator parameters
- 58,773
- typing-generator parameters
7 open data sources
Patterns in movement and typing rhythm
Training set · 82 participants
Training set · 343 participants
From the baseline to the final generator
Architecture, rhythm memory and loss functions
Trajectory, corrections and duration
Holds, intervals and timing dependencies
Data preparation
Combined open datasets, checked recording quality and identified characteristic movements and typing rhythms. Prepared data for training and evaluating the models.
Two research tracks
Developed generators for mouse trajectories and keystroke rhythm. Iterated on architectures, data representations and loss functions, comparing the quality of generated movement and typing.
A working system
The API returns trajectories, clicks and keystroke sequences. The planner supports EN/RU/Unicode, while an interactive demonstration makes the generated behaviour visible.
Engineering takeaway
Context
A research product for reproducible input simulation, covering data preparation, model training, an API and an interactive demonstration.
- Mouse models were compared on separate groups of participants; typing was also checked on new people from the same data source. Other devices and languages require separate evaluation.
- The resulting plan combines neural generation with application-level execution rules.
- Results apply to the evaluated scenarios; integration into a particular environment requires its own checks.
Approach
Built a complete research workflow: corpus preparation, training, model comparison and integration. The API is separated from the models, while the demo shows motion, clicks and multilingual typing.
- Separated data preparation, model training and the generation of executable input plans.
- Compared candidates by output quality and consistency across evaluated scenarios.
- Checked typing rhythm on new participants and selected a model that balances pauses, typing pace and dependencies between keystrokes.
Lessons
- Generation quality should be assessed across results rather than a single training metric.
- Reproducibility and a clear interface turn an ML model into an inspectable applied tool.
Stack
- Python
- PyTorch
- NumPy / SciPy
- PyArrow
- scikit-learn
- FastAPI
- Vue / SVG