A NEWT Lab research thrust, from fingertip to gaze
The Internet of Bodies
The richest signals in any room come from people: where a hand presses, where the eyes rest, how a body moves. We build sensing that reads them: woven into fabric, riding on everyday wearables, and open-sourced for everyone.
Touch, gaze, and motion are the interfaces people already use with the physical world. Instead of asking humans to speak computer, this thrust teaches computers to read human: tactile textiles that feel, gaze that prompts, and wearables that know your pose.
A glove that feels
A live tactile sensing glove: press it and watch the pressure map, or replay a grasp.
Press and drag on the glove; you are the object it feels. Pressure fades the way real contact does.
Under the fabric is a matrix of resistive taxels: pressure-sensitive pixels. Push anywhere and the local resistance drops; a small readout board scans the rows and columns and streams the whole pressure image wirelessly, frame after frame, like a camera for touch.
Tactile images are surprisingly eloquent. The panel on the right is doing what our systems do at full scale: counting contacts, tracking force over time, and recognizing the pattern of a grasp. A pinch reads differently from a power grip, a keystroke from a caress, an empty hand from one holding a mug. From patterns come applications: activity recognition, rehabilitation, robot teleoperation, and interfaces that never ask you to hold a controller.
The WiReSens Toolkit open-sources this entire stack (sensor layouts, wireless readout hardware, and software) so that building a tactile interface costs an afternoon rather than a lab. It earned the Best Demo Honorable Mention, Jury's Choice, at ACM TEI 2026.
The thrust
Touch, for everyone. Tactile sensing has been trapped in robotics labs by cost and wiring. The WiReSens Toolkit makes wireless, battery-powered tactile textiles (gloves, sleeves, mats, whatever the application wants) buildable from open designs, and its software adapts the readout to each sensor so the data is clean out of the box.
Gaze, as an implicit prompt. Your eyes annotate everything you read: where you linger, where you skip, where you stumble. GazeSummary feeds that signal to large language models so a summary emphasizes what you cared about, with no extra effort from you; the same gaze stream detects the words an English learner doesn't know, before they ask. Sensing becomes personalization.
Motion, from what you already wear. UltraPoser squeezes full-body pose out of the ultrasound and inertial sensors in consumer earbuds and watches, with no camera and no suit, pushing wearables from step-counting toward body understanding.
The through-line is implicit interaction: interfaces that read the body's own language of pressure, gaze, and motion, so people can stop translating themselves for machines. This thrust is joint work with the labs of Yiyue Luo and Shwetak Patel at UW.
Publications in this thrust
Lab members in bold. The full lab record lives on the main page.
GazeSummary: Exploring Gaze as an Implicit Prompt for Personalization in Text-based LLM Tasks
Poster: Unknown Word Detection for ESL Learners Using Gaze and Pre-trained Language Models
UltraPoser: Pushing the Limits of IMU-based Full-Body Pose Estimation with Ultrasound Sensing on Consumer Wearables
Open source
The WiReSens Toolkit is free to build on, from sensor layouts to wireless readout hardware to software: github.com/WiReSens-Toolkit.
Tactile sensing for the cost of an afternoon
Recognition
Best Demo Honorable Mention, Jury's Choice, at ACM TEI 2026 for the WiReSens Toolkit, and gaze-driven personalization at ACM HotMobile 2026.
The interaction thrust, recognized
Work on the Internet of Bodies
PhD applicants and UW students: the lab is recruiting.