How it works

Research → prototype → study → real-world testing

Feel the Ball, EnAct, and Just Look each began with a real barrier to participation. Here is how each one works, from the user need and research question, through the technical approach and working prototype, to its official ISEF record and the priorities that come next.

01 Sense · Shared experience

Feel the Ball

Haptic access to live ball movement
Current stageValidated with users
01 · User need & question

Beyond captions and audio

Commentary and audio can describe a match, but they don't convey the ball's continuous spatial movement, which is an essential part of how play unfolds and where its excitement lives.

Research question: Can fast ball motion in broadcast video be tracked reliably and translated into intuitive force feedback a fan can feel?
02 · Technical approach

From broadcast motion to touch

A Swin Transformer combined with DeconvNet and Long Short-Term Memory tracks small, blurred, fast-moving balls. A psychology-guided haptic method then translates that motion into force feedback through a joystick.

Student contribution: Developed the computer-vision tracking approach and psychology-guided haptic translation that turn spatiotemporal ball motion into force feedback through a joystick.
03 · Prototype in action

Working prototype

Small, fast ball motion tracked from broadcast video and translated into dynamic force feedback through a joystick.

Explore the Feel the Ball product →
05 · Product next

Next steps toward deployment

  • Recruit fans to test the deployed system live and expand on our validated results.
  • Harden tracking reliability and end-to-end latency across more sports, broadcasts, and difficult motion.
  • Line up venue and broadcast partners for a production pilot.
02 Act · Physical interaction

EnAct

Safety-aware guidance for everyday reaching
Current stageValidated with users
01 · User need & question

The everyday reach problem

Scene descriptions can say what is nearby, but not how to reach it safely. Without precise spatial guidance, a simple reach can knock objects over, create spills, introduce safety risks, or require help from another person.

Research question: Can vision-language reasoning and depth perception turn scene understanding into a safety-aware reaching path a person can follow?
02 · Technical approach

From description to action intelligence

Before movement, individual vision-language agents handle user-query reasoning, spatial reasoning, safety assessment, and path planning. During movement, YOLO8-World and Depth Pro check reach status, avoid collisions, and correct direction until the target is reached.

Student contribution: Developed the safety-aware actionable-guidance system, including the multi-agent reasoning architecture and the real-time reach checking, collision avoidance, and directional-guidance pipeline.
03 · Prototype in action

Working prototype

Add EnAct demo video

Multi-agent vision-language reasoning working with visual and depth perception to plan, monitor, and correct a safety-aware object-reaching path.

Explore the EnAct product →
04 · Recognition & publications

ISEF record · SOFT039

View the EnAct ISEF record
05 · Product next

Next steps toward deployment

  • Recruit vision-impaired volunteers to test the deployed system across everyday daily-living tasks.
  • Harden path accuracy, latency, and recovery when detection or depth estimates are uncertain.
  • Finalize privacy, fail-safe, and human-override safeguards for real-world use.
03 Express · Communication

Just Look

Object-gaze communication built around intent
Current stageValidated with users
01 · User need & question

Communication beyond letter-by-letter typing

Gaze keyboards ask people with ALS to spell thoughts one letter at a time. The process is slow and exhausting, and even predictive text keeps communication centered on character entry rather than human meaning.

Research question: Can looking at a real object communicate intent more directly than entering language one character at a time?
02 · Technical approach

Ground, reason, then clarify

The system grounds a fixation in the live scene, produces a small set of plausible intent hypotheses, and ranks them using device state, time, ambient cues, and interaction history. When ambiguity remains, a two-stage confirmation resolves it with one additional glance.

Student contribution: Designed the object-gaze system around structured scene grounding, context-aware intent ranking, and a two-stage confirmation step for resolving genuine ambiguity.
03 · Prototype in action

Working prototype

Add Just Look demo video

A gaze on a real object grounded in the scene, interpreted through context-aware intent reasoning, and clarified with an additional glance when needed.

Explore the Just Look product →
04 · Recognition & publications

ISEF record · SFTD028

View the Just Look ISEF record
05 · Product next

Next steps toward deployment

  • Recruit AAC users to test the deployed system across everyday communication tasks.
  • Broaden compatibility across gaze-tracking hardware, environments, objects, and lighting conditions.
  • Finalize privacy controls for contextual signals and interaction-history data.
What's next · real-world testing

Studied, and ready for real-world testing

Each product has gone from working prototype through initial human-subject studies. The next step is putting the deployed tools in real hands. If you are blind, low-vision, or nonspeaking, you can volunteer to test what we've built and help shape where it goes next.