Haptic Monitor

The ask

How might we enable creatives working with media to design haptics for video?

Challenges and assumptions

Audio designers and video editors presumed to be ideal candidates, but MVP timeline likely will only support one. As a result, we need to determine the fastest way of interfacing with established workflows. We decided to focus on enabling audio designers in Avid Pro Tools due to the belief that audio and haptics go hand in hand.

Initial results from research

We initially attempted automatic auto-to-haptics conversion as part of a Pro Tools plug-in. Unfortunately, the results could never replicate quality of hand crafted effects. As a result, we decided to focus on the Haptic Monitor application to allow audio designers to feel haptics in real time as they were designed.

The MVP was analog

With automatic conversion not viable, the need was to empower audio designers to create haptics in their DAW. The challenge was clear: you have to experience what you create in realtime. Audio is simple but haptics are touch, which requires the user to move their hands away from their computer. We also did not have time to make custom computer drivers, so we came up with an elegant solution: we would convert audio to haptics in realtime on an Android device. This lets us ship just an app while using off-the-shelf audio parts to route a mono audio signal from the user's computer to the mobile phone.

Caveats led to design decisions

This did present some challenges: audio in this way would require calibration. If the signal was too high, it would wash out effects. Too low and strong effects would feel weak on the device. To remedy this situation, we made two design choices: 1. we would support a calibration option, with clear signal when calibration was good or not, and 2. the style of the app would mimic a VU meter. This would allow us to give a sense of signal strength in a familiar way that audio designers already knew how to use. With audio signal, there is always risk that volume might be increased with the mistaken turn of a knob. The meter visualization would also provide error states in the event we detect a maximum signal greater than what the calibration expects. The final challenge of this version is that the haptics felt were slightly different from their encoded due to the analog signal and the method used to render haptics.

Version 2 - The change to digital

The change to digital removed the value of the VU meter visualization. At this time, Google's Material Design language was the preferred style for the application to follow. The digital version of the monitor required a custom MacOS driver, which could be confusing for some users. To resolve this issue, the disconnected state of the application showed an illustration of the driver's popover with instructions to connect. The data connection also solved 2 major pain points with version 1: battery drain and effect quality. Connected to the computer meant the battery would be charged and the digital signal meant effects felt more true to their encoded versions.

Version 3 - Connecting to the cloud

With the completion of version 3 of Haptic Studio Mobile (formerly Haptic Monitor), the end-to-end
workflow for haptic content was completed. On-boarding flow tested well for introducing new users to the application, as well as reintroducing those coming from previous versions. It has been employed for haptic advertising, game effect prototyping, and UI effect prototyping. Creators are easily able to design, encode, and share haptics.