Łukasz Roth

Speak to Pick

Inventor, Lead Interaction Designer

I developed a new method for interacting with smart home devices using voice and gestures through a smartwatch. This innovation, for which I am a co-author of four European patent applications, was showcased at the Orange Research exhibition in Paris.

  • UX Design
  • Strategy
  • UX Research
  • Usability Testing

Background

Voice assistants gained significant popularity, leading major companies to integrate them into their devices and services, with options like Siri, Alexa, Google Assistant, Cortana, and Bixby becoming ubiquitous. However, the accuracy of these assistants in understanding human commands proved to be far from perfect. Users frequently experienced frustration as their spoken commands were misunderstood, leading to repeated attempts and tiring follow-ups to correct errors. This persistent issue of miscommunication made the experience more frustrating than helpful for many users.

A person speaking to a voice assistant
47%questions are answered correctly by Siri

Solution

I designed Smart assistant interface on a smartwatch that shows user commands in real life as they speak. The text is then translated into visual symbols in split seconds.

This allows user to clearly see if the command was understood correctly, and fine tune the settings for the devices they want to control.

How it works: say what you need, your command is displayed, it is translated into pictures, you can see other relevant options, press “do it” and it’s done

Prototype

Fully functional apple watch app integrated with selected Smart Home devices

The smartwatch prototype controlling smart home devices

Design Resulting in Four European Patent Applications

Patent drawing: a command “Could you turn on the light in the bedroom?” shown as pictograms, with reference numerals 302.1, 302.2 and 302.3

User Input processing for controlling remote devices

Europe EP15306051.2 Issued June 30, 2015

European Patent Application for Human interaction with remote devices (Internet of Things)

Enriched instructions for remotely controlling electronic devices

Europe EP15306052.0 Issued June 30, 2015

European Patent Application for Human interaction with electronic devices such as laptops, touchpads, mobile phones, Smartphones, wearable devices or smart watches.

Remote control of an electronic device with a selectable element

Europe EP15306045.4 Issued June 30, 2015

European Patent Application for a method of managing a remote electronic device using an electronic device, a database associating argument types with alternative argument values and comprising a compatibility mesh between the alternative argument values for a given type and contextual data.

Updating of former instructions to control a remote electronic device

Europe EP15306050.4 Issued June 30, 2015

European Patent Application for Human interaction with remote devices (Home Automation, Internet of Things)