Digital twin technology uses virtual assets in virtual space to replicate the characteristics and behaviour of physical assets in real space to deliver outcomes utilizing data and information that connects both the physical asset and its corresponding virtual asset. Various industries, including manufacturing, transportation, and healthcare, have recently used digital twin technology. One area of healthcare where digital twin technology is up-and-coming is smart glasses.
A digital twin of glasses refers to a virtual representation of a pair of glasses that can be used for various purposes, from ideation to commercialization, such as product design, manufacturing, and maintenance. For example, an eyewear manufacturer may use a digital twin of a pair of glasses to simulate the manufacturing process and identify potential issues before physical production begins. In maintenance, a digital twin can also simulate the wear and tear of glasses, allowing for proactive maintenance and replacement of parts before they fail.
In the context of low-vision users, smart glasses are wearable device that helps users with significant vision loss achieve the enhanced vision (visual acuity) to help improve daily functioning. The smart glasses are equipped with sensors and cameras that capture the user’s surroundings and display the information on a small screen in front of the user’s eye. This allows the low-vision user to see more clearly and navigate their environment more easily.
An exciting and practical application of digital twin technology is to provide live remote support to low-vision (legally blind) users of smart glasses. Using digital twin technology, it’s possible to create a virtual replica of smart glasses that can be accessed remotely. This allows the remote support team to monitor the smart glasses in real-time (live video feed of what the user sees), detect any issues, and troubleshoot problems. The remote support team can also update the software, firmware, and settings of the smart glasses remotely and adjust the device’s configuration. Additionally, the digital twin also allows the collection of data and analysis the usage patterns, which can help to improve the performance of smart glasses over time.
Two primary personas providing live remote support to low-vision users are caregivers like friends and family, care providers, and the backend support team to set up the device (enhancing the onboarding experience) and providing ongoing support during the lifetime of the smart glasses. Seeing what the user sees is a game changer in helping calibrate and configure the device per the end user’s needs (user personalization). Also, caregivers can see in real-time what their loved ones are seeing and can help support them, like providing navigation support in an unfamiliar environment.
From a technical perspective, live remote support can be implemented using a variety of well-established protocols such as WebRTC, Websockets, REST, and MQTT. These protocols can support different types of communication, including duplex, publish-subscribe, and request-response. Combining these protocols can provide a robust and flexible solution for remote support. Many cloud providers also offer off-the-shelf services that can be easily integrated to implement a live remote support solution. One such service offered by Amazon is known as Kinesis Video Streams. Remote connection to the smart glasses, usually behind NATs over the internet, is achieved using STUN, TURN and ICE.
However, there are some challenges to implementing remote support, particularly regarding the limited resources of smart glasses, such as computing and memory limitations. Additionally, in specific regulated industries, such as healthcare, there may be strict firewall restrictions in place that can pose further challenges to implementation.
Providing live remote support is also subject to the regulatory compliance laws of the user’s country, such as the General Data Protection Regulation (GDPR) in Europe for EU citizens. These laws cover many areas, including obtaining user consent before initiating remote support, ensuring secure data streaming, and providing audiovisual cues to inform users and people in the vicinity of the live video streaming. For example, a beep sound or flashing LED can alert users that remote support is taking place and indicate when the process is complete. It’s essential to comply with the legal requirements of the user’s country to protect the user’s privacy and ensure that remote support is provided securely and transparently.
Overall, the integration of digital twin technology in live remote support for smart glasses for low-vision users can help improve the functionality and reliability of the device, provide real-time live support to the user, greatly benefit the support team and enhance the user’s experience.
