PUBLIC

NXP, THE NXP LOGO AND NXP SECURE CONNECTIONS FOR A SMARTER WORLD ARE TRADEMARKS OF NXP B.V.

ALL OTHER PRODUCT OR SERVICE NAMES ARE THE PROPERTY OF THEIR RES PECTIVE OWNERS. © 2022 NXP B.V.

J U N E 2 0 2 2

BL Edge Processing

NXP EDGEREADY

SMART TOUCHLESS HMI

SOLUTION OVERVIEW

1PUBLIC

i .MX RT117H MCU BASED SMART TOUCHLESS HMI SOLUTION

i.MX

RT117H

K32W41A

M/061

IW416

Machine Vision, Voice Control and Graphics Display UI all on one MCU Platform for Advanced and Innovative HMI

Image Sensor

RGB 720p

Image Sensor

IR VGA

Digital

Microphone

Digital

Microphone

Speaker

Flash

SDRAM

MikroE Click™

expansion header

Status Dates

Customer Sample Mar 2022

Market Launch Jun 2022

Solution Kit: SLN-TLHMI-IOT

RSL $299

Graphics Display User Interface

• 5.5-inch MIPI LCD display at 720*1280 resolution

• 2D graphics accelerator supported by LVGL

• GUI demo for Coffee Machine, Smart Home Panel and Elevator

Machine Vision

• 720p RGB and VGA IR Image Sensors

• Support user identification through face recognition

• Support gesture control through palm shape recognition

Voice Control

• 2 digital microphones on board and another 2 more through expansion

• VoiceSeeker integrated to support acoustic echo cancellation, noise reduction, beamforming and barge-in

• VIT for wake word and local voice command (English, Mandarian)

• Cyberon for wake word and local voice command with more languages support and separated charge through 3rd party

Wireless Connectivity

• Dual band Wi-Fi 4 + BLE combo: IW416

• Cloud based device management, user data management, OTA, etc.

Misc.

• 32MB flash, 32MB SDRAM, 2W louder speaker

• MikroE Click™ Expansion Header, PIR sensor

2PUBLIC

F U T U R E P O S S I B I L I T I E S

Smart Touchless HMI Machine Vision Remote Conference Payment

• Large screen display with 2D GUI

• User identification

with face recognition

• Touchless control with gesture

recognition (hands, head)

• Touchless control with far field

offline voice control

• Wireless connectivity

• 3D or 2D camera input

• Object detection and/or identification

• Human posture or

drowsiness detection

• Wired or wireless connectivity

• Speaker identification

with face recognition

• Speaker face detection and tracking

• Far field full-duplex communication

• Far field voice control

• Wired or wireless connectivity

• Face recognition with liveness

detection for payment

• Wireless connectivity

• EdgeLock security for payment

SMART TOUCHLESS HMI TARGET APPLICATIONS

3PUBLIC

SMART HOME APP

4PUBLIC

SMART ELEVATOR APP

5PUBLIC

i .MX RT117H MCU-BASED SMART TOUCHLESS HMI SOLUTION

• Development kit SLN-TLHMI-IOT – $299

• MCU

− i.MX RT117x (1 GHz Cortex-M7 + 400 MHz M4 Cores)

− 2 MB on chip SRAM

• Camera(s)

− RGB + IR (optional) CMOS Sensors

• Radio

− Dual band Wi-Fi 4 + BLE (NXP IW416)

• Display driven by LVGL graphics

− 5.5” LCD Touchscreen (720 x 1280)

• Audio hardware

− 2x DMICs + Speaker + Amplifier

• MikroE Click™ Expansion Header

− E.g. add NFC, more DMICs, IEEE 802.15.4 radio (Zigbee/Thread/Matter), etc.

6PUBLIC

FACE RECOGNIT ION TOUCHLESS HMI SOLUTION DEVELOPMENT KIT (SLN -TLHMI - IOT)

HARDWARE BLOCK DIAGRAM

32 MB QSPI

NOR FLASH

32 MB SDRAM

PC200

QSPI SEMC

Antenna

WHITE LEDPWM

LED DRIVER

LOW COST

LDO(s)

USB-C (5V)

CONNECTOR

Wi-Fi/BLEIW416

24 MHz

CRYSTAL

USER/RESET

SWITCH

MCU

i.MX RT117H

PDM

3.3 V BUCK

REG

EXT. 2xMIC

HEADER

SDIO

UART

GPIO

PIR

SENSOR

RGB

STATUS LED

GPIO

GPIO

GPIO

IR LEDPWM

LED DRIVER

IMAGE SENSOR

(HIGH RES. RGB)

DIGITAL

MICROPHONE

DIGITAL

MICROPHONE

PDM

MQSAUDIO OUT

AUDIO

AMPLIFIER

MIPI-DSIHIGH RESOLUTION

COLOR DISPLAY I2C / GPIO

IMAGE SENSOR

(RGB)

EXPANSION

HEADER

UART SPI I2C

ADC PWM GPIOs

IMAGE SENSOR

(IR)

FLEXIO

I2C / GPIO

MIPI-CSI

I2C / GPIO

DVP-CSI

I2C / GPIO

7PUBLIC

Fre

eR

TO

S

OTA FW Update

Boot Loader

Face Detection

NXP Edge Ready Machine Vision Library

NXP Facial Biometric Authentication & Gesture Recognition API

NXP MCU inference engine

Face Alignment

Face Quality 2D Liveness Algorithm

Face Recognition

Memory Management

Min

i C

V

Beamforming

Echo

Cancellation

and Barge-in

Full duplex

speaker phone

Audio Front End Runtime Library

Automatic Speech Recognition

Wake Words & Commands

ML Speech Engine Runtime Library

Gesture Recognition

Hardware Abstraction Layer

Algorithm Manager

HAL

Output Manager

HAL

FrameworkAlgorithm

ManagerOutput

Manager

Multicore

Manager HAL

Multicore

Manager

VIT (Voice

Intelligent

Technology)

or

Hardware Abstraction Layer

Display Manager

HALCamera Manager

HAL

Framework

Display

ManagerCamera

Manager

Multicore

Manager HAL

Multicore

Manager

Input

Manager

Output

Manager

Power Manager

Manager

Input Manager

HAL

Application Control

GUI Shell Wi-Fi/BLE Voice Input

Camera

Fle

xIO

MIP

I-C

SI2

CS

I

I2C

Display

LC

DIF

MIP

I-D

SI

GPU

PxP

GP

U2D

LED

PW

M

Wi-Fi

SD

IO

BLE

UA

RT

Dual Core

MU

SE

MA

4

SH

AR

ED

ME

MO

RY

Touch

I2C

Dual Core

MU

SE

MA

4

SH

AR

ED

ME

MO

RY

DMA

PD

M t

o

PC

M

Speaker

MQ

S

CM7 Driver Layer

CM4 Driver Layer

MSD FW Update

FLASH

QS

PI

File-system

Serial-manager

lwIP

mBedTLS

MQTT

LVGL

VGLite

H.264

Multi-Core

Manager

Opus decode

Middleware

Output Manager

HALPower Manager

HAL

SLN-TLHMI - IOT ( i .MX RT117H) EDGEREADY SOLUTION SOFTWARE BLOCK DIAGRAM

8PUBLIC

SLN-TLHMI - IOT ( i .MX RT117H) EDGEREADY SOLUTION SOFTWARE ARCHITECTURE

IR & RGB

Frames

CM7 CM4IPC with Shared Memory

Framework

MQS HAL

Vision Algo HAL

Voice Algo HAL

Peripherals

Voice Algo

Result

Notification

Message

Framework

Camera HAL

Display HAL

Graphics HAL

LVGL & VGLite HAL

Vision Algo

Result

Display

Camera

PxP

Touch

GPU2D

Peripherals

Mic

Speaker

• CM7 (Vision & Voice algorithm accelerator):

− Framework

− Vision algorithm with VGA input frames

− Voice algorithm (AFE + ASR) with mic input

− MQS audio playback

− IPC communication with shared memory

• CM4 (UI & System control unit):

− Framework

− CSI/MIPI Camera preview @VGA

− LVGL GUI @720p with VGLite 2D GPU acceleration

− Vision algorithm input frames color space conversion with PxP

− Touch panel input

− IPC communication with shared memory

9PUBLIC

SMART TOUCHLESS HMI SOLUTION USE CASE EXAMPLES FROM

• Face recognition for user

identification & customization

• Hand position and

gesture recognition

• GUI display, voice control and machine vision enabled with single MCU solution

• Combines face recognition & voice control from i.MX RT106F & i.MX RT106S

showroom.nxp.com

1 0PUBLIC

NXP EDGEREADY FACE RECOGNIT ION SOLUTIONS – ADVANTAGES SUMMARY

Low cost• MCU implementation reduces BOM cost as much as 50% vs. MPU + Linux® solutions

• Not an add-on solution - allows developer to replace host MCU with i.MX RT

− Available MHz & memory, and ability to trade off inference time vs. CPU load

Quick & easy

to use

• Turnkey solution – examples of concept to production in only four months

• Familiar RTOS based environment for IoT developers

• No face recognition, speech recognition, audio or ML/AI expertise necessary

• One stop shop, no third-party software

• MCU + RTOS solution minimizes boot time, enabling deep sleep for long battery life

• Plug and play out-of-box-experience

• Worldwide distribution network for sales and support

• Easy to use development tools (MCUXpresso IDE, SDK, configuration & provisioning)

Flexible

• Choose between low-cost CMOS sensors or high-performance 3D SLM cameras

• Common platform for smart lock, touchless HMI, and more to come…

• Offline capabilities lower latency & total cost of ownership, while preserving privacy

• Remote registration capability available on mobile and PC platforms

NXP, THE NXP LOGO AND NXP SECURE CONNECTIONS FOR A SMARTER WORLD ARE TRADEMARKS OF NXP B.V. ALL OTHER PRODUCT OR SERVICE NAMES ARE THE PROPERTY OF THEIR RESPECTIVE OWNERS. © 2022 NXP B.V.