Framework for Posture and Face Recognition using Kinect an ambient-

intelligence method

LIVIU VLADUTU, COSMIN MARIAN

Faculty of Automatic Control and Computer Science

University “Politehnica” of Bucharest

Address Splaiul Independentei no. 313

Sector 6, Bucureşti, 060042

ROMANIA

more Kinect sensors, data acquisition stations, and crunches (data servers).

The articles tackle two of the most important issues for ambient intelligence: accurate determination of the

person and his or her facial landmarks (to understand emotions) and fast and reliable posture determination (to

take appropriate measures in case of an accident).

Key-Words: - face detection and recognition, postures classification, Kinect, Ambient-Intelligence, Open

Source, Ambient Assisted Living, sensor networks, facial landmarks.

Received: February 26, 2022. Revised: November 22, 2022. Accepted: January 15, 2023. Published: February 22, 2023.

fertility, and a certain decrease in the mortality

rate (life expectancy has increased all over

Europe, Japan, and North America). We

designed a data pipeline for online acquisition,

processing, and storage of localization and

tracking of an older person in a room, as

previously described in [3] and [1], in order to

deal with an increased population of elder

inhabitants and the need for constant care and

supervision (for example, to avoid unwanted

supervision adapted for people with special

needs (older people, persons with disabilities),

from a clinical and non-clinical perspective.

Typical emotion expressions are seen in various

voice, and behavior and affect physiological

parameters such as the heart rate or the body

temperature."

Our paper is organized into the following nine

main chapters: The second chapter briefly

introduces the organization of the Ambient

Intelligence Laboratory from the Faculty of

Automation and Computers at the Polytechnic

University of Bucharest, showing the physical

structure of the laboratory and the main

the posture of the persons that are "seen" by the

Kinect sensors.

Chapter 4 continues with a summary

introduction of our proprietary skeleton viewer,

written in Scala, a new programming language

that smoothly integrates features of object-

oriented and functional languages, enabling

Java and other programmers to be more

productive.

setup of the laboratory, including wiring,

computers, cabled and wireless networks and

carried out at the Polytechnic University of

Bucharest's Ambient Intelligence Laboratory

(AmI-Lab). The laboratory contains both a

hardware platform and a software platform [15],

as extensively presented elsewhere: [1]. The

hardware platform consists of nine Kinect

cameras placed around the room, data

acquisition computers, and network and data

processing computers.

and data crunching PDUs, which use face

recognition and voice recognition webservices

of the window of the application is presented in

Figure 3. The user can load a file of data

consisting of measurements in JSON format

([1] and [8]) with 3D skeleton data and RGB

data captured by the video camera. At the

bottom of the window, there is a slider that

allows the user to go from frame to frame and

inspect and annotate the data.

The window contains a 3D visualization area

corresponding buttons, the user annotates a

certain skeleton to have that posture. The

posture list is loaded from a text file, making

the application itself independent of the

considered set of postures. In the bottom left

corner, the user has a list of the features

considered, showing the value of each for the

current selected frame. He can select certain

features in order to display a color-coded map

of the value of that feature under the slider (see

Figure 3).

understand the data.

image as JPEG with the Base64 encoding, and

for skeleton, the 3D and 2D positions of joints).

viewer application and annotate the relevant

skeletons with the correct postures. It is

important to note that some postures were not

considered to match any of the postures in our

set, so they were not annotated. The annotated

single file. Each line of the file contained a

labeled measurement in the JSON format

containing the value of each feature, the label

associated with it, and the timestamp. A total of

2527 postures were annotated in a file of 1.6

MB. The next step was to write a Python script

that loads the data and uses the scikit-learn

with a split of 75% of the data for training and

25% of the data for validation. Scikit-learn is an

open-source, simple, and efficient tool for data

mining and data analysis built on NumPy,

SciPy, and Matplotlib; see [10], [11].

We have found this tool very easy to use for the

tasks we face, allowing us to write only 105

lines of Python code for loading the data,

applying and validating the learning algorithms

on the data, and displaying the results. The

Scala code for our viewer will be uploaded to

recognition we used >320 images with

approximately 175 for testing. There were 8

studied and faster implementations are to be

written in C++ (see last chapter), using

OpenCV and gcc (work in progress). On all

acquisition stations, we’ve used OpenCV

(version 3.4.3) and Ubuntu 18.04 and 22.04

(LTS) as operating systems. The code for face

detection and a new folder, "FaceRecognition,"

have been uploaded to GitHub in the

"FaceDetect" folder [15].

That folder also contains the code for

7. Experimental Results

As we already mentioned, from 3736 postures

gathered we annotated 2527 postures with the

1209 postures were found not to match any of

the classes. For the purpose of supervised

learning, we have chosen to test four learning

algorithms already implemented in scikit-learn.

For each algorithm, we computed a score given

performance of the results, we can conclude

that we obtained good results for the framework

we proposed and for the relatively short list of

postures we considered. Even if the list of

(Table 8).

8. Related Work

1. human gesture recognition for AAL (ambient

assisted living) scope (our aim);

for the acquisition and classification of human

postures; we haven’t used any commercial

software whatsoever; all the code was designed

to attenuate a (simulated) fall. These are the

main conclusions about the importance of

AmI-based elderly monitoring: