Computer Control with Face and Eye Movements Using Deep Learning and Image Processing Methods

Abstract

Today's computing is one of the basic needs of every human being. Many actions are done with the mouse in the use of the computer. Individuals with physical disabilities, paralysis from the neck down, or ALS patients who have difficulty in making physical contact with the computer are having difficulty using computers. In this study, an artificial intelligence-assisted system has been developed for these individuals, where they can control the mouse with head and eye movements. In this system, facial movements and eyes are detected in real-time through the library of Haar Cascade, Dlib, and Open CV from the images acquired through the camera. When Haar Cascade is used to detect the face region, the Dlib library is used to acquire right and left eye region images from this detected face image. These eye areas are provided as an introduction to the CNN model, which is trained with 2874 eye data (https://github.com/iparaskev/simple-blink-detector), and it is determined that the eye is closed or open. The CNN model 1500 is trained on a public eye image dataset representing open and 1374 closed-eye conditions. The left eye closed and opened state allows the mouse to click left and the right eye to close and open, and the right mouse to click. In addition, the location of the face detected with Haar Cascade is used to model mouse motion. The developed system is a real-time hybrid system with a combination of different methods and has been tested on different users. According to the test results, it was observed that the system correctly identified the eyes and the closed state of these eyes, classifying the blink event with CNN in both eyes correctly. However, it has been determined that there has been a slowness in modeling mouse movement or a poor fit to facial movement. The next study will focus on this issue and improve it by fine-tuning the system with data from many people.