PUB - Publikationen an der Universität Bielefeld

Entering confidential information is the classical application for optical skimming. For example entering a PIN or password to authenticate banking transactions is one of the most sensitive moments of a transaction. Up to now it was unclear how efficient optical skimming attacks can be mounted even on mobile devices. In this paper we show that filming the entering of a password with the camera of a standard mobile phone is enough for a fully automated recovering of the sensitive information. Our analysis method of the recorded video stream leads to a success rate of more than 90%. In our model a user enters his password into an Android or IOS driven touchscreen device while being filmed by the attacker’s smartphone. The goal of the attacker is to derive the password from the movie instantly, e.g. to use it in a real time man-in-the-middle attack. On first sight such an attack seems hard to mount due to many disturbing factors like movements of the device, bad light conditions etc. However, we show that many of these disturbing factors can be mitigated by smart video analytics. We implemented the whole attack in our lab simulating real conditions. Parts of the setup where a Samsung S7 for filming and an iPhone 6 as target. For real time processing we use the computer vision library OpenCV which supports most of the common image and video processing algorithms. Our goal is to cover most of the important cases while holding as little knowledge as possible about the video the algorithm has to work on. Starting from movable or instable position for camera and device we apply a discrete Fourier transformation to obtain a constant plan view as starting point for our algorithm. To obtain the necessary information for this the user has to select the four corners of the keyboard which is observed by hand. In real live situations the device is hold by hand of the user so we decided to use the MOSSE algorithm for stabilization to deal with the problem of a jiggling device. The keystroke detection is based on the “pop up” feature of common smartphone keyboards. This event is remarkable enough to get detected and located by simple gray value subtraction of following frames. Not only the magnified keys are the primary changes we observed and analyzed. Minor changes like noise can be removed by sufficiently blurring the frame. Subtracting areas where normal skin colors are detected yields information about the position of the keystrokes. The relative position obtains a probability for the pressed keys by analyzing the activity of the regions where the “pop ups” usually are. Our work showed that optical skimming of passwords are practical even on mobile devices. We show that it is possible to make a simple attack with basic computer vision techniques. Up to date mobile devices have enough computing power to provide for this kind of attacks. We give recommendations for entering passwords securely.