Modelling a smart environment for nonintrusive analysis of attention in the workplace

Nowadays, the world is getting increasingly competitive and the quality and the amount of the work presented are one of the decisive factors when choosing an employee. It is no longer necessary to only perform but, to achieve a product with quality, on time, at the lowest possible cost and with the minimum resources. For this reason, the employee must have a high score of attention when performing a task, and the factors that influence attention negatively must be reduced. This is true in many different domains, from the workplace to the classroom. In this paper, we present a nonintrusive smart environment for monitoring people's attention when working in teams. The presented system provides real time information about each individual and information about the team. It can be very useful for team managers to identify potentially distracting events or individuals because when the attention of an individual is not at its best when performing the proposed task, her/his performance will be negatively affected, with consequences for the individual and for the organization.

The recognition of these behaviours is already used in new technologies. For example, the recognition of users as a form of authentication on a device or software has done by tracking behavioural biometrics, such as fingerprints, face recognition, or iris recognition (Benevenuto et al., 2009).
According to Yampolskiy and Govindaraju (2008), this behavioural biometrics can be classified in five categories based on information type: • behavioural biometric based on the analysis of text extracts or drawings produced by the user; • behavioural biometric based on computer interaction; ○ device usage, such as mouse and keyboard. Devices that can capture the muscles' actions; ○ software interaction: strategy and knowledge; • behavioural biometric based on monitoring data coming from low level action in software, such as access log, storage activity, and calls systems; and • behavioural biometric based on data from motor skills.
One of the variables is attention. To recognize the lack of attention on user behaviour, different ways of capture and classify attention were discovered. One of the first ways to try to quantify the attention level was a questionnaire. This questionnaire is presented to the user. After the user answers the questionnaire, the study's author analyses the answers and presents the conclusions. However, this approach is more qualitative and depends on the author's interpretation .
The more quantitative approach is the one that use biometrics behavioural. It is an approach more focused on the perception stimuli.
Eye-tracking is one of the most used techniques. Using this technology when performing a task on a computer, it is possible to know the screen area where the eyes are directed and, consequently, where is the focus of attention. So it is possible to conclude if the user was with the visual attention directed to the screen area where the task is positioned (Duc, Bays, & Husain, 2008;. This is, however, a very intrusive methodology that raises privacy concerns, especially in the workplace.
Other approach is based on neural activity. The most used technique is electroencephalogram, which uses electrodes placed in the scalp that read the frequency of brain waves and, with the acquired data, analyse the brain activity during a task. In many studies, the most important component is mismatch negativity. This component is the indicator of a brain reaction to a preattention process. Other techniques used are functional imaging, functional magnetic resonance imaging, magnetoencephalography, functional imaging, and positron emission tomography scan .
Regarding techniques that measure attention, the first two techniques described are intrusive approaches. In the case of brain activity measures, it is necessary to place some devices on the user's body to capture the data. This can interfere with the performance of the user and cause stress as it is intrusive. Regarding eye tracking, and although it does not require the use of devices placed on the body, the technique is an approach that has raises issues related to privacy and data protection (especially with underaged individuals), which makes it unsuitable for academic environments.
In the case of the stress study , the data were captured during an online exam of medical students. The captured data were related to mouse movement and keyboard usage. After the data were collected, the data were analysed in two phases. In the first phase, it used statistic methods and the first conclusions were obtained. These data were analysed in two ways: to find common behaviour in groups of students; individual data were analysed for each student. In the second phase, it used a machine learning method to model the students' response to a stress event. There were two classifiers used: one to classify the all students' data and other to classify data from each individual student. This study  concluded that stress affected the students' performance on an exam but also pointed out that some students can improve their performance even with stress.
The mouse and keyboard tracking technique is a non-invasive approach because the data captured are compiled by a background software, and the user does not have the perception that he/she is being monitored. This is an advantage over the two previous approaches as the user is not affected in any way by the collection of data.

| Smart environments
Important features of the already defined smart environments include a certain degree of autonomy, the capacity to adapt themselves to changing environments, and the communication with humans in a natural way (Cook & Das, 2005). A smart environment is a tool in which technology is embedded, hidden in the background, sensitive, adaptive, and responsive to the presence of people and objects. This system also preserves security and privacy while using information when needed and with an appropriate context (Weber, Rabaey, & Aarts, 2005). This is achieved by creating digital environments that are sensitive to people's needs and can respond to their requirements, anticipate behaviours, and adjust the response accordingly (Costa, Julián, & Novais, 2017).
In this paper, we propose an adaptive system that aims at supporting the monitoring work process. This system is adaptable, which means that it considers the worker's knowledge, background, interest, goals, targets, and/or choices.

| A SMART ENVIRONMENT FOR ATTENTION MANAGEMENT
To implement the proposed approach, a smart environment for attention management was developed. The key element in this environment is a data collection tool that registers all the events describing the user's interaction with the computer. These events support the generation of 14 interaction features, described in Section 3.1. These features, describing the interaction of each user with each device, are processed, transformed, and sent to a server, which continuously builds a user interaction profile, as described in Section 3.2.
Part of this framework was developed in previous work. The first version focused on the analysis of the individuals' interaction patterns with the computer, including features such as mouse velocity or acceleration, click duration, typing speed or rhythm, among others. For a complete list of features and the process of their acquisition and extraction, please see Durães et al. (2016). However, a limitation was also identified in this previous work. In fact, a user that opened a no work-related application and did not interact anymore with the computer until the end of the task had 0% of attention. On the other hand, if the user opens a work-related application and does not interact with the computer after that, the user's attention will be classified as 100% when he/she is most likely not even at the computer.
The present work adds a new feature to this previously existing framework, by providing a precise measure of attention based not only on the key typing or mouse movement patterns but also on the actual application usage and switching patterns. It thus constitutes a much more precise and reliable mechanism for attention monitoring, while maintaining all the advantages of the existing system: nonintrusive, lightweight, and transparent.

| Features extraction
Although the user conscientiously interacts with the computer and takes his/her decisions and actions, a parallel and transparent process takes place in which the smart environment uses the information. The process of feature extraction starts with the acquisition of interaction events, which is carried out by a specifically developed application that is installed in each of the computers or smartphones. This application runs in

| Real time analytics
In the context of an organization, the gathering and analysis of metrics describing user behaviour and the providing of tools for visualization (particularly real time analytics) enables better decision making and data-driven actions that consider the state and well-being of each individual user. Such initiatives can nowadays be scaled to hundreds or thousands of users, through the use of big data tools and techniques, without compromising performance and availability. The features depicted in Section 3.1 describe the behaviour of each individual while interacting with the computer. Each of these instances also contains a timestamp.
From the architecture of the developed environment described in Figure 1, it is possible to collect data that describe the interaction with both the mouse and keyboard in the devices in which students work. These devices have software that generates raw data, which they store locally until it is synchronized with the web server in the cloud. This layer encodes each event with the corresponding necessary information (e.g., timestamp, coordinates, type of click, and key pressed). These data are further processed, stored, and then used to calculate the values of the behavioural biometrics. Mouse movements can also help to predict the state of mind of the user and keyboard usage patterns.
After the raw data are stored in a data store engine, the analytic layer processes the data received (from the storage layer) in order to evaluate those data according to the metrics presented. Some data preparation tasks are also carried out in this phase, such as removing outliers (e.g., the backspace key being continuously pressed to delete a group of characters is not a regular key press). The system receives this information in real time and calculates, at regular intervals, an estimation of the general level of performance and attention of each student.
In the classification profile layer, the indicators are interpreted. Based on data from the attentiveness indicators and building the metadata that will support decision making, the system will classify the user profile. When the system has a sufficiently large data set that allows making classifications with precision, it will classify the inputs received into different attention levels in real time, creating each student learning profile.
With these results, it is possible to obtain a profile of the learning style.
Finally, the user attention information is displayed in the visualization layer, and it can be used to personalize instructions according to the specific user, enabling the administrator to act differently with different users and to act differently with the same user, according to his/her past and present level of attention. In the visualization layer, it is possible to obtain some graphical modules that allow the display of information in an intuitive way to the user. This graphical module user interface is composed by a module that allows the creation of charts (CHART) and the layer that allows the creation of virtual teams (ROOM) so that the administrator may view intuitively the user's behaviour.

| EXPERIMENTAL STUDY
In our line of research, we have been studying ways to assess attention level. In the present work, besides developing and presenting a smart environment for supporting the whole process, we also analyse the potential influence of external factors on attentiveness. In the early version of the system, the server classified the level of attention of the user based on the interaction patterns. With this work, the server now has access to a new type of raw data denoting that a given user switched to a specific application at a given timestamp. Because the server receives these data at regular intervals, it receives a list of triplets for each user. Thus, the new raw data received at regular intervals is as follows: (Id; Username; Timestamp; [(AppName; Timestamp)]). Id denotes the unique identifier of this group of data, Username uniquely identifies the user, Timestamp marks the end of the time interval for this group of data, and the last element is a list of pairs containing, in each element, the name of the app that the user switched to and the timestamp in which this happened.
This environment is designed to be used in the workplace by team managers to analyse the evolution of the attentiveness of the group during the day or during wider time periods. It will allow establishing each individual's optimum working cycle, improving productivity and well-being.
The present work adds a new feature to this previously existing framework, by providing for each task the level of interaction of the keyboard and of the mouse. It constitutes a more precise and reliable mechanism for attention monitoring, while maintaining all the advantages of the existing system: nonintrusive, lightweight, and transparent.

| Study design
As previously mentioned, a system with these characteristics may prove useful in very different domains, including organizational, academic, or any environment in which people operate computers. To validate the proposed system, we have been using it for the past months in the Caldas das Taipas High School, located in northern Portugal. In the Portuguese academic context, this system gains increased relevance as current policies move toward the creation of larger classes, which make it increasingly difficult for the teacher to individually address to each student. In this section, we show several tools supported by this system that, when at the disposal of the teachers may allow him/her to • decide, in real time, which students to focus on, according to their level of attention; • evaluate, a posteriori, which contents are more prone to generate distraction, providing a chance for improvement; • identify, in real time, fluctuations in attention, improving decision making concerning aspects such as when to make breaks or when to dismiss the class.
In this work, we compare the same class from a vocational course while performing an activity based on Microsoft Access in a lesson. The goal was to determine the level of attention when measured with the work-related task and compared with mouse and keyboard interactions. With these results, the smart environment can support future decision making.
To validate this system, we are following several cohorts of students during their academic activities. This data collection process will allow assessing the influence on attention of aspects such as breaks, time of day, class contents, and class objectives, among others. For this purpose, a group of 14 (all girls) students were selected to participate in this experience. Their average age is 15.9 years old (SD = 1.5 years).
The experiment was applied in a lesson, where they had access to an individual computer and 100 min to complete the task. Students received, at the beginning of the lesson, all necessary data with the goals of the task. For this class, the lesson started in the afternoon and students received, at the beginning of the lesson, a document with the goals of the task, which in this case required the use of Microsoft Access and Adobe Acrobat Reader.
This application runs in the background, which makes the data acquisition process a completely transparent one from the point of view of the student. It collects data from the students' interaction with the mouse and keyboard, which act as sensors. The mouse and keyboard sensing layers are responsible for capturing information describing the behavioural patterns of the students while interacting with the peripherals.

| Data analysis
In this subsection, we show the existence of different behaviours in the class. To quantify attentiveness, the following methodology was followed.
Aside from capturing the interaction of the user with the computer, the monitoring system also registers the applications with which the user is interacting. We analysed all the applications used by all users and labelled each of them as belonging to the task or not. We then quantified the amount of time that each user spent interacting with applications related to the task versus other applications. In this sense, the data were analysed in two different ways: First, a general analysis was carried out in which statistics methods are used to obtain preliminary conclusions and second, an individual analysis was done to compare the different moments.
Algorithm 1 presents the code to obtain the list of pairs and computes the time during which each window was active in order to obtain the time that the user spent on each task. However, there are often cases in which the user does not change applications for a large amount of time. In these cases, which are represented by a pair with an empty AppName, the time is added to the last known AppName (because this means that the user is still interacting with it).
Algorithm 1 Creating triplets with the intervals and timestamp of each application.

Data:
appnamei To identify the work-related applications, we analysed all different applications used by all users and labelled each one as belonging to the task or not. We then quantified the amount of time that each student spent interacting with applications related to the task versus other applications.
For that, it was necessary to compute the level of attention of the user as detailed in Algorithm 2.
Algorithm 2 Creating triplets at regular intervals with the timestamp the quantification of work-related.

Data:
t -A list of triplets of the type (AppName, Timestamp, Duration).
st -The starting time of the task.
inter -The interval to update attention.
setthe set of regular expressions. Class = 10 N. Task1 = {"*access*", "*Acrobat*"}. To do this, we measured the amount of time, in each interval, that the user spent interacting with work-related applications. The algorithm thus needs knowledge about the domain in order to classify each application as belonging or not to the set of work-related applications. This knowledge is provided by the teacher and is encoded in the form of regular expressions. The teacher uses a graphical interface to set up rules such as "starts_with Access" or "contains Access", which are then translated to regular expressions that are used by the algorithm to determine which applications are work-related and which are not.
Whenever an application that does not match any of the known rules for the specific domain is found, the application name is saved so that the teacher can later decide if a new rule should or should not be created for it. By default, applications that are not considered work-related are marked as "others" and count negatively toward the quantification of attention.

| RESULTS
In this section, and as mentioned in Section 2.4, we focus on the new feature that was added to the previously existing framework. We present an example with all the necessary steps to calculate the level of attention of the users. Thence, we briefly analyse the data collected for the same cohort of students (10 N). First, it is necessary to know the interaction with the mouse and with the keyboard that each task will have.
In some tasks, the interaction with the mouse will be higher whereas in others that will happen with the keyboard, and finally in others, it will practically be the same level of interaction. However, it is difficult to know, a priori, what the exact percentage of interaction of the mouse or the keyboard will be. To know these values, we first count the number of times that each key is pressed and the number of times that the mouse is clicked down for each user. In Figure 2, we presented the results of the total number of keys pressed for the keyboard and the total number of mouse clicks for each user in the overall of the lesson. In this lesson, as mentioned in Section 4.1, the tasks defined were exercises in Microsoft Access.
Second, we calculated the percentage of the interaction of the mouse, which is the average number of times that the mouse was pressed in the class taking into account the total of interactions with the mouse and keyboard of the class. Similarly, we calculated the percentage of the interaction of the keyboard, which is the average number of times that the keys were pressed in the class taking into account the total of interactions with the mouse and keyboard of the class. Table 2 presented the average of the interactions of the mouse and keyboard in the class and the global average percentage of the interaction of the mouse (59%) and keyboard (41%) of the class in this lesson.
On the order hand, we obtained the amount of time that each student spent at the computer (task duration) and the amount (and percentage) of time that each student devoted to work and to other activities in the lesson as explained in the algoritm1 of the preview section. Table 3 allows the teacher to analyse this result.
However, the time that the student spent in the task-related does not indicate the level of attention in some cases because in some situations, the user might have opened the task-related and during that time did not interact with the computer. In this case, if the level of attention was measured only by the task-related, the student has 100% of attention, and in reality, his attention level should be 0%. For these situations, it is necessary to analyse the amount of interaction with the mouse and keyboard and cross these data.
The next step is to calculate the percentage of usage of the mouse for each user. The formula to calculate this value is where: Mouse_Down Count is the number of the times that the mouse was pressed by a user; MAX Mouse_Down Count is the higher score of usage of the mouse down in the class.    (2) where: Key_Down Count is the number of the time that one key was pressed by a user; MAX Key_Down Count is the higher score of usage of the Keyboard in the class.
Based on Formulas 1 and 2, the results presented in Table 4 were obtained, which contain the results for all students in this lesson.
Finally, in order to obtain the level of attention, it is necessary to combine the results of the interaction of the behaviour biometrics of Table 4 with the results showed in the time that each student spent in the task-related, presented previously in Table 3. This level of attention is the relative attention of each student, as this level of attention is compared with the other user of the class. The formula that calculated these values is where: Perc. Mouse is the percentage of utilization of the Mouse for a user; Perc. Keyboard is the percentage of usage of the Keyboard for a user; Work-related is the time of a user spent in the task defined for the lesson. Formula 3 provides the results for each user, presented in Table 5. Figure 3 presents the graphically result between percentage of time interacting with work-related applications and relative attention. It can be observed that the relative attention decreases in general, because the system takes in account the interaction of the user with mouse and keyboard. That is, someone who spent 100% of the time using the application that was supposed to but was interacting only 50% of the time, would have a score of attention of 50%.

| DISCUSSION AND CONCLUSIONS
The task data from the user's interaction with the computer are the most crucial information because it derives most part of the attention level. To obtain the task results, the task rules received in the request are used to get a perception of how much time the user has spent on the applications related with the task rules.
As far as the mouse and keyboard results are concerned, their role is present on how the user interaction is occurring and it helps the manager/ teacher to understand if any user, who has the application active, is really working on it or not. To do that, we have used the two features one from the keyboard (key down time) and other from the mouse (time between clicks).
Concerning attention, it is an important theme because it is one of the factors that most influences a person's performance while performing a task. Therefore, it is a much-studied theme by several areas, such as psychology, neuroscience, and computer science. Understanding how attention varies and in which situations the attention varies for each person, it is possible to act in the right moment and right time, to bring the user's attention level to the ideal value.
In order to obtain better results, it is necessary to supervised users over a long period of time in order to create a more reliable data. Using this, the manager may take different measures depending on the profile of the group and/or profile of each user. The classification layer has access to the current and historical state of the group from a global perspective but can also refer to each user individually. After the classification, the enhanced user behaviour profile is updated in the Database Behaviour Profile.
A framework was proposed to address these issues, especially to monitoring user. Narrowing the scope of the study, a process to detect attentiveness was proposed, through the use of a developed log tool. With this smart environment, it is possible to detect potentially negative factors dynamically and nonintrusively, making it possible to foresee negative situations, allowing to take actions to mitigate them. This may, in turn, minimize issues such as stress and anxiety, which can negatively influence the user's results and are closely related to the occurrence of conflicts. The work developed so far resulted in a useful system for the team manager, who can monitor, in real time, the level of attention of users.
The door is thus open to intelligent platforms that allow to analyse user's profiles, taking into account their individual characteristics, and to propose new strategies and actions. By providing managers with access to this information, we allow them to better manage their interactions with the users, namely, by pointing out the most problematic cases of inattention in real time.