Evaluation of the Brain-Teleoperated Robot

We outline a general evaluation of the teleoperated robot and its two main functionalities: the brain-computer system and the robotic system.

Overall system

Following [19], [18] we propose the subsequent metrics for the study:

  • Collisions: number of collisions.
  • Path length (m): distance traveled by the robot.
  • Time (sec): total time taken to accomplish the task.
  • Missions: number of missions to complete the task (a mission is an action sent to the robotic system).
  • BCI accuracy: recognition rate of the BCI system.

METRICS TO EVALUATE THE OVERALL SYSTEM
Task 1 Task 2
min max mean std min max mean std
Path length
 10.99 13.53 11.84 0.90 19.68 21.83 20.68 0.63
Total time
 685 1249 918 163 706 1126 910 154
# of missions
 12 19 13.9 2.3 10 15 11.7 1.6
BCI accuracy
 0.83 1.00 0.92 0.07 0.78 1.00 0.89 0.07

All the subjects solved two times each task demonstrating that they were able to combine the navigation and camera control capabilities of the device. There were no collisions. The path length and the number of missions were similar for all the subjects in the two tasks, which indicates a similar performance. The variability of the total time across subjects is significative since the number of stimulation sequences of the BCI changed among them. This is beause the number of sequences had to be customized for each subject to achieve a minimum of BCI accuracy (more sequences involves more accuracy, but also more stimulation duration). The BCI accuracy on average was always above 90% (just one subject reached an accuracy below this).

In conclusion, the results suggest a high performance of the brain-teleoperated robot. Notice that both tasks were designed to test the combination of both teleoperation modes in different working conditions (navigation in constrained and open spaces; and visual search of one or two targets that do not fit in the inital camera field of view).

Brain-Computer System

We divide this evaluation in two parts: an evaluation of the pattern recognition strategy and an evaluation of the design of the visual display. Based on [20], [19], we propose the next metrics to assess the pattern recognition strategy:

  • Total BCI errors: incorrect selections.
  • Useful BCI errors: incorrect selections that user decided to reuse to accomplish the task.
  • Real BCI accuracy: BCI recognition rate.
  • Practical BCI accuracy: BCI recognition rate (computed using the correct selections and useful errors).

METRICS TO EVALUATE THE PATTERN RECOGNITION PERFORMANCE
Task 1 Task 2
min max mean std min max mean std
Total errors
 0.81 3.00 1.50 0.92 0.73 6.00 2.06 1.91
Total time
 0.00 2.00 0.60 0.84 0.00 5.00 1.20 1.81
Real BCI accuracy
 0.81 1.00 0.90 0.08 0.73 1.00 0.86 0.09
Practical BCI accuracy
 0.83 1.00 0.92 0.07 0.78 1.00 0.89 0.07
Transfer rate (bits/min)
 8 14 11 2 9 12 10 1

The convention [21] used to assess that a person is able to use a BCI is when his accuracy is above 80%. In our experiments, the real accuracy was 90% and 86% (on average). We have distinguished between the real and the practical accuracy, since in some situations, although the BCI system failed, the selection was reused by the subject to achieve the task. These useful errors were almost 50% of the total errors, thus the practical accuracy (92% and 89%) turns to be greater than the real one. The BCI system set two incorrect missions to the robotic system in all the executions (representing in total a 0.78%), which is twice the theoretical probability of this situation (0.3%).

The other aspect of the brain-computer system is the design of the visual display. We adapted the metrics used in [19], [18] to assess the user interfaces of autonomous wheelchairs: the number of selections per mission (usability rate), the number of errors by misunderstanding of the interface, and the number of far goals and turns belonging to the navigation mode (command utility).

METRICS TO EVALUATE THE VISUAL DISPLAY
Task 1 Task 2
min max mean std min max mean std
Usability rate
 2.11 3.08 2.54 0.34 2.36 3.40 2.80 0.39
Misunderstandings
 0 0 0 0 0 1 0.10 0.32
# far goals
 0 2 1.40 0.84 2 6 4.70 1.16
# of turns
 2 6 3.70 1.57 0 3 0.80 1.03

All the subjects were able to perform both tasks switching between navigation and camera mode by using the functionalities of the visual display. The usability rate was slightly grater than 2 (ideally, it should be 2 because two selections are enough to set a mission) due to the errors in the pattern recognition strategy. There was only one misunderstanding of the interface reported in the experiments that arose at the very end of one experiment (the user set an unreachable goal behind the goal wall and validated it). The command utility was greater than zero, indicating that the subjects used all the functionalities (there were no redundant commands). The frequency of usage of some commands (far goals and turns) suggests that the users perform the driving tasks in a different way, as reported in similar studies [19].

Regarding the BCI in general, an important aspect is the information transfer rate (i.e. number of bits per minute transferred from the user to the machine). This rate varies from subjects since it depends on the number of sequences of the stimulation process. As mention before, this parameter was customized for each subject to achieve a minimum accuracy. On average, the transfer rate was 10.5 bits/min.

In summary, all these results indicate that the braincomputer system is suitable for the brain-teleoperation of a robot to develop navigation and visual exploration tasks.

Robotic System

To evaluate the navigation system of the robot, we proposed metrics such as the number of collisions, the path length per mission, the time taken per mission and the minimum and mean distance to the obstacles (obstacle clearance) [19]. Moreover, we propose the following metrics to evaluate the camera exploration system: the total angle explored and the total time during which the camera was in movement.

METRICS TO EVALUATE THE NAVIGATION AND EXPLORATION MODES
Task 1 Task 2
min max mean std min max mean std
Path length per mission
 1.06 1.61 1.34 0.18 1.90 2.81 2.41 0.29
Time taken per mission
 58.75 95.50 72.99 11.42 70.25 98.63 81.74 8.30
Clearance min
 0.32 0.52 0.45 0.05 0.41 0.62 0.52 0.07
Clearance mean
 0.89 1.12 1.03 0.07 1.09 1.19 1.14 0.03
Angle explored (rad)
 1.21 6.37 2.79 1.56 0.16 0.88 0.37 0.25
Time camera in movement
 24 46 30.70 8.76 12 31 19.20 6.92

In total, the navigation system successfully carried out 177 missions without collisions, traveling a total of 325 meters with a mean velocity of 0.09 m/sec (8 times less than the usual human walking velocity). The path length per mission and the time taken per mission were greater in task 2 than in task 1, showing that the navigation system adapted to the complexity of the tasks. According to the results of the minimum clearance (0.45 and 0.52 meters on average) and the mean clearance (1.03 and 1.14 meters on average), the robot avoids obstacles within safety margins, which is one of the typical difficulties of autonomous navigation [14].

On the other hand, the camera system performance was very high since all the exploration tasks were solved (all the subjects identified the correct trajectory in the circuits). The camera control system successfully carried out 79 exploration missions, exploring a total of 32 radians in angular length. The angle explored and the total time during which the camera was in movement was greater in task 1 than in task 2, which indicates that the exploration mode adapted to the environment characteristics.

In general, the navigation and exploration subsystems successfully solved all the missions in the two tasks in environments with different conditions reporting no failures.