The aim of this exploratory study was to examine changes in collaboration patterns, and how this relates to usability problems during repetitive usage of desktop computers for distributed, unstructured research meetings. Here we present how 4 people collaborated during three months and 10 sessions in virtual environments.
The results showed that positive experiences in collaborating contribute to higher effectiveness, independently of the tasks. Several properties, like the structure of a meeting, how concrete the actual goals were and how well the roles in the group was defined, have been identified as factors influencing use and usability. Additionally, two processes of adaptation have been identified, one to the system (the technology) and another to the group (between the users).
K.4.3 [Computing Milieux]: Computers and Society, Computer-supported collaborative work.
Experimentation, Human Factors, Design.
Collaboration, effectiveness, usability, virtual environments, unstructured meetings.
Virtual environments are increasingly being used in networked situations for distributed collaboration. There is still very little research about social and technical aspects of collaboration and how these relates to issues influencing overall usability of the environments.
There are several benefits of distributed environments for meeting situations, for example they can save traveling time and expenses, and replace face-to-face meetings [1], with special respect to collaborative virtual environments (CVEs) [2, 3].
To study mediated collaboration Scott [4] has summarized several group collaboration models in an input, process and output framework, called the meeting process model. A benefit of this model is that it includes task characteristics in the input, grouped according to McGrath's [5] decision typology. The tasks are classified in four general groups: to generate (create a plan), choose, negotiate and execute. Scott emphasizes that technology in this model is treated only in relation to outcomes, which in turn depend on task decision performance, efficiency, member satisfaction and communication. Since task decision performance is similarly defined as the effectiveness in the definition of usability, it can be concluded that the meeting process model, with 'outcomes', also measures the usability of the meetings. However, this model does not explain how certain internal variables about relationships and structure influence each other.
For designing collaborative VEs, Tromp has identified main areas (narrative affordances, collaboration, automation, education, and layered design), and a long list of thirty-eight main collaboration problems contributing directly to usability [6]. She identified the design of social space as the most immediate area, which at this stage is "far removed from the need of the users".
Based on Scott's and Tromp's collaboration models, this exploratory study examines repeated uses whereby four participants took part in ten, minimum one-hour long, meetings in a desktop virtual environment, Active Worlds (AW, see www.activeworlds.com). Only to support the distributed meetings, when it was considered necessary and time saving, a real life meeting has been inserted between two distributed meetings. There were 2 real life sessions for 120 minutes before and after the virtual meetings, and for 30-60 minutes after the first, second, fifth and ninth virtual meetings.
Table 1. The virtual sessions,
duration time and meeting profile
(P = plan, L=learn, PS = problem solving, E=explore)
| No | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| Time | 90 | 70 | 70 | 100 | 90 | 60 | 70 | 60 | 70 | 120 |
| Profile | P |
P L |
P PS |
P PS |
PS | PS |
PS E |
PS | E |
P L |
The analysis is based on questionnaires, including individual evaluations and results reported both by subjective comments and on a 7-point scale after each session. The questions concerned navigation, manipulation, communication, collaboration, adaptation, inconsistencies, presence and copresence. After each session someone summed up the individual results, and the whole group has commented on these results afterwards (via e-mail). After the 5th, the 7th and the 10th session, the entire group evaluated all of the previous sessions.
Two different types of goals are distinguished for this study, viz. the goal of a meeting occasion, i.e. defined for each meeting and the goal of the entire experimental study, i.e. to manage to reduce the real meetings that support the virtual meetings. The data regarding the observed collaboration and usability for a single meeting as well as for the whole experimental study are included. The study design and some result regarding the length of the time spent in the environment is partly reported in a previous work [7].
The collaboration in the groups can be considered to have seen chaotic at the beginning. During the first session the participants had several problems, with usage of the technology and with each person having to focus on the three others all the time in the virtual world, looking at the text messages and checking whether they match what can be heard via the audio. The novices had to handle two new tools AW and the audio at the same time. Already the first meeting showed a need for structuring the sessions. Even though the initial plan was to try to have unstructured sessions, the need for greater coordination became increasingly obvious. After the first session in the virtual environment, a few minutes at the beginning of each meeting were spent on structuring the activities, the collaboration, and deciding who would do what together with whom. The estimated collaboration is shown in Figure 1.
Figure 1 Collaboration
The goal of a meeting was decided at the beginning of each meeting, and the participants spent the rest of the time doing their best to reach it. This makes it difficult to correctly estimate the effectiveness of each meeting, since the difficulties and uncertainties of achieving the goal during the sessions were part of the sessions. As has been shown in the questionnaires and in the summative evaluations, the sessions (1, 3, 4), where the process of organizing the meeting and finding a goal took a long time, were evaluated as being worst. An exception to this is session 8, estimated as being bad, even though it had a clear goal (to continue building) and divided labor (two and two). It was mentioned before that coordination is difficult in AW. The subsequent sessions (5, 6, 7) were estimated as having been good, as the participants became accustomed to the system and to collaboration. Session 8 began with deciding that the work done in session 7 should be continued; however the participants were disturbed in their coordination. During this session there were social disturbances by virtual and real visitors. Those sessions which had a leader early on during the session, and those where the labor was divided and concrete for each participant, were evaluated as being best (except session 8). In sessions 1, 3, and 9 no leaders were chosen, and the same holds for the first half of session 4. These sessions are estimated to have been quite bad. For session 2 one expert was chosen to be the teacher, and for session 10, a participant was the presenter. The worst estimations were obtained from the fourth session. After two meetings of a more concrete character, with a leader and divided labor, this meeting mainly concerned exploration, what to do next, and what an avatar can achieve in AW. There were three sessions, 4, 7 and 9, of a more explorative character. During session 7, the exploration was characterized by looking for concrete, usable objects. Until session 9 the exploration was for looking for novel spaces, nice features that one of the experts experienced during the time that she spent in AW. The evaluations of the efficiency of the meetings are shown in Figure 2.
Figure 2 Efficiency
By calculating the trend for the efficiency of the meetings, it was seen that the efficiency increases during the ten meetings that shows that the study was not long enough to be called long-term, as efficiency was still increasing when the study was terminated. Nevertheless the results show how people (primarily the novices) learned to use the system and get used to it in such a way that the group could become a regular user group for AW. The adaptation process observed in this study is not only to the technology, but also a social adaptation between the users.
One goal of the study, to see to what extent real-life meetings were needed to support the virtual meetings, was demonstrated: the real meetings become fewer. It was confirmed during the study that it is difficult to coordinate or to initiate new ways of coordination in the CVE. This enhances the value of real-life meetings, especially in cases where people have to discuss different coordination possibilities and choose one that is suitable.
By calculating the mean values for collaboration and for efficiency there can be observed a covariation between these values for the different meetings, see Figure 3. The efficiency varies with the character of the meetings too.
Figure 3 Efficiency and collaboration
These clearly demonstrate the need to pay attention to collaboration in usability studies.