Do students really possess the competencies? Assessing generic skills through scenarios

Abstract

Generic skills become increasingly important for university graduates. Assessing the skills however is still a challenge since most skills contain “know-hows” that need to be observed in certain contexts. The assessment should focus more on students’ abilities to analyse, interpret and apply the skills than the memorisation of facts. A limitation of the commonly used questionnaire is that it may not reflect higher levels of the skills development. Consequently, generic skills may not be best assessed by “generic” items. An appropriately designed assessment that is self-directed and reflective is therefore desirable. This study proposes a scenario-based assessment method. Using scenarios as the relevant contexts for students to apply their skills, the tool can identify students’ self-awareness and “know-hows”. The information can then be used to engage students in critical reflection on their skills development. A pilot study was conducted with 99 undergraduate students from multiple disciplines in a research intensive university in Hong Kong. The method was deployed to assess teamwork skills during the member recruiting process of the university’s Robotics Team. In the pilot study, scenarios were drafted by a teamwork researcher and reviewed by three staff members heavily involved in student development. Each scenario contained some undesirable teamwork behaviours. Students were required to judge the effectiveness of the behaviours and write down their preferred actions. Apart from the scenarios, an open-ended question was used to solicit students’ teamwork conception. The analysis of students’ responses showed interesting findings. Approximately 55% of the students regarded ignoring non-productive members and dividing the work with minimum communication as effective teamwork behaviours. Furthermore, half of the students who indicated that communication was important through their responses to the open-ended question were not able to recognise the lack of communication in most scenarios. In addition, business students performed better in identifying the lack of communication than science and engineering students. The study demonstrated the value of using scenarios in assessing and developing generic skills, especially the potential in revealing the self-awareness and misconceptions that can have an influence on the skills development.