Before finalization of the tool, user testing are being carried out to optimize the interface and visualization tools.
In parallel with finalizing the tool, the prototype is tested on school students. The following evaluation template is used:
Information about Anna
Anna is an online tool, developed by a consortium of Europe’s leading universities, as part of a project called INGDIVS (INcreasing Gender DIVersity in STEM): http://www.ingdivsproject.eu
Specifically, Anna tries to provide information from current engineering students and working engineers in an easily accessible manner to potential students – allowing them to better relate to the skills, attributes, attitudes and experiences of people who have chosen that route. It is our hope, based on previous research, that by making it easier to relate to people who have previously made that choice (to study engineering), that potential engineers are able to see themselves in these roles and make a more informed choice of course study.
How you can help
We have built a first prototype of the tool, and we are looking for feedback on how easy it is to use, and how useful it might be. We hope that you will be able to supervise some user testing. We will not ask you for, nor will we record, any identifying information from your students. We hope the information below is clear and informative, and your local university partner should be able to assist you with any queries.
You need two people for this process. One person takes the lead in communicating with the students, while the other has the role of rapporteur – taking notes of students behaviours, reactions and responses. The lead person may wish to refer to these or other notes, but doesn’t require any other resources. The rapporteur may choose to use a paper version of the enclosed form, or to complete it electronically. The students (3-5) will each need access to a computer, with the tool loaded and ready to be used. The total time for an individual group will be approximately 30-35 minutes, and we hope that you can test 3 groups. These groups, if applicable, should comprise an all male group, an all female group and a mixed gender group.
How the Test Process Works
There are three principal phases. Firstly you give a short introduction to the group of students (we suggest 3-5 students) – we’ve prepared a short script which you can use if you like (about 5 minutes). Secondly the students take the test, individually, at the computer. This will take about 10 minutes. Finally, you bring the students together (the focus group) and prompt them to discuss their experience. There are 5 topics we would like the students to discuss. We have provided some sample prompts for each topic, and then a ‘wrap-up’ question to summarise in a quantitative way the collective opinion of the group. The 5 topic areas are:
-Understanding of the tool’s purpose
– Ease of use
– Any other comments or feedback
What sort of students should we seek?
Any students from the pool of students who could potentially progress to engineering programmes at university. They do not need to be only those who have strongly decided already – in fact it is probably better if they aren’t. Equally, it is probably better if they have not strongly rejected Engineering. The ideal students are ones who either have decided, or who have another preferred option but are not negative towards engineering studies.
Sample prompts/scripts for introduction and focus group
|Content||Sub-area||Sample script you can use||Explanation/Key information|
|Introduction||Hi everyone, we are hoping you can help us with an interesting project. You are probably aware that relatively few women in most countries choose to study engineering or technology. It turns out that in many cases female students find it difficult to foresee themselves in engineering roles – while they have the raw ability (of course!) they may wonder if they have the skills, knowledge or interest in this area. Often they have no easy access to role models or people they can talk to find out more. In this project we have created a simple software/website tool to allow potential students to browse (anonymised) profiles of real engineering students and working engineers – to better understand what it is like to be an engineer. We’d like your feedback on this tool so we can make it as useful and effective as possible. No data will be recorded on your use. Testing will take about 10 minutes – you’ll each use the tool on your own computer, and then we’ll come together for about 15 minutes to discuss your thoughts. X (Rapporteur) will take some notes to give some anonymous feedback to the developers.||
|Focus Group||Intro||Hi everyone, now that you’ve had a chance to use the tool, we’d like to discuss your thoughts and experiences. I’ll give you some prompts to get the discussion going and then X will ask some questions also.||
|Focus Group||Guidelines||Remember there are no right or wrong answers during this discussion, we are trying to obtain your honest opinion. We encourage a group discussion but please respect your classmates and allow for one person to speak at a time.Ask politely to turn phones off etc.||
|Focus Group||Purpose||What do you think the developers main goals were with this tool?||
|Usefulness||Would it be useful for you to know more about individuals who have chosen the career path you are interested in?||
|Ease of use||What words come to mind from your experience of using the tool?
Were you satisfied with the tool doing what you wanted/expected?
Can you describe any frustrations you encountered?
|Improvements||What area do you think the developers should concentrate on improving for the next version of the tool?||
|Comments||OK, thank you for your help. Are there any other observations, comments or suggestions you’d like to pass on to the project team?|