Pilot testing

Ongoing work:

Before finalization of the tool, user testing are being carried out to optimize the interface and visualization tools.

In parallel with finalizing the tool, the prototype is tested on school students. The following evaluation template is used:

Information about Anna

Anna is an online tool, developed by a consortium of Europe’s leading universities, as part of a project called INGDIVS (INcreasing Gender DIVersity in STEM): http://www.ingdivsproject.eu

Specifically, Anna tries to provide information from current engineering students and working engineers in an easily accessible manner to potential students – allowing them to better relate to the skills, attributes, attitudes and experiences of people who have chosen that route. It is our hope, based on previous research, that by making it easier to relate to people who have previously made that choice (to study engineering), that potential engineers are able to see themselves in these roles and make a more informed choice of course study.

How you can help

We have built a first prototype of the tool, and we are looking for feedback on how easy it is to use, and how useful it might be. We hope that you will be able to supervise some user testing. We will not ask you for, nor will we record, any identifying information from your students. We hope the information below is clear and informative, and your local university partner should be able to assist you with any queries.

Resources Required

You need two people for this process. One person takes the lead in communicating with the students, while the other has the role of rapporteur – taking notes of students behaviours, reactions and responses. The lead person may wish to refer to these or other notes, but doesn’t require any other resources. The rapporteur may choose to use a paper version of the enclosed form, or to complete it electronically. The students (3-5) will each need access to a computer, with the tool loaded and ready to be used. The total time for an individual group will be approximately 30-35 minutes, and we hope that you can test 3 groups. These groups, if applicable, should comprise an all male group, an all female group and a mixed gender group.

How the Test Process Works

There are three principal phases. Firstly you give a short introduction to the group of students (we suggest 3-5 students) – we’ve prepared a short script which you can use if you like (about 5 minutes). Secondly the students take the test, individually, at the computer. This will take about 10 minutes. Finally, you bring the students together (the focus group) and prompt them to discuss their experience. There are 5 topics we would like the students to discuss. We have provided some sample prompts for each topic, and then a ‘wrap-up’ question to summarise in a quantitative way the collective opinion of the group. The 5 topic areas are:

-Understanding of the tool’s purpose

– Usefulness

– Ease of use

– Suggestions

– Any other comments or feedback

What sort of students should we seek?

Any students from the pool of students who could potentially progress to engineering programmes at university. They do not need to be only those who have strongly decided already – in fact it is probably better if they aren’t. Equally, it is probably better if they have not strongly rejected Engineering. The ideal students are ones who either have decided, or who have another preferred option but are not negative towards engineering studies.

Sample prompts/scripts for introduction and focus group

Content	Sub-area	Sample script you can use	Explanation/Key information
Introduction		Hi everyone, we are hoping you can help us with an interesting project. You are probably aware that relatively few women in most countries choose to study engineering or technology. It turns out that in many cases female students find it difficult to foresee themselves in engineering roles – while they have the raw ability (of course!) they may wonder if they have the skills, knowledge or interest in this area. Often they have no easy access to role models or people they can talk to find out more. In this project we have created a simple software/website tool to allow potential students to browse (anonymised) profiles of real engineering students and working engineers – to better understand what it is like to be an engineer. We’d like your feedback on this tool so we can make it as useful and effective as possible. No data will be recorded on your use. Testing will take about 10 minutes – you’ll each use the tool on your own computer, and then we’ll come together for about 15 minutes to discuss your thoughts. X (Rapporteur) will take some notes to give some anonymous feedback to the developers.	Female students can have concerns over how they will be perceived in taking a ‘male’ role Female students are sometimes concerned that there will be lots of implicit knowledge/skills that male students will possess Both male and female students may lack good information on what engineers do This tool is a simple and easy tool to allow you to search profiles of current students and practicing engineers We are looking for feedback on its design Participation is anonymous – no individual information is recorded
Focus Group	Intro	Hi everyone, now that you’ve had a chance to use the tool, we’d like to discuss your thoughts and experiences. I’ll give you some prompts to get the discussion going and then X will ask some questions also.	The key merit in a focus group is that helps individuals explore and contextualise their experience in an open-ended way. The potential benefit is that you get more ‘honest’ reflection on what you’re testing, rather than people telling you what they think you want to hear. The challenge is encourage all to participate, rather than one strong voice to dominate A good prompt is one which encourages people to reflect on their experiences and responses. The sharing that happens can elicit deeper, more subtle perceptions. We will try to capture these responses through keywords or phrases that seem to resonate strongly We conclude each phase with a more quantitative question, asked by X that summarises the groups feelings
Focus Group	Guidelines	Remember there are no right or wrong answers during this discussion, we are trying to obtain your honest opinion. We encourage a group discussion but please respect your classmates and allow for one person to speak at a time.Ask politely to turn phones off etc.	Ensure the participants are comfortable and know that while they are having a discussion, there are still some rules they should follow in order to get the most out of the session
Focus Group	Purpose	What do you think the developers main goals were with this tool?	Asks for opinion rather than a ‘correct answer’
	Usefulness	Would it be useful for you to know more about individuals who have chosen the career path you are interested in?	‘Permit’ students to have another career choice and to visualise whether such a tool for that career choice might be interesting
	Ease of use	What words come to mind from your experience of using the tool? Were you satisfied with the tool doing what you wanted/expected? Can you describe any frustrations you encountered?	Let the students express more general reactions first – sharing words, and then prompt further for elaboration. E.g. if the student says ‘fun’ ask them to ‘tell me more about that fun experience – what made it so?’ If there are more ‘negative’ words – e.g. ‘boring’ ask what they might do differently to make it easier to use or more useful Allow them more general reactions first before steering back to questions of whether they could achieve what they wanted.
	Improvements	What area do you think the developers should concentrate on improving for the next version of the tool?	Sends a message that this is not expected to be perfect and that their input is valuable
	Comments	OK, thank you for your help. Are there any other observations, comments or suggestions you’d like to pass on to the project team?