PROJECT
After extensive discussions, my team and I identified three evaluation goals to assess the S.U.C.C.E.S.S. website's effectiveness in meeting the needs of new immigrants to Canada.
Evaluate the web experience and navigation for immigrants.
Find out how the current S.U.C.C.E.S.S website & services attend to their target audience.
Reveal gaps in S.U.C.C.E.S.S. website and services.
The overarching evaluation methods we followed were conducting a usability study using a prompted think-aloud method and conceptual model abstraction, both supported by constructive interaction. We chose these methods to drive our study because they work well together to capture a holistic perspective of our participants’ ideas of how they envisioned the S.U.C.C.E.S.S. website to function.
I observed and analyzed how participants anticipated achieving their task goals, taking note of their comments and actions during the search process. Through constructive interaction, I guided and assisted them in their search. Every task posed a unique challenge, and each challenge required a different approach. By breaking down each approach into a series of expected outcomes, I was able to identify patterns in the participant's behavior and determine which paths led to quicker task completion.
Our study focused on error-prevention as the primary usability construct, as well as learnability and satisfaction. To identify which design, functionality, and usability domains were responsible for errors, we formulated questions to guide our data collection. These questions included what caused repeated mistakes, what features caused delays in task completion, and which design elements were responsible for preventable errors. Through a combination of observational insights during tasks and post-questionnaire responses, we were able to gather data that helped us identify areas of the website that were more prone to errors.
I was interested in monitoring participants' ability to learn a website upon first use. The fewer errors, the more momentum, and vice versa. Even a single mistake can be enough to throw a participant off-course, causing them to lose momentum and get lost in the process. It is important to note that learnability as a construct was also deeply tied to error prevention, efficiency and satisfaction when correcting mistakes.
After extensive discussions, my team and I identified three evaluation goals to assess the S.U.C.C.E.S.S. website's effectiveness in meeting the needs of new immigrants to Canada.
- Questions to get to know the participant demographics and how regularly they use technology
- Identifying goals and prior use and familiarity with S.U.C.C.E.S.S.’ website
-Tasks prompted participants with mock scenarios to follow
- From looking for housing to seeking language learning workshops
- The goal was to observe how each person engaged with the website
-Measure participant satisfaction with web experience and tasks
-Understanding the navigation patterns and intentions of each participant
-Overall outcomes and impressions of interesting or frustrating features
We conducted the usability study sessions both online through Zoom and also in-person in a S.U.C.C.E.S.S. workshop room. Then, if we were conducting the session on Zoom, we would begin the video recording, followed by conducting a pre-test interview while briefly getting to know them.
After we made the participants comfortable, we would begin going through each of the tasks created with the aim to assess the usability and overall experience of S.U.C.C.E.S.S' website. The prepared tasks were all presented through briefly described scenarios for participants to reference. By following the browsing goals of our participants, we hoped to identify real-world scenarios that immigrants using the S.U.C.C.E.S.S. website would encounter and get insight into the potential actions they would take in that process.
After collecting post-test questionnaire data and participant comments, we analyzed and grouped them into two sections. 'Observations & Insights' categorized recurring behavioral patterns as positive, negative, or neutral, which formed the foundation of our results and recommendations. For instance, we noted participants' tendency to use the 'Find a Service' button to look for events or workshops. I added such recurring navigation patterns or insights to our list when they occurred repeatedly or provided valuable insight.
Throughout our study with the seven participants, we found out a common issue that continues this when done.
-
-
-
-
-
-
If you like what you see and want to work together, get in touch!
darwinlurker6@gmail.com