The USSD mobile phone application
The final print version of the adolescent reproductive health content (Supplemental File 1: Appendix A) was programmed into the web interface provided by Echomobile® for the USSD platform.
To open the app, users dialed a 7-digit code on a mobile phone, which then prompted them to enter a pre-assigned PIN linked to their mobile phone number. When users were authenticated, they selected their gender and age on the following screens before selecting a sexual and reproductive health topic of interest. Users then interacted with the content on a screen-by-screen basis guided by their input. A selection of interactive screens in the mobile application is shown in Fig. 2.
Teenage users could select sexual and reproductive health topics of interest and access information. Figure 3 shows how a user accessed content about sex.
For the alpha test, participants had to dial a 7-digit USSD code on a feature phone or smartphone. Users were then guided through authentication and gender and age selection. Table 1 shows the demographics of teens who were successfully tracked and used the app at least once during the one-month alpha usability testing period. The median age of participants was 15 for those under 18 and 18 for those 18 and over. All have attended secondary school. Of the 38 teens enrolled, 12 were successfully tracked, only 9 had used the app at least once during the one-month period. Only survey questions applicable to the USSD application were analyzed. Questions about app customization, sharing, app description on playstore, buttons and icons were excluded.
Participant feedback on the engagement, functionality, aesthetics, and quality of information provided by the mobile app is shown in Table 2. The MARS was rated from 1 to 5, with 1 being the lowest score and 5 the highest. The highest scores were on app engagement with an average score of 4.4 and on functionality with an average score of 4.3. The MARS consisted of 15 items and the Cronbach’s alpha value for the survey was α = 0.83.
Under aesthetic aspect, information about layout, “layout and size of buttons/icons” and graphics “quality/resolution of graphics used for buttons/icons/menus/content” have been excluded from the final analysis. The research team noted that these features were not applicable to the USSD app, the app does not contain buttons, icons or graphics. Since no issues were identified during alpha testing of the app, nothing was changed prior to field testing.
Field usability test
After the alpha usability test was completed and the application performance validated, field usability testing was performed. Participants had to dial a 7-digit USSD code on a feature phone or smartphone, before going through authentication and gender and age selection. Once in the app, users had a list of options to choose from based on their information needs. For field testing, 146 adolescents were enrolled, 113 were followed up, and 109 provided complete study data. Of these, 62 used the app at least once during the 3 month period. Demographics of participants who used the app during field usability testing are shown in Table 3. These p-values verified the potential statistically significant difference in demographics between participants under 18 years of age. and those over the age of 18. The Mobile Application Rating Scale scores for field usability testing are shown in Table 4. Only survey questions applicable to a USSD application were analyzed. Questions about app customization, sharing, app description on playstore, buttons and icons were excluded.
In field usability testing, app engagement achieved an average score of 4.3 (0.44). The average functionality score was 4.6 (0.38), with navigation and gesture design ratings in the functionality score reaching an average score of 4.8 (0.43) and 4.8 (0.35 ) respectively. The overall mean score for information was 4.4 (0.31), with quantity of information reaching 4.6 (1.11) and quality of information 4.5 (0.71). The Cronbach’s alpha value for the field trials was α = 0.54. It is important to note that 109 adolescents were successfully followed after the 3 month period. However, only 62 had used the USSD application at least once. The usability test interview was only administered to teenage participants who had used the app at least once during the 3-month period. Table 5 shows participants’ comments on the app’s features. Only options selected by users are included in this table.
Teens found the app entertaining, with 43.6% (27) of users indicating that the app was fun to use. Most users, 70.9% (44), found the app very interesting. Teens felt that app content was directed appropriately, with 54.8% (34) of users indicating it was well targeted. Only 3.2% (2) of users felt that the content was not well targeted or inappropriate.
When it comes to the accuracy of app functionality, 72.5% (45) of teens felt the functionality was perfect and had no bugs. Teenagers were able to learn to use the app quickly, with 72.6% (45) finding it easy. When it comes to app interaction, 85.5% (53) of teens found the content consistent and intuitive across all screens.
Over 90% of teens found the content provided in the app relevant to their sexual and reproductive health needs. Gender differences were not significant. The content was complete according to 87.1% (54) of the teenagers. Notably, at least one in two participants, 54.8% (34), indicated that they would recommend the app to other teens. The app received a high rating, with 72.6% (45) of users describing it as the “best app” for providing information on adolescent sexual and reproductive health.
Although 45.2% (28) of users “strongly agree” that the app can increase awareness of sexual and reproductive health information, there are significant gender differences within this category. While 66.7% (20) of male participants “agree” that the app can increase awareness, less than half of female users, 43.7% (14), “agree”. The majority of participants, 56.3% (18), “strongly agree” that the app can increase awareness. In terms of improving knowledge, 59.7% (37) of users ‘agreed’ that the app was likely to increase knowledge about sexual and reproductive health.
Users also felt that the app could change attitudes towards adolescent sexual and reproductive health, with 69.4% (43) agreeing that a change in attitude would improve service delivery and use . The app was considered to potentially increase uptake of interventions providing adolescent sexual and reproductive health by 81.0% (44) of participants. On whether the app would encourage help-seeking behavior in adolescents about reproductive health issues, 66.1% (41) of participants agreed. A reduction in adolescent sexual and reproductive health problems through app use was predicted by 66.1% (41) of participants.
A number of user experience evaluation components scored significantly differently from male and female users. Of the app’s overall rating, 72.6% (45) ranked the app as the “best app”: of these, 84.4% (27) were female users versus 60.0% (18 ) of men, with a p-value = 0.011. On the USSD app, which raises awareness about adolescent sexual and reproductive health, 56.3% (18) of female users “strongly agree” compared to 33.3% (10) of male users. Regarding the app’s likelihood of improving knowledge about sexual and reproductive health, 46.9% (15) of female users ‘strongly agree’, but only 30.0% (9) of male users.