Loading

Leveraging A/B Tests to improve tools & access to information Lessons from Kenya

Background

PAD’s “MoA-INFO” Platform was designed as an interactive, two-way SMS service to disseminate advisory information and collect data about Fall Armyworm (FAW) infestation, and was developed in partnership with the Government of Kenya's Ministry of Agriculture and CABI. The service has grown in scope and reach since it commenced operations with 20,000 farmers in the second quarter of 2018. In 2019, MoA-INFO had cumulatively serviced almost 416,000 farmers with advisory on six crops and a range of associated pests and diseases.

The Fall Armyworm Monitoring Tool

As of June 2019, the Fall Armyworm monitoring tool had been initiated by approximately 22% of MoA-INFO platform users and completed by just less than 9% of platform users. 2019 included the third and fourth season in which the FAW tool was being offered to farmers. At the conclusion of the third season, PAD staff were noticing a drop in farmer engagement and were concerned about the potential for messaging fatigue. Motivated to improve the service, and to more effectively understand user experience and preferences, the team deployed tests to learn from our farmers.

The monitoring tool is activated by a farmer by texting the word “CHECK” or “ANGALIA” (Swahili for “check”) to the Kenyan phone number 40130. Once activated, the tool asks farmers to sample 10 plants at five different spots on their farm (50 plants total) to measure the infestation rate of FAW in their fields. Once the farmer has submitted their observational data about the presence of FAW, the tool generates a context-specific recommendation based on the infestation rate and the size of their maize stand. Farmers are advised to use non-chemical controls if the infestation is below the threshold where pesticides will be cost effective, or to use pesticides if the infestation is more severe.

The Fall Armyworm - scourge of smallholder maize farmers

MoA-INFO users access content on the service via a "push" or a "pull". A push occurs when we use information that we know about the user to determine appropriate advisory content that we then push to the user. For example, if the rains have started in a particular area, we will send planting information to farmers who we know live in that area. A pull occurs when a user seeks out information from the service for themselves. For example, a user can text the word “MENU” to the service and browse information about different topics. In this context, a farmer pulls the FAW tool when they send an SMS with the keyword word “CHECK” and the service then send them the monitoring tool.

Pushing the tool directly can be better for engagement at a particular time, but sending the keyword (“CHECK” or “ANGALIA”) may encourage users to pull the tool later at a time more convenient to them or relevant to their needs. When designing a service like MoA-INFO there is always a trade-off between pushing and pulling information, and the need to weigh the risk information overload.

Deploying the A/B Test

In order to better understand the relative benefits of communicating the tool to farmers, and to maintain or improve uptake rates, PAD staff designed an A/B test.

To execute the test, we randomized both the phrasing of invitation messages, and the time of day that messages were sent to see how this might affect user uptake.

Survey design was informed by the following research questions:

  1. What is the best message to send to users to maximize their likelihood of completing the FAW monitoring tool?
  2. What is the best time of day to send these invitation messages?

Design:

In this experiment we randomized two types of messages: a regular message, which advertises the tool keyword (“CHECK” or “ANGALIA”), and a message which instead takes the users directly into the monitoring tool, bypassing the first question. Each user surveyed as part of the A/B test had a 50% chance of receiving either question.

Regular message:

Hi. Now is a great time to scout your maize for Fall Armyworm! Send CHECK to 40130 for help scouting your maize.
Hujambo. Sasa ni wakati mwafaka wa kuchunguza mahindi yako ikiwa yana Fall Armyworm! Tuma neno ANGALIA to 40130 kwa usaidizi wa kukagua mahindi yako.

Make it easy message:

Are you ready to assess the extent of Fall Armyworm damage on your farm and receive advice? You will need to go to your shamba [field or plot]. A. Yes, B. No
Je, uko tayari kuchunguza kiwango cha uharibifu wa Fall Armyworm shambani mwako na upokee ushauri? Utahitajika uende shambani. A. Ndio, B. La

Second, we randomized the time of day that the messages were sent in such a way that each user had a 25% chance of being sent their message at 7AM, midday, 3PM and 6PM.

The total sample for the experiment was 40,000 MoA-INFO users, selected randomly among those that had opted in to receive maize messages during the long rains (March to May) season of 2019.

Results and Analysis

Of the two different message phrasings, the “make it easy” formulation stimulated 14% more recipients to initiate the tool. Moreover, of recipient farmers who initiated the tool in response to either message, those responding to the “make it easy” message were 12% more likely to complete the test.

Messages sent at 6pm returned the highest number of users who accessed the monitoring tool, but not by a great deal - a one to two percentage point improvement over other times. The midday message, while eliciting the lowest commencement rate, resulted in the highest completion rate across the four times.

Intuitively, midday is a time when it is most likely that farmers will be in their fields; a requirement for undertaking and completing the assessment tool. The midday message resulted in significantly better information generation than the 7AM message. Comparisons in the completion rate between the 3PM and 6PM rates were not statistically significant, but both were below the completion rates associated with the midday call.

Learning and Iterating Design

Based on the results of the A/B test, the service was re-designed to send the advisory tool at midday using the “make it easy” phrasing.

In conjunction with improved farmer targeting, the monitoring tool was pushed to farmers 77,568 times in the 2019 Kenyan short rains season (October-January). A sample of 50,779 farmers received messages (some received it more than once); 16,234 invitations were accepted for the survey (21% of invitations; 32% as a share of farmers) and 5,213 surveys were completed (10.3% of farmers, but 32% of those who initiated the tool).

Of the 5,213 farmers who completed the test, 1,934 were encouraged not to use pesticides because it would not be cost effective for them while the balance were encouraged to use pesticides to address the infestation. The phrasing of the “make-it-easy” message has also been adapted for use in other messages sent to farmers inviting them to receive content.