Introduction
Guided by our value of Health Equity, LY Foundation pays close attention to the issue of grassroots healthcare in China. There are approximately 1.3 million rural doctors in China, who have long served a rural population of 500 million. In recent years, the accessibility of primary medical and health resources has been significantly improved under the guidance of policies, but the quality of primary medical services still has considerable room for improvement.
In recent years, smartphones have been widely used in primary care settings, and rural doctors generally have experience in retrieving information through mobile phones and the Internet. Regarding the training and empowerment of rural doctors, the medical field and the social sector have carried out a series of internet-based experiments, including online training, remote consultation and artificial intelligence-assisted diagnosis (CDSS) . Since 2022, we are pleased to see that the rapid development of the field of generative artificial intelligence provides innovative opportunities. The advantages of Generative AI are simple user interaction, powerful functions, round-the-clock response, and low learning curve. This may become a game changer for grassroots diagnosis and treatment and for traditional village doctor training.
At the beginning of 2023, LY Foundation and the Center for Experimental Economics in Education of Shaanxi Normal University (hereinafter referred to as "CEEE"), which has long been concerned about rural grassroots diagnosis and treatment issues, jointly launched the "Using Generative AI to Empower Rural Doctors" project, aiming to verify the application of generative AI at the grassroots level. This article summarizes the process and outcome of the project pilot in 2023 and our reflections. We are excited to share with charity peers. Your comments and feedback are welcome.
In order to improve the quality of grassroots diagnosis and treatment services and explore innovative means to support and empower rural doctors, LY Foundation developed an AI-assisted consultation product "Village Doctor Assistant" and cooperated with the CEEE team to launch a small-scale pilot in Shaanxi Province in 2023.
The experiment found that after the pilot, rural doctors exhibited higher ratings of satisfaction and ease of use of AI products. The doctors who we surveyed believed that the products could solve 70% of their daily work needs and could also improve their clinical knowledge and skills. however, standardized patient tests also showed that the short-term intervention did not significantly improve the quality of diagnosis and treatment.
In the future, the LY Foundation will continue to focus on grassroots diagnosis and treatment in rural China as one of its high-priority topics. Through funding other external projects, we will accumulate insights and refine our own theory of change for how to improve rural healthcare quality in the next 10 years.
What did we do?
In the first half of 2023, the project team investigated the current status of diagnosis and treatment problems for rural doctors. We went to Shaanxi province to conduct interviews, learned about rural doctors’ needs for diagnosis and treatment auxiliary tools, and made initial design decisions about the use case of generative AI in the rural scenario. In the second half of 2023, the project team conducted product research and development of auxiliary diagnosis and treatment tools for village doctors and completed a small-scale pilot evaluation in a county in Shaanxi.
March-June 2023: Analyze the current situation of primary medical services through field surveys and desk research.
- The project team visited 2 township health centers and 6 village clinics, and interviewed the directors of the health centers and village doctors to learn about local patient profiles, common diseases, medical conditions, village doctors’ training status, and the scientific and technological literacy of village doctors.
July-September 2023: Plan the intervention model, complete the product development of "Village Doctor Assistant", and design AI prompts.
- Based on the needs of village doctors, the project team designed and developed a "Village Doctor Assistant" tool based on generative AI technology, which was deployed on mobile phones and computers in the form of web links. After multiple rounds of experiments and tests with real doctors, the project team customized prompts for village doctor assistants. By engaging in multiple rounds of dialogue with the Village Doctor Assistant on their phones, doctors were able to obtain suggestions on diagnosis and treatment, which integrated both Chinese and Western medicine knowledge.
- In order to improve the user experience for users of the Shaanxi dialect, the project team cooperated with the iFlytek Sansheng AI charity project team of the University of Science and Technology to add Shaanxi dialect voice dictation capabilities. In this way, users were able to "talk" directly to the Village Doctor Assistant in their own dialect.
- The project team also developed a set of user guides specifically tailored to the needs of village doctors, as shown below.
September 2023: Go to the project site for product deployment and training.
- The project team selected fifteen doctors from two towns. Five village doctors from Village A and Five township doctors from Village A served as the intervention group of the project to receive the Village Doctor Assistant. Five village doctors from Village B served as the control group of the project.
- The project team conducted training sessions on the Village Doctor Assistant in the township health center, where they installed it on doctors’ mobile phones and conducted case demonstrations and exercises. Additionally, the project team conducted on-site visits in five village clinics.
August-September 2023: Post-intervention research and evaluation
- In order to analyze the intervention effect, the project team conducted pre- and post-tests on the diagnosis and treatment of village doctors, using interview questionnaires, clinical scenario knowledge tests and standardized patient methods to gain an in-depth understanding of the condition of village clinics, doctors’ work needs, internet literacy, medical service quality, and other information. The effectiveness of Village Doctor Assistant was tested.
In 2023, LY Foundation's total spending on the project was 172,675 CNY. During the development process, the foundation team used open source large language model technology and free public resources to reduce development costs. The specific details are as follows:
What did we learn?
During the 13-day pilot, rural doctors treated many patients with the support of Village Doctor Assistant. On average, each consultation involved 2-3 rounds of dialogue with the tool. The tool was used a total of 141 times. Click this link to get the Project Report on this experiment. The main points are as follows.
Regarding the user experience of the Village Doctor Assistant, surveys reflected high satisfaction. Users indicated that they believed the Village Doctor Assistant could solve 70% of their daily work needs. They felt that their clinical knowledge and skills improved, as well as their confidence in diagnoses. In terms of internet technology literacy, doctors’ attitude toward the use of AI medical tools, trust in information, and trust in functions all improved. 83% of patients indicated support for the doctors’ use of Village Doctor Assistant.
At the same time, doctors reported that Village Doctor Assistant sometimes gave irrelevant or lengthy answers. Most village doctors were not proficient in using Pinyin or the dialect-to-text functions, which caused difficulties with the user interaction.
In terms of intervention effectiveness, standardized patient testing results found that the 13-day short-term intervention did not significantly improve the service quality of the doctors.
This phenomenon not only reflects the limitations of the Village Doctor Assistant itself, but also points to problems in the application of AI products in rural primary care scenarios overall.
- The auxiliary diagnosis and treatment capabilities of generative AI rely on the consultation habits of the assisted doctors, which happens to be the shortcomings of grassroots doctors. Oversimplified questions would cause the AI to be unable to provide accurate and comprehensive information, affecting the overall intervention effect.
- Due to the lack of equipment and medicines in the rural clinics, the examination suggestions put forward by the Village Doctor Assistant were often inapplicable.
- In addition to daily consultations, village doctors were also responsible for providing public health services. However, Village Doctor Assistant only covered their needs with diagnosis and treatment.
- Most village doctors had frequent questions about Traditional Chinese Medicine (TCM), but Village Doctor Assistant was unable to provide high-quality TCM diagnosis and treatment suggestions.
Overall, the pilot confirmed the user affinity of AI in grassroots diagnosis and treatment scenarios. However, whether the AI-empowered intervention model is the optimal solution in the current scenario requires further product iterations and long-term, large-sample experiments and evaluations. The cost of the current intervention model is low, but considering that the payment capacity of the rural primary medical system is extremely limited, the feasibility of long-term operation of the project at the grassroots level needs further verification.
Future Directions
At this stage, we believe that the Village Doctor Assistant does not have the prospect of large-scale development, unless there exist practical and effective methods to respond to the four problems listed above. The team needs to accumulate more insights into the real needs of rural doctors. Domestic substitution of Generative AI products also needs to be put on the agenda to improve product capability and user experience while ensuring the compliance.
LY Foundation will continue supporting initiatives that improve primary care diagnosis and treatment in rural China as one of its key focus areas. In the next 10 years, through funding external projects, we will accumulate insights, build new partnerships, and work towards more powerful technology-enabled primary care diagnosis and treatment interventions.
If you have a project in the field of primary care that is in line with LY Foundation's funding strategy, please feel free to contact us. Thank you.
Standardized patients, also known as simulated patients, refer to normal people or patients who can accurately represent the actual clinical problems of patients after standardized and systematic training. The standardized patient method is a method of measuring the quality of medical services. By asking standardized patients to simulate disease symptoms and collect information without the knowledge of the treating doctor, it can objectively evaluate the clinical skills and communication skills of the treating doctor.
The standardized patient method has long been considered the "gold standard" for evaluating clinical practice in developed countries, and it has also begun to be popularized in developing countries in recent years. Compared with other commonly used methods, this method has obvious advantages in evaluating the quality of medical services. First, because clinicians are not aware that they are being evaluated, they will not act differently (the "Hawthorne effect"); second, this method can measure actual clinical medical behavior, instead of the doctor's clinical knowledge; third, the standardized patient approach (especially if the visit is recorded) is less affected by recall bias caused by interviewing the patient after leaving the clinic; finally, because the cases are common and standardized, the standardized patient method can be used to compare the quality of medical services provided by doctors at different levels and in different locations.
