Video Quality Assessment and Analysis of Gastroesophageal Reflux Disease on TikTok and Bilibili: Cross-Sectional Study

Introduction

Gastroesophageal reflux disease (GERD) is common in the community, and the aging population has led to an increasing incidence, which according to a meta-analysis state that the prevalence of GERD in the community is 14.8%;¹ in addition, GERD treatment burdens the healthcare system.²

Online health information-seeking (OHIS) refers to searching for detailed information about symptoms, diagnosis and treatment of various diseases.³ Online health information-seeking behavior can obtain information related to diseases, and relevant people who care about their health can take more proactive measures to prevent diseases or improve their symptoms, but at the same time, it will increase negative emotions such as anxiety.⁴ Searching for information on the Internet affects anxiety differently across individuals, with individuals with moderate to high levels of anxiety experiencing an increase in anxiety after retrieving information about the disease and individuals with low levels of anxiety about the disease experiencing a decrease in anxiety.⁵

It has been shown that online health information searching promotes lifestyle changes, improves patients’ healthcare-seeking behaviors, and increases treatment adherence.⁶ Meanwhile, a consensus suggests that clinicians should provide patients with information on GERD-related mechanisms, weight management, lifestyle, dietary habits, and relaxation strategies,⁷ which are very easy to obtain from the Internet, while different video uploaders can provide explanations from different perspectives in order to allow patients to choose their own aspects of interest or concern.

As a new type of science communication, online popular science videos have gradually become an important way for people to acquire scientific knowledge. TikTok and Bilibili were founded in September 2016 and June 2009, respectively, with TikTok becoming the most globally distributed Chinese app in a short period of time, while the Bilibili website has developed a large multicultural community around its users, artists, and content, with a high level of youth activity.^8,9

Currently, no study has analyzed the quality of the videos about GERD published on Chinese video platforms, so it is crucial to assess the information of related online popular science videos. This study aimed to evaluate the quality and reliability of the sources of GERD-related videos published on Bilibili and TikTok and provide opinions for the public to obtain relevant information from videos.

Methods

Search Strategy

Our strategy was to sign up for a brand new account as well as clear the browser cache and search history to retrieve the keyword “胃食管反流病” (GERD in Chinese) on TikTok and Bilibili to reduce the bias of the platform’s algorithmic push videos, which have a deadline of October 31, 2023, for publication.

TikTok has three sorting modes, and integrated sorting is the default. The other modes include “latest published” and “most liked.” Since the most-liked mode can only sort for videos published within six months and most users use the default value, we collected the integrated sort to retrieve 100 videos. Bilibili uses play count sorting to collect videos with play counts in the top 100. Videos that are repetitive, irrelevant, about babies and toddlers, in languages other than Chinese and English and those with commercials were excluded.

All videos were collected and downloaded by a single individual; two investigators categorized video types and uploaders. The video links were provided in tabular form to the two raters, two longtime gastroenterology specialists. The order of the video links was disrupted to minimize scoring errors. Before evaluating the videos, the two raters carefully read the scoring details of the Journal of American Medical Association (JAMA), modified DSCERN (mDISCERN), and Global Quality Score (GQS). Both raters watched the videos independently at the same time, rated the videos, and categorized the videos according to source and content. If the scores did not agree between the two raters, a thorough discussion was held with the other observer to reach a consensus.

Video Characteristics

The video content analysis included days-since-upload, duration, views, likes, collections, comments, shares, coin toss, and time-sync comments (coin toss and time-sync comments specific to Bilibili).

Upload sources were categorized as medical professionals and non-medical professionals. Medical professionals including Gastroenterologist, Traditional Chinese Medicine (TCM), and other medical professionals, while non-professional individuals include non-profit organizations, Science communicators, and GERD patients.

The type of video includes information about the disease, treatment, and lifestyle.

Assessment Tools

The Journal of American Medical Association (JAMA) benchmark criteria (Table 1) were used to assess the reliability of the video source and the accuracy of the content.¹⁰ Scores range from 0–4, with higher scores representing more accurate and reliable videos.

Table 1 Journal of American Medical Association (JAMA) Benchmark Criteria

Modified DISCERN (mDISCERN) (Table 2) was used to assess the quality and comprehensiveness of video content. It contains five criteria: (1) The video is clear, concise, and understandable; (2) Sources of information used are reliable; (3) Information provided is balanced and unbiased; (4) Additional sources of information are listed; (5) Areas of uncertainty are mentioned, and judgments are made by whether or not, with a 1-point score for “yes” and a 0-point score for “no”.¹¹ The more “yes” the assessment contains, the better the quality and comprehensiveness of the video.

Table 2 Modified DISCERN (mDISCERN) Criteria

The Global quality scale (GQS) criteria¹² (Table 3) was used to assess the content value of GERD videos on a scale of 1–5, with higher scores representing higher quality content that is more likely to give viewers effective help.

Table 3 Global Quality Scale (GQS) Criteria

Statistical Analysis

Categorical variables are expressed as frequencies and ratios (%), and continuous variables are expressed as Mean±SD; since our data were nonparametrically distributed, the median (IQR) was used for the descriptive statistics. An unpaired t-test was used to compare two sets of normally distributed quantitative variables; the Kruskal–Wallis test was used for between-group comparisons of non-normally distributed quantitative variables; Mann–Whitney U-test was used to compare two groups of non-normally distributed quantitative variables; Spearman Correlation Analysis used to assess the correlation between quantitative variables. This research adopts SPSS25.0 software (developed by IBM Corp) for data analysis. Statistical significance was set at p < 0.05.

Results

Video Screening Process

“GERD” was used as the keyword for the search, and 100 videos were included from TikTok and Bilibili, respectively. According to the exclusion criteria, Bilibili excluded 28 irrelevant videos, three duplicates, and eight videos related to infants and children; TikTok excluded one duplicate and four videos related to infants and children, resulting in a total of 156 videos included in the study for further analysis. (Figure 1).

Figure 1 Video Selection Flowchart.

Video-Related Information

Table 4 lists the video features. TikTok uploaded videos posted between 2022.02.19 and 2023.10.19; Bilibili uploaded videos posted between 2018.01.10 and 2023.06.20. The largest number of TikTok video uploaders were gastroenterologists (n=66,69.5%), with a small number of uploaders in other identities (Traditional Chinese Medicine [n=23,23.2%] and other healthcare industry personnel [n=6,6.3%]); the largest number of Bilibili video uploaders were gastroenterologists (n=23,37.7%), followed by Traditional Chinese Medicine(n=17,27.9%) and other healthcare professionals (n=11.18.0%), with a small number of uploaders in other capacities (non-profit organizations [n=3,4.9%], personal scientists [n=,8.2%], and GERD patients [n=2,3.3%]). TikTok has more medically-related video uploaders, while Bilibili has video uploaders in more fields Figure 2.

Table 4 Detailed Characteristics of Gastroesophageal Reflux Disease

Figure 2 Distribution of sources of video uploaders.

Evaluate Videos Based on the Nature of the Video Uploader

Tables 5 and 6 shows the video characteristics of different uploaders in TikTok and Bilibili. There are three types of uploaders in TikTok and six in Bilibili, including gastroenterologists, Traditional Chinese Medicine, and Other medical professionals.

Table 5 Comparison of Video Sources According to Video Features on TikTok

Table 6 Comparison of Video Sources According to Video Features on Bilibili

On TikTok, the highest GQS score was given by other medical professionals (3.17), followed by gastroenterologists (2.91) and traditional Chinese medicine (2.78, p<0.05). The average length of videos uploaded by other medical professionals was 100 seconds (26–669), 256–7350 likes (median: 1990), 59–4555 Collections (median: 631), 91–1727 shares (median: 734), and 10–368 comments (median: 126).

On Bilibili, the highest JAMA score was given to gastroenterologists (2.82), followed by traditional Chinese medicine (2.65) and other medical professionals (1.82, p<0.05), and the highest GQS score was given to science communicators (2.80), followed by gastroenterologists (2.78) and other medical professionals (2.73, p<0.01). The average length of videos uploaded by gastroenterologists was 341.5 seconds (7–1219), 88–1462 likes (median: 336), 2–1033 number of coins (median: 90), 24–2403 collections (median: 188), 7–816 shares (median: 67), and 23–293 comments (median: 69).

Video Quality and Reliability Assessment

Analyzing the video content for both TikTok and Bilibili platforms (Figure 3), the content categories of the videos mainly included disease knowledge (TikTok: n=44; Bilibili: n=26), treatments (TikTok: n=24; Bilibili: n=17), and lifestyles (TikTok: n=44; Bilibili: n=36).

Figure 3 Comparison of video conte nt on the platform.

The number of videos featuring multiple contents simultaneously was higher (Table 7). TikTok videos predominantly consisted of disease knowledge content (n=34, 36%), while treatment and lifestyle content had the lowest proportion (n=4, 4%). There was a lack of categories that encompassed both disease knowledge and treatment. In Bilibili videos, lifestyle content accounted for a larger proportion (n=24, 39%), whereas disease knowledge and treatment content had the smallest proportion (n=2, 3%).

Table 7 Distribution of Video Content on the Platform

TikTok had significantly more likes, collections, shares and comments than Bilibili (P<0.005) (Table 8).

Table 8 Comparison of Video Features Between the Two Platforms

By comparing the scores, TikTok’s JAMA (2.95), GQS (2.87), and DISCERN (3.31) were higher than Bilibili’s, and the observed differences were statistically significant (P < 0.01, P < 0.005, and P< 0.001).

Correlation Analysis

Spearman correlation analysis revealed a high correlation (r: 0.8–1) between the following TikTok video parameters: likes and collections (r=0.921, P<0.01), likes and shares (r=0.952, P<0.01), likes and comments (r=0.944, P<0.01), collections and shares (r=0.929, P<0.01), collections and comments (r=0.888, P<0.01), and shares and comments (r=0.916, P<0.01; Table 9).

Table 9 The Relationship Level Between Video Variables on TikTok

The following variables in Bilibili video metrics exhibit high correlation (r: 0.8–1): collections and shares (r=0.887, P<0.01).

Variables with strong correlation (r: 0.6–0.8) include: views and likes (r=0.665, P<0.01), views and shares (r=0.623, P<0.01), likes and coins (r=0.650, P<0.01), likes and collections (r=0.785, P<0.01), likes and shares (r=0.795, P<0.01), likes and time-sync comments (r=0.659, P<0.01), coins and collections (r=0.768, P<0.01), coins and shares (r=0.727, P<0.01), coins and time-sync comments (r=0.613, P<0.01), time-sync comments and comments (r=0.686, P<0.01).

Variables with moderate correlation (r: 0.4–0.6) include: duration and coins (r=0.494, P<0.01), views and collections (r=0.554, P<0.01), views and time-sync comments (r=0.519, P<0.01), views and comments (r=0.478, P<0.01), likes and comments (r=0.561, P<0.01), coins and comments (r=0.507, P<0.01), collections and time-sync comments (r=0.547, P<0.01), collections and comments (r=0.451, P<0.01), shares and time-sync comments (r=0.561, P<0.01), shares and comments (r=0.445, P<0.01; Table 10).

Table 10 The Relationship Level Between Video Variables on Bilibili

For GERD videos on TikTok, the DISCERN score was positively correlated with duration (r=0.221, P<0.05). The JAMA score was negatively correlated with days since upload (r=−0.253, P<0.05). The GQS score was positively correlated with duration, collection, and shares (P<0.05, r=0.209; r=0.208 and r=0.202; Table 11).

Table 11 Pearson Correlation Analysis Between Video Quality Scores and Video Variables on TikTok

For GERD videos on Bilibili, the DISCERN score was positively correlated with duration (r=0.256, P<0.05). The JAMA score was negatively correlated with Time-sync comments (r=−0.259, P<0.05). The GQS score was positively correlated with video duration, likes, toss coins, collection, shares and time-sync comments (r=0.471, P < 0.01; r = 0.354, P < 0.01; r = 0.418, P < 0.01; r = 0.266, P < 0.05; r = 0.330, P < 0.01 and r = 0.262, P < 0.05; Table 12).

Table 12 Pearson correlation analysis between video quality scores and video variables on Bilibili

Discussion

GERD is defined by recurrent and troublesome heartburn and regurgitation or GERD-specific complications, is one of the most common chronic diseases globally, consuming large amounts of healthcare and societal resources, leading to a reduced quality of life as well as an increased risk of serious complications.¹³ The overall global prevalence of GERD is 13.98%, with wide variations between countries and regions.¹⁴

The American Gastroenterological Association (AGA) and American College of Gastroenterology (ACG)^7,15 suggest that up to 50% of patients, however, do not derive adequate relief with empirical proton pump inhibitor (PPI) therapy and that long-term use or abuse of proton pump inhibitors can increase the incidence of adverse events, in addition to suggesting that dietary and lifestyle changes can be effective in relieving GERD symptoms. This medical knowledge is easily accessible and understandable on social platforms. Therefore, social media can be an effective tool for relevant professionals to raise public awareness of GERD, proper lifestyle and individualized treatment by actively searching or ensuring there are multiple pushes to the target group, which can conveniently and effectively allow patients to reap the benefits of the information.

However, social media has become a vehicle for misinformation, manipulation, and malicious influence. Feed algorithms classify user preferences by collecting behavioral data to match users with precise and continuous information. The misuse of algorithms by a number of video platforms can lead to the spread of misleading information, misinformation,^9,16 whereas public health agencies can respond quickly to the dissemination of misinformation on more conventional platforms and correct it.¹⁷

Video analysis of the two social media platforms showed that the source of the video uploaders was mainly medical staff, who were professionally certified by the platform and obtained identification, which reduced the spread of misinformation and disinformation to a certain extent. Bilibili videos uploaded by foreign content creators and a larger number of non-professionals, as well as videos that have been authorized for translation into Chinese by YouTube uploaders. This can improve the reliability of the video information if the platform can professionally scrutinize the videos in question. The analysis results indicate that TikTok’s videos exhibit higher popularity, BiliBili’s videos have longer durations, and TikTok outperforms Bilibili in terms of JAMA, GQS, and DISCERN scores. This suggests that videos with higher scores are more likely to attract viewers.

This study used one more JAMA scoring tool than the study by Ying Cai,¹⁸ which was to assess the reliability of the video source and the accuracy of the content, and TCM-related videos were included in the study because Chinese patients would consult TCM practitioners about their illnesses, and the results of the study by Ying Cai et al showed that there was no variability in scores between the two video platforms, which was not in consistent with the findings of this paper, which may require a more in-depth study with other scoring tools to analyse the variability.

This study also investigated the correlation between various parameters of the video, the scores of the three scoring tools and various parameters of the video, we found that video JAMA scores on Bilibili were negatively correlated with Time-sync comments, and video JAMA scores on TikTok were negatively correlated with the number of days since upload had been uploaded, which may be related to the timeliness of the video, the platform push mechanism, the more recent the time, the more likely that the video will be actively Push. In our study we found that lower-quality videos had better popularity. The TikTok recommendation algorithm, as pointed out by Fei Sun,¹⁹ determines that videos with a higher number of likes are more likely to be recommended. Consequently, this leads to the increased popularity of low-quality popular videos, further exacerbating the disparity between video quality and popularity. In addition, we found that most of the videos did not upload relevant references, did not list other sources of information, and did not present the treatment comprehensively enough, among other reasons that can lead to lower video ratings. DISCERN scores on both platforms were positively correlated with duration, suggesting that the longer the video, the more comprehensive and informative the uploader’s explanations of the various aspects of GERD.

Some study^20,21 analyzed the videos of GERD on YouTube and concluded that the quality and reliability of the videos uploaded by healthcare professionals were better than that of non-professionals and that the video quality was better when animation was added. If these platforms work with public hospitals or on certain high-traffic sites, while improving the quality of information on social platforms by further clarifying the objectives, increasing the relevance of the information to the objectives, and providing relevant additional support and details of the sources of the information wherever possible, this could increase positive attitudes of users towards the platforms.²²

Healthcare professionals and organizations should be encouraged to provide more reliable scientific knowledge and animated videos to viewers seeking comprehensive and reliable information on the Internet, with relevant references appropriately attached for viewers’ reference and judgment and to provide subsequent updates. At the same time, medical-related videos uploaded on both video platforms were uploaded by individuals and the authenticity of the video content was not vetted. The platforms should create better vetting mechanisms and improve the provisions of the relevant video policies to minimize the dissemination of misleading and misrepresentative videos. The platforms should establish better censorship mechanisms and improve the provisions of the relevant video policies to minimize the spread of misleading and misinformed videos.

Our findings can help GERD patients to be able to find high-quality videos more easily, so that patients can change their own behavioral habits and improve their self-management skills based on relevant high-quality level of videos, reduce the burden of healthcare, and the combination of behavioral change and medication therapy improves the efficacy of medication, as well as providing a basis for improving the reliability of the videos and a direction for improvement.

Limitations and Future Study

First of all, we only use a single keyword, “GERD”, to retrieve the video. Since most people do not know the proper medical terms, we should consider adding easy-to-understand keywords relating to symptoms, such as reflux, heartburn, etc. Moreover, only the top 100 videos of the search list were included in the study, and after exclusion, there were less valid data.

This study only assessed the quality of GERD videos from two platforms with Chinese videos; more platforms and languages such as TikTok Global Edition and YouTube, should be included in future studies for comparative analysis. Due to the platform algorithms, searching in different regions and at different times may lead to different search results.

In conclusion, TikTok and Bilibili videos on GERD were able to provide the public with valuable information on relevant disease knowledge, treatment methods, and good lifestyles, but most of the videos scored mediocre in terms of content and quality, and need to be further optimized and improved to ensure that the public has access to accurate and effective knowledge on GERD.