Preliminary Development of a Culturally Adapted Questionnaire on Mental Health Literacy for Chinese Junior High School Students

Introduction

In recent years, junior high school students’ mental health conditions have become increasingly severe. If mental disorders are not promptly identified and effectively treated, they can have negative effects and impose substantial burden on individuals, families, society, and nations, and some of these adverse effects may persist into adulthood, compromising the quality of life of those affected.¹ However, it is challenging for junior high school students to recognize the mental health issues they face, and even if they recognize such issues, they are typically unwilling to seek professional psychological help.² The primary reason for this is the relatively low level of mental health literacy (MHL) among junior high school students.

Table 1 Rotated Factor Loading Matrix for 31 Items

The concept of MHL was first proposed by Jorm et al in 1997.³ Initially, they defined it as

knowledge and beliefs about mental disorders which aid their recognition, management, or prevention. (p 182)³

With the continuous deepening and development of related research, scholars have enriched and perfected the conceptual connotations of MHL. Currently, the most widely accepted view comes from the Canadian Alliance on Mental Illness and Mental Health.⁴ It states that MHL includes understanding how to achieve and maintain good mental health, knowledge about mental disorders and their treatments, reducing stigma related to mental disorders, and enhancing help-seeking efficacy (knowing when and where to seek help, developing skills to improve mental health, and self-management).⁵ When MHL was introduced to China, Chinese scholars conducted localized research based on their cultural context. Jiang et al proposed a relatively systematic and comprehensive concept, defining MHL as

the knowledge, attitudes, and behaviors developed by individuals in promoting their own and others’ mental health, and in addressing their own and others’ mental disorders. (p 232)⁶

MHL is closely associated with one’s mental health status. Teixeira et al found a significant positive correlation between MHL and positive mental health status, and a significant negative correlation with psychological vulnerability.⁷ Some intervention studies have also found that enhancing MHL can increase individuals’ awareness of preventing and proactively addressing mental disorders, reduce the stigma associated with mental illness, and increase individuals’ willingness to seek professional help.^8–10 In summary, enhancing MHL is crucial for promoting individual mental well-being.

The aforementioned is especially true for junior high school students, as they are in a crucial period for enhancing their MHL.⁴ One reason is that during this period, the prevalence of various mental disorders is notably elevated,¹ and by enhancing MHL, such disorders can be prevented and their negative impact, mitigated.⁸ Another reason is that these students’ attitudes and behaviors are highly malleable. Therefore, an accurate and scientific assessment of MHL provides a foundation for further investigation.

Existing research has examined the current state and characteristics of MHL among Chinese junior high school students. However, owing to variations in sample selection, measurement instruments, and evaluation criteria, the findings have been inconsistent. Some studies suggest that the MHL of Chinese junior high school students is at a low¹¹ or below-average level,¹² while other investigations indicate that this population’s MHL is at an average¹³ or above-average level.¹⁴ Furthermore, current research on the differences in MHL among variables such as gender and grade is not entirely consistent.^12,15 The inconsistency in the research findings hinders a comprehensive understanding of the current status and characteristics of MHL among Chinese junior high school students. Hence, it becomes challenging to develop targeted interventions to enhance their MHL.

Scholars have recognized the importance of measurement tools and accordingly developed some instruments to assess MHL. However, most of these scales have been created based on Western cultural contexts. For instance, the questionnaire developed by Jorm et al presents several situational vignettes describing patients with certain mental illnesses. After their presentation, a series of open-ended questions are posed around each vignette to examine participants’ knowledge about mental illnesses, their ability to recognize and respond to mental illnesses, and their attitudes toward the stigma associated with these conditions.³ O’Connor et al further developed a standardized scale to measure MHL; this consists of 35 items across three dimensions based on Jorm’s theoretical framework.¹⁶ While these scales have demonstrated good reliability and validity in international studies, researches have shown that the conceptualization of MHL can be influenced by cultural contexts. A cross-cultural study conducted in the United Kingdom, Hong Kong, and Malaysia revealed significant differences in participants’ ability to recognize mental disorders and their attitudes toward professional help-seeking across these sociocultural backgrounds.¹⁷ Furthermore, a qualitative study by Wu et al involving semi-structured interviews with Chinese experts from various fields found that the concept of MHL in the Chinese cultural context encompasses unique dimensions such as leading by example, harmonious coexistence with others, and maintaining balance in various aspects of life.¹⁸ Therefore, MHL questionnaires developed based on Western cultural frameworks may not adequately capture the characteristics of MHL among Chinese populations.

In China, scholars have developed several instruments to measure MHL. For instance, Wu et al constructed the “National Mental Health Literacy Questionnaire” with 60 items for the general adult population in China.¹⁹ Li developed the “Adolescent Mental Health Literacy Assessment Scale” targeting medical students.²⁰ Although these questionnaires have overcome the issue of cultural differences, they may not fully capture the developmental characteristics of MHL among junior high school students. Scholars have pointed out that the features of MHL can vary across different age groups and occupational populations.²¹ This observation has been supported by the qualitative research conducted by Ma et al²² on the MHL of Chinese armed police officers. Junior high school students are in the early stage of adolescence characterized by distinct psychological development, and they face unique mental health challenges compared with other age groups. Therefore, a measurement tool that specifically considers the physical and psychological developmental characteristics of this population is necessary to accurately and scientifically measure its MHL.

Some researchers believe that the dimensional structure of MHL only refers to knowledge related to mental health (illness),^23,24 while others argue that MHL includes not only the knowledge dimension, but also dimensions of attitudes, skills, and behaviors.^6,25 A comprehensive analysis of the existing structural dimension models reveals that the main debate among researchers is whether to incorporate knowledge, attitudes, and behaviors simultaneously into the conceptual framework of MHL. To address this issue, the relationship between “knowledge” and “action” must be clarified.

In the history of Chinese philosophy, Wang Yangming’s concept of “unity of knowledge and action” has served as a paradigm for addressing the aforementioned relationship. This concept holds that knowledge contains action, and action contains knowledge; thus, knowledge and action are inseparable.²⁶ Accordingly, Chen and Li define MHL as

the comprehensive integration of favorable elements for the development of mental health acquired by individuals in their post-natal environment. (p 121)²⁷

and propose that its connotative structure includes three dimensions: knowledge, attitudes, and behavioral tendencies. Mental health knowledge refers to theoretical content related to mental health or disorders acquired through learning. It is the primary component of MHL and varies according to one’s social and cultural background and age group. It is objective and distinguishes between right and wrong. It is the most easily assessable and intervenable of the three MHL dimensions. Some mental health knowledge can transform into mental health attitudes or behavioral tendencies. Mental health attitudes refer to people’s views on matters or behaviors related to mental health (disorders); they are the motivation for behavioral change, capable of transforming into mental health behavioral tendencies or influencing the acquisition of mental health knowledge. Mental health behavioral tendencies refer to the inclination to perform behaviors that prevent disorders or maintain mental health. It is the most stable of the three MHL dimensions, meaning that it is not easily altered by interventions. This dimension is an ideal indicator for assessing MHL levels; however, it is also the most challenging to measure.

The spiral model also elaborates on the dynamic relationships among its three dimensions. When mental health knowledge levels improve, part of it is converted into positive mental health attitudes and behavioral tendencies, while another portion may diminish over time. The transformation from mental health knowledge to attitudes and behavioral tendencies can also be acquired through practice, experience, and other implicit learning processes. Positive mental health behavioral tendencies can further encourage individuals to enhance their mental health knowledge, which would allow their MHL to undergo transformative changes among dimensions at a higher level, thus exhibiting a spiral form of change.

Furthermore, based on structured interviews with junior high school teachers, Chen and Li proposed a “three-dimension and nine-domain” framework for the MHL of junior high school students (Figure 1).²⁷ Specifically, the knowledge dimension included fundamental knowledge of mental health, mental disorders, and psychological counseling. The attitude dimension encompassed attitudes toward mental disorders, awareness of mental health, attitudes toward others’ mental health issues, and self-efficacy in dealing with mental health issues. The behavioral tendency dimension included the tendency to actively seek professional help and resources related to mental health.

Figure 1 Theoretical structure diagram of the MHL questionnaire for junior high school students.

In this study, we rigorously followed the basic principles and standardized procedures of questionnaire development, conducted expert evaluations and large-sample testing, and performed statistical analysis to develop a questionnaire with reliable psychometric properties suitable for assessing the MHL of Chinese junior high school students. Our goal was to provide a scientifically valid, standardized measurement tool for assessing the current status of MHL among this population. This could help in developing targeted intervention programs and evaluating the effectiveness of such programs.

Informed consent was obtained from all individual participants and their parents included in this study. The research protocol was approved by the ethics committee of the Institute of Psychology and Behavior of Henan University (No. 20211101002).

Study 1: Questionnaire Development

Item Pool Acquisition

Based on the theoretical framework of MHL among junior high school students, the questionnaire was divided into three dimensions: mental health knowledge, attitudes, and behavioral tendencies. The development of items for the Junior High School Students’ MHL Questionnaire involved the following steps:

1. Items were developed based on the structural dimensions of a “three-dimension and nine-domain” MHL for junior high school students;
2. Items from existing domestic and international questionnaires or scales were adapted to better align with the reading and cognitive characteristics of junior high school students, and
3. Items were devised according to MHL theory, tailored to real-life contexts of junior high school students.

Ultimately, a pool of 60 items was established for our questionnaire. Consistent with previous methodologies applied in the MHL field for assessing knowledge,¹⁹ the knowledge section of the questionnaire used a true-or-false format, whereas the sections on attitudes and behavioral tendencies employed a Likert-type five-point scale. All correct answers were determined through discussions among all members of the research team until a consensus was reached.

Expert Evaluation

Eight experts from related fields were invited to evaluate the initial questionnaire. These included two experts in psychiatry, two in psychological counseling, and four in adolescent development and education. All experts held doctoral degrees and were well-acquainted with MHL. They rated the content validity (appropriateness of items for measuring specific MHL components) of all 60 items on a scale of 1 to 5, and provided feedback on item wording.

Fifteen items scored below four for content validity. Based on expert ratings and suggestions, the following two items were deleted: “If I encounter mental health issues, I have the ability to ask questions and seek advice from mental health service providers” and “Even a psychologically healthy individual can benefit from psychological counseling to facilitate self-growth.” Additionally, 21 items were revised or replaced. For example, the item “Weak willpower and an inability to withstand setbacks are manifestations of poor mental health” was changed to “Inability to withstand setbacks is a sign of poor mental health.” Two new items were added, namely, “I can reasonably evaluate and screen the quality of mental health resources” and “If my friends or family members encounter mental health issues, I would recommend that they seek help from professional psychologists.” After these modifications, the questionnaire comprised 60 items, with 29, 19, and 12 items in the knowledge, attitude, and behavioral tendencies dimensions, respectively.

Study 2: Item Testing

Participants and Procedure

Convenience sampling was used to select junior high school students from several schools as participants. We distributed the survey and collected data using Wenjuanxing online platform. After excluding participants with excessively long or short response times and those outside the target age range of middle school students, we obtained a final sample of 510 valid participants. Among the participants, 213 were men and 297 were women; their average age was 14.29 years (SD = 0.97); 124 were in grade 7; 158 were in grade 8, and 228 were in grade 9.

This study employed the two-parameter logistic model from item response theory for item analysis, and used Mplus 8.0 software to analyze data from the initial version of the MHL questionnaire. Exploratory factor analysis was conducted using SPSS 25.0.

Results

Item Analysis

First, we calculated the discrimination (a) and difficulty (b) parameters for each item of the initial questionnaire. Referring to previous studies^28,29 and following our research group’s criteria, items with a discrimination parameter of a < 0.5, or difficulty parameter of b < −3 or b > 3 were removed, resulting in the deletion of 16 items. The revised questionnaire retained 44 items, comprising 17, 16, and 11 items for the knowledge, attitude, and behavioral tendencies dimensions, respectively.

Item characteristic curves (ICCs) describe the relationship between participant ability levels, item parameters, and response outcomes. An ideal ICC is an S-shaped curve that visually represents item difficulty and discrimination. Using Mplus 8.0, ICCs were plotted for the 44 items. Some items displayed ideal ICCs (Figure 2), but others did not meet the quality standards for difficulty and discrimination, resulting in less-than-optimal ICCs (Figure 3).

Figure 2 Example of item characteristic curve for ideal items.

Figure 3 Example of item characteristic curve for suboptimal items.

Exploratory Factor Analysis

We employed the minimum average partial method for factor extraction,³⁰ which resulted in four identifiable factors. Based on the analysis outcomes, we restricted the number of factors to four and conducted exploratory factor analysis on the remaining 44 items using maximum likelihood and optimal oblique rotation methods. The results showed that 13 items functioned poorly as they were cross-loaded in multiple factors or had factor loadings less than 0.40.³¹ The factor loading matrix for the retained 31 items is shown in Table 1.

Theoretical dimensions were determined based on the items contained in each factor. Items under Factor 1 belonged to the Behavioral Tendencies dimension, those under Factor 2 (except Item 37) belonged to the Knowledge dimension, and those under Factors 3 and 4 belonged to the Attitude dimension. After making theoretical considerations, Factors 3 and 4 were merged, and Item 37 was deleted owing to interpretative difficulties, leaving a final set of 30 items for the Junior High School Students’ MHL Questionnaire (Formal Version), which encompasses three dimensions, namely, Knowledge (9 items), Attitude (12 items), and Behavioral Tendencies (9 items).

Study 3: Psychometric Evaluation

Participants and Procedure

Junior high school students were selected from multiple schools using convenience sampling. After communicating with school administrators or teachers, informed consent forms, response instructions, and questionnaire QR codes were distributed to homeroom teachers, who then read the consent forms and instructions to the students. We distributed the survey and collected data using Wenjuanxing online platform. After excluding participants with excessively long or short response times and those outside the target age range of middle school students, we obtained a final sample of 551 valid participants. Among the participants, 261 were men and 290 were women; the average age was 14.08 years (SD=0.86); 139 were in grade 7; 321 were in grade 8, and 91 were in grade 9.

Data processing and analysis were conducted using Mplus 8.0 and SPSS 25.0. Confirmatory factor analysis was performed using Mplus 8.0, and reliability and validity analyses of the questionnaire were conducted using SPSS 25.0.

Materials

The Junior High School Students’ MHL Questionnaire included three dimensions with a total of 30 items, specifically 9, 12, and 9 items in the Knowledge, Attitude, and Behavioral Tendencies dimensions, respectively. The Knowledge dimension items were presented as true-or-false questions with a third “Uncertain” option. Respondents judged their answers based on their knowledge, with correct answers scoring one point and incorrect or uncertain answers scoring no points. For example, for item 19 “Confiding troubles to friends is a good way to maintain mental health”, selecting the “true” option would earn one point, while selecting the “false” or “uncertain” option would both earn no points. Items in the Attitude and Behavioral Tendencies dimensions used a five-point Likert scale. Respondents rated their agreement ranging from “totally disagree” to “totally agree” (from 1 to 5 points, respectively), with reverse scoring applied for negatively stated items. For comparability, the Likert scale scores were converted to binary scores equivalent to true-or-false judgments.¹⁹ Specifically, options rated 4 or 5 scored one point, whereas options rated 1–3 scored zero. For example, for Item 55 “If I encounter emotional distress, I will seek help from professional psychology practitioners”, selecting the “moderately agree” or “totally agree” option would earn one point, while selecting the other options would earn no points. The sum of the scores for all 30 items constituted the total score, which ranged from 0 to 30, with higher scores indicating better MHL.

The Attitudes toward Seeking Professional Psychological Help Scale-Short Form(ATSPPH-SF) by Fischer et al comprises ten Likert-scale items scored from 0 (“strongly disagree”) to 3 (“strongly agree”), including five reverse-scored items (Items 2, 4, 8, 9, 10).³² Higher scores indicate more positive attitudes toward seeking professional help. The Cronbach’s α for this scale in this study was 0.74, and McDonald’s ω was 0.81. This questionnaire was used to examine the criterion-related validity of our MHL questionnaire.

The Stigma Scale for Receiving Psychological Help (SSRPH) by Komiya et al consists of five Likert-scale items scored from 0 (“strongly disagree”) to 3 (“strongly agree”), where higher scores reflect stronger stigma toward receiving psychological help.³³ In this study, the scale’s Cronbach’s α was 0.87, and McDonald’s ω was 0.93; it was also used to assess the criterion-related validity of our MHL questionnaire.

Results

Confirmatory Factor Analysis

This study employed the robust weighted least squares method to conduct confirmatory factor analysis to examine the structural validity of the questionnaire. The fit indices for the single-factor, first-order three-factor, and second-order three-factor models are presented in Table 2.

Table 2 Fit Indices for MHL Questionnaire

We used multiple indices such as χ², χ²/df, root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker-Lewis index to assess the model fit. According to traditional evaluation standards, if χ²/df < 5, RMSEA < 0.08, and Tucker-Lewis index and CFI ≥ 0.90, the model fit is considered acceptable As shown in Table 2, both the first-order and second-order three-factor models met these criteria. A comparison of the fit results of the single-dimensional models with those of the three-factor models confirmed that the latter was optimal. According to Park et al,³⁴ a significant difference in model fit occurs if ΔCFI ≥ 0.01 and ΔRMSEA ≥ 0.015. The comparison revealed that the three-factor model significantly outperformed the single-dimensional model (ΔCFI = 0.059 ≥ 0.01, ΔRMSEA = 0.021 ≥ 0.015). Considering the need to aggregate the total scores for practical applications, a second-order three-factor model was selected as the optimal structure. These results confirmed the validity of the questionnaire.

Reliability Analysis

We employed Cronbach’s α, McDonald’s ω, and a one-month interval test-retest reliability to assess the questionnaire’s reliability.

The results of the internal consistency reliability assessment of the Junior High School Students’ MHL Questionnaire are presented in Table 3. In this questionnaire, the α coefficient for items scored using the Likert scale was 0.93, (0.86 and 0.91 for the Attitude and Behavioral Tendencies dimensions, respectively). Given the conversion to binary scoring, we also computed the KR20 coefficient, which showed reliability coefficients of > 0.80 for the total scale and > 0.70 for the sub-dimensions, indicating satisfactory stability.

Table 3 Results of the Reliability Analysis (N=551)

Given the limitations of Cronbach’s α,³⁵ we supplemented it with McDonald’s ω. Results showed that all McDonald’s ω coefficients exceeded 0.90, thus confirming satisfactory reliability. The results are presented in Table 3.

For the 147 junior high school students tested one month apart, significant correlations were observed between the total scores and scores on the three dimensions, indicating good stability. The test-retest reliability of the total score of MHL was 0.88 (p < 0.01). The test-retest reliability coefficients for the knowledge, attitude, and behavioral tendency dimensions were 0.67 (p < 0.01), 0.75 (p < 0.01), and 0.75 (p < 0.01), respectively. These results indicate that the Junior High School Students’ MHL Questionnaire has satisfactory temporal stability.

Validity Analysis

Three types of validity assessments were performed: content, criterion-related, and construct validity. Experts in the field assessed the questionnaire for its theoretical rationality, appropriateness, representativeness, and comprehensiveness, confirming good content validity.

To examine criterion-related validity, we assessed the extent to which the MHL correlated with the ATSPPH and SSRPH. As shown in Table 4, MHL was significantly positively correlated with the ATSPPH (r = 0.51 p < 0.01), and significantly negatively correlated with the SSRPH (r = −0.29, p < 0.01), indicating that the Junior High School Students’ MHL Questionnaire has good criterion-related validity. Exploratory and confirmatory factor analyses corroborated the questionnaire’s sound construct validity.

Table 4 Results of Correlation Analyses

Discussion

The Junior High School Students’ MHL Questionnaire produced in this study was developed based on the Chinese cultural context and followed a rigorous scale development process. Initially, a pool of 60 items was constructed based on interviews with junior high school teachers and references to existing scales, to ensure adherence to statistical measurement principles. Expert reviews ensured high content validity, and were followed by large-scale testing and item analysis, which reduced the items to a final set of 30. The reliability and validity analyses provided empirical support for the questionnaire’s structure and dimensions, ensuring its scientific rigor and reliability as a measurement tool. The questionnaire can be used to assess and compare MHL across various dimensions among junior high school students, and to evaluate the effectiveness of MHL intervention programs.

Validity of the Questionnaire

Validity analysis is crucial for measuring the effectiveness of a tool. This study examined the validity of the Junior High School Students’ MHL Questionnaire based on three aspects: content, criterion, and construct. Experts in the field of mental health unanimously agreed that the dimensions and content of the questionnaire could comprehensively and effectively reflect the current state and characteristics of the MHL of junior high school students, indicating that the questionnaire had ideal content validity. For criterion validity, attitudes toward seeking professional psychological help and stigma toward receiving psychological help were selected as criteria. Correlation analysis revealed moderate correlations between the Junior High School Students’ MHL Questionnaire scores and the two criteria, thus meeting the consistency and heterogeneity requirements in psychometrics and demonstrating good criterion validity of the questionnaire. Based on the Maximum A Posteriori analysis results, we limited the number of factors to four and conducted an exploratory factor analysis on the retained items. The results showed that the statistical calculation results could well support the theoretical framework of a three-dimensional structure of MHL. The results of the confirmatory factor analysis also indicated that the questionnaire had good construct validity. Overall, the questionnaire aligned well with the “three-dimension and nine-domain” theoretical model in terms of content and factor composition, which indicates that it can measure the psychological traits expected from our theoretical framework.

Reliability of the Questionnaire

Reliability indicates the consistency of a measurement instrument. Given that the knowledge dimension of the Junior High School Students’ MHL Questionnaire used a distinct scoring approach compared with the other dimensions, internal consistency coefficients were computed independently for each. The findings revealed that the internal consistency coefficients and composite reliability for all dimensions exceeded 0.80, which aligns with psychometric criteria and affirms the questionnaire’s strong reliability.

Advantages of the Questionnaire

Although scholars from both China and abroad have developed MHL tools,^3,16,19,36 no tool has been specifically designed for junior high school students. The questionnaire developed in this study was theoretically based on Wang’s philosophy of the unity of knowledge and action, and primarily sourced its item pool from interviews with junior high school teachers, thus reflecting the MHL characteristics of Chinese junior high school students. Previous research and teacher interviews indicated that the psychological issues commonly encountered at this stage are developmental rather than related to mental illness;³⁷ therefore, compared with existing domestic and international tools, this questionnaire contains fewer items related to mental illness and more items related to common psychological issues among junior high school students, and thus aligns well with their cognitive levels and psychological traits.

Limitations and Future Directions

This study is subject to several limitations. First, the sample size was restricted; therefore, future studies are warranted to evaluate the questionnaire’s reliability and validity in a larger sample. Second, the questionnaire’s predictive validity was not evaluated; hence, further research is needed to validate and assess its predictive capacity. Third, a convenience sampling method was employed, potentially restricting the generalizability of the findings. Lastly, the questionnaire was culturally tailored to China and focused on the psychological development traits of junior high school students, which potentially constrains its cross-cultural transferability. Future research may test and refine this questionnaire in various cultural contexts to enhance its broader utility.

Conclusion

The Junior High School Students’ MHL Questionnaire, rooted in traditional Chinese cultural theory and tailored to the psychological development traits of junior high school students, demonstrates robust psychometric qualities. Comprising 30 items distributed among dimensions of knowledge, attitudes, and behavioral tendencies, this questionnaire is more suitable for Chinese junior high school students than other MHL assessment tools. The results also offer insights for evaluating MHL in diverse populations.