Usability Evaluation of Foods with Function Claims Labelling as Health Information in Japan: A User-Testing Study

Introduction

Health Information Provision and Consumer Understanding in Japan

Numerous health foods saturate the Japanese market. These are not defined by law in Japan but refer to the whole of a food widely sold or used as food contributing to health conservation and enhancement.¹ However, the prevalence of inaccurate and unreliable health information sources can mislead consumers, potentially leading to inappropriate use of the product and associated health risks.¹ Health information materials serve as crucial tools for effective risk communication.

An online survey in 2016 revealed that only 16% of consumers clearly understood the characteristics of Foods with Function Claims (FFCs).² Another survey in 2017 reported that 17% of consumers using health food products experienced poor physical conditions.^3,4 In March 2024, tragically, 81 people were fatally poisoned by FFCs containing beni kōji fermented rice in Japan.⁵ While the incident was likely caused by a contaminant, it underscored the challenge consumers face in checking the safety of FFCs, which are readily available. Consequently, the provision of easy-to-understand information is crucial to ensuring safe product usage and empowering consumers to make informed choices. A comparative analysis of Japanese and European consumer health literacy surveys^6,7 indicated that 41.8% of respondents in Japan, 36.2% in Europe, and 30.1% in the Netherlands had difficulty understanding information on food packages.^6,7

Given the health literacy gap between professionals and consumers, establishing a communication system that ensures the quality of information aligns with consumer needs is imperative.

Previous studies investigated consumers’ comprehension of the nutrition facts label, health claims, and food labels using online surveys, including questionnaires.^8–10 A qualitative study was conducted to investigate how claims can affect consumers’ perceptions and behavior.¹¹ While these studies investigated food labelling, they were not specific to FFC labelling. Although surveys have been conducted in Japan on consumers’ awareness and attitudes towards FFC,¹² no surveys have been conducted to assess HCPs’ and consumers’ perspectives on labelling.

Currently, there is no system available in Japan for evaluating the usability of health information materials. To enhance the utility of these materials, it is vital to evaluate information from the perspective of HCPs and further verify it from the users’ perspective. Previously, we developed a usefulness evaluation index for FFC labelling from the HCPs’ or providers’ perspective.¹³ In this study, we developed and evaluated a user testing to gauge the accessibility and comprehensibility of the same FFC materials from consumers’ perspectives. In addition to user testing, we conducted interviews with a qualitative analysis of the comments obtained from the consumers. The development of these integrated methods considering the HCPs’ and consumers’ perspectives represents the first study on the comprehension of health information using FFC labelling.

Labelling of Foods with Health Claims in Japan

Based on the Health Promotion Law, the “Foods with Health Claims” system was established in April 2015 to facilitate the appropriate use of such foods for self-care.^14,15 This system comprises Foods for Specified Health Uses (FOSHU), Foods with Nutrient Function Claims, and FFCs (Figure 1). FOSHU undergoes “Individual Evaluations” for efficacy and safety and is approved by the Secretary General of the Consumer Affairs Agency (CAA).¹⁶ In contrast, FFCs can display function claims based on scientific evidence, with the responsibility lying on the food business operator. Prior to marketing, information supporting the safety and efficacy of the product is submitted to the Secretary-General of the CAA.¹⁷

Figure 1 Classification of Food with Health Claims. Note: The copyright of the FOSHU approval seal belongs to the Consumer Affairs Agency, but prior permission is not required for its use. Available from the following website. https://www.caa.go.jp/policies/policy/food_labeling/foods_for_specified_health_uses.

As of 18 August 2024, there were 8683 notified FFCs,¹⁸ while 1042 FOSHU products received approval.¹⁹ An FFC must feature 16 specified items (Cabinet Office Ordinance No. 10, 2015; Table 1 and Figure 2). In addition, FFC labels should bear the following: the product’s name, storage method, best before date or expiration date, ingredients, additives, nutritional ingredients, total weight, calorific value of nutritional ingredients, and the name and address of the food business operator.²⁰ In the actual labels of the product containers and packaging, the order, font size, and position of these items differ from the examples given by the CAA (Figure 2).

Table 1 Labelling Items on Containers and Packaging of Foods with Functional Claims

Figure 2 Model of the label for foods with function claims by the Consumer Affairs Agency. Note: The labelling is displayed on box-type packages, but it is recommended to use the same wording for bottles and other packaging formats by the Consumer Affairs Agency.

Evaluation of FFC Labelling

Evaluation from the Perspective of HCPs

Foods with Health Claims should present clear information on their labels so that the information is easily understandable to consumers with diverse levels of health literacy. In recent years, public organizations in Europe and the United States have introduced standards to facilitate the creation and provision of health information that is easily understandable for consumers and patients. In the United States, various tools such as “Clear & Simple”²¹ and “Toolkit for Making Clear and Effective Information”²² are available. Notably, the Centers for Disease Control and Prevention (CDC) released the “Clear Communication Index (CCI)” in 2014 as a research-based tool for developing and accessing health communication materials.²³ The CCI comprises 20 items, including the main message and action recommendations, with the CDC recommending a score of 90% (18 items) or higher. In this context, a group comprising six university employees, all of whom were qualified as pharmacists and public health professionals, has developed our own CCI for evaluating the FFC labelling (F-CCI) (Table 2).

Table 2 Clear Communication Index for Evaluation of FFC Labelling (F-CCI)

Using the F-CCI index, we assessed five FFC products from the perspective of healthcare professionals (HCPs), achieving an approximate level of 70% (12–14 items), details of which have been previously published.¹³ None of the five products met the acceptance criteria for the following questions: “Does the material consistently use language familiar to the primary audience? (F-CCI Q7)”, “Is the most important information that the primary audience needs summarized in the first paragraph or section? (Q10)”, and “Does the material consistently explain the meaning of the numbers and units used? (Q16)”. With regard to Q10, usage precautions, such as advising immediate discontinuation of product usage and recommending consultation with a doctor if any physical changes are noticed, were described at the bottom of the label without any particular emphasis. For Q7, certain sentences indicated by the CAA included technical jargon that was not commonly used by the public. The results indicated that the readability and location of the main message, in particular, should be improved.

Evaluation of FFC Labelling Through User Testing

In addition to evaluation from the perspective of HCPs, it is imperative to assess FFC labelling from the users’ standpoint. Therefore, a comprehensive evaluation from both perspectives is crucial.

User testing is used to assess whether users can easily access and understand FFC.^24–26, It is widely used for assessing the efficacy of health information, ranging from booklets and leaflets to online resources. User testing aims to enhance the understanding of the information provided to consumers and patients.^25–29 The interviewer asked participants to answer questions about the content of the information materials. When conducting user testing, it is recommended to employ a cohort of 10 participants at a time. This approach is well-established and supported by European Union (EU) and Australian guidelines, and meeting user-testing criteria is one of the conditions for the approval of new medicines in the EU.^27–29 This methodology has been widely used, as demonstrated by Raynor et al.^30–32

Our initial user testing in Japan targeted the Drug Guides for Patients, which are the label information of prescription drugs for patients.³³ Subsequently, we have continued user testing and gained experience in this field.³⁴ In the test, the passing criterion is as follows: 90% of the participants should successfully locate and understand the information.

In this study, we evaluated five FFC labels using the F-CCI and user testing on five FFCs (Figure 3). Interviews with 50 participants (five cohorts of 10 participants) in the user testing were conducted to gain insights into users’ attitudes and enhance the overall quality of the FFC labelling.

Figure 3 Integrated evaluation framework of usability of health information materials. Note: This system evaluates health information material through healthcare professionals (experts) and consumers (laypersons). Experts conduct the initial evaluation, followed by user testing with consumers.

Materials and Methods

Materials

On the FFC search site provided by the CAA,¹⁸ we searched for FFC relevant to keywords “triglyceride”, “presbyopia”, “absorption of sugar and fat”, “hypertension”, and “cholesterol”. These topics are of particular interest to middle-aged and older adults. After reviewing approximately 100 labelling of FFCs in a preliminary study, we selected five products, each with distinct claims of functionality that were considered commonplace. Table 3 provides an overview of these five products. Subsequently, we purchased each product and evaluated its labelling content. The labelling and labelling sample (Form VI of the submitted claim) can be found on the CAA website (accessed on 5 January 2020).

Table 3 Characteristics of Foods with Function Claims Subjected to User Testing

Table 4 Characteristics of Participants in the User Testing of Foods with Function Claims

Participants

For generalizability of the outcomes of user testing, it is imperative to carefully recruit a participant sample that accurately reflects the characteristics of users of the specific product under consideration.^27–29 The distribution of variables such as age, sex, literacy level (eg, education), and others within the participant sample should closely mirror the distribution observed among the actual users of the product in question. It is noteworthy, however, that the utilization of random sampling may not be necessary in all cases.²⁹ We conducted the recruitment between April and September 2022 using recruitment flyer distribution, social networking services, and a market research company.

Criteria for Participant Suitability

The designated number of participants per product is set at 10, given the execution of five cohorts, resulting in the recruitment of a total of 50 participants. To guarantee a comprehensive representation within the target group, the following criteria are established for the inclusion of participants.

(1) Age: Individuals aged between 30 and to 70 years, aligning with the age range during which FFC products are most commonly utilized.

(2) Sex: Each sex category must be represented by a minimum of four individuals.

(3) Literacy Level: High school and vocational school graduates or equivalents are to be included, ensuring diverse educational backgrounds within the target group.

(4) Occupation: Includes two or more people who do not regularly use written information as part of their occupation.

Exclusion Criteria for Participants

The following four points were set as criteria for the exclusion of participants.

(1) Individuals who are currently using or have used FFC products under investigation within the past 6 months.

(2) Individuals whose family members are using FFC.

(3) Individuals involved in health professions, pharmaceutical professions, occupations associated with health products, or those with prior work experience in these domains.

(4) Individuals who have been participants of a user testing within 6 months.

We considered a balanced distribution in terms of sex, age, and literacy level (education background), as described in Table 4.^27,28 We provided potential participants with written explanations outlining the purpose and methods of the test and obtained their written informed consent. No people refused to participate or dropped out of the user testing.

User-Testing Procedure

The user testing was conducted as follows:

(1) Preliminary preparation

i) Development of protocols

The user-testing procedures and methods were consolidated into a protocol. Specific questions were developed for products.

ii) Interviewers

The two interviewers underwent training to standardize their levels of observational and listening skills before engaging in user testing. They are pharmacists and university employees with Ph.D. They are qualified interviewers accredited by the Japanese Interviewer Association.

iii) Conducting a pilot test

The pilot user testing was conducted from October 1st, 2020, to January 31st, 2021, with three participants to assess the appropriateness of the testing procedure, the manner and wording of the questions, and response time settings. The protocol was then adjusted based on these findings.

(2) User testing

The user testing took place between April 1st, 2021, and December 30th, 2022. Written informed consent was obtained from all participants involved in the study.

i) Place and timing of the interview

A quiet room with adequate privacy was prepared for the participants to relax and be interviewed at our workplace. Each interview was scheduled to last approximately 1 h, including the time needed to explain the user testing procedure and obtain consent. The interviews were recorded with the participants’ consent.

ii) User-testing questions

We developed a dozen questions on labelling according to the characteristics of each of the five FFC labels. Among them, 10 common questions were selected that were considered important. Their order was arranged randomly rather than following that on the label. These questions were short and open-ended, as outlined in Table 5. Standardized questions were prepared addressing the appropriate and safe use of the five products. Finally, participants were asked to provide feedback on the comprehensibility, issues, design, and layout of the labels (Table 6).

Table 5 Questionnaire on the Content of Foods with Function Claims Labelling

Table 6 Questionnaire on Labelling Foods with Function Claims for Participants’ Comments

Outcome Measurement

The participants were asked to locate the relevant information, and the response time was recorded for each question across the five FFC products. In addition to providing answers, participants were asked to rephrase the information in their own words to assess their understanding of the materials. Two cut-off points, at 1 min and 2 min of response time, were used to evaluate participants’ understanding. The 1-min cut-off was established based on our previous user testing on drug information, which indicated that, on average, people need 1 min to understand 1000 characters of information correctly.^32,33 Given that FFC labelling contains ≤ 1000 characters, in theory, 1 min should suffice. The 2-min cut-off was also used based on the results from the pilot test, indicating that participants needed approximately 2 min to answer each FFC-related question. The product was considered to pass the test if 90% or more of the participants could find and correctly understand the information for all 10 questions before the specified cut-off time. If a participant could not find the answer within 2 min, their response time was recorded as 2 min. Descriptive statistics were summarized using the median and interquartile range (IQR). The relationship between respondents’ age and response time to each question was examined using Spearman’s rank correlation coefficient, and p<0.05 was judged to be statistically significant. This analysis was performed using SPSS Statistics Version 29.0.1.0 (IBM, Armonk, NY, USA).

Qualitative Analysis of Participants’ Comments

Using semi-structured interviews, participants were asked to respond to the questions following each user testing (Table 6). In our analysis, we incorporated elements of the KJ Method, a qualitative research strategy developed by Kawakita.^35,36 Qualitative analysis was conducted for each question to gain insights from participants’ responses.

Comparison Between the Current Version and the Revised Version of the Standardized Wording

Our analysis using the F-CCI indicated potential areas for improvement in the standardized wording included in the FFC label, originally developed by CAA. Some of the wordings were considered difficult to understand. Furthermore, the user testing conducted in this study showed that the current wording was difficult to understand and time-consuming. Therefore, we developed a revised version of the standardized wording to be used on the FFC label and compared it against the current version developed by CAA (Figure 4). To enhance the comprehensibility of container and packaging labels, a QR code can be added to the label. This code can direct the user to a page with clear and concise explanations. The terms “Submitted Claim” and “Individual Evaluations”, as well as the distinction between FFC and FOSHU, were explained.

Figure 4 Current standardized version by the CAA and its proposed revision in the label. Note: The copyright of the FOSHU approval seal belongs to the Consumer Affairs Agency, but prior permission is not required for its use. Available from the following website. https://www.caa.go.jp/policies/policy/food_labeling/foods_for_specified_health_uses.

Twenty participants who underwent user testing of products C and D assessed the current and revised versions on a 5-point scale (5 = very easy to understand, 4 = easy to understand, 3 = neither, 2 = difficult to understand, and 1 = very difficult to understand) across four questions (Q1: Size, legibility, and length; Q2: terms and sentences; Q3: Usefulness of the information; Q4: Overall evaluation). The Wilcoxon signed-rank test was performed to compare participant evaluations of the current and revised versions. This test was chosen because the data consisted of paired ordinal measurements collected on a 5-point Likert scale. Results are reported as medians with interquartile ranges (IQR), and statistical significance was set at p<0.05. This analysis was performed using SPSS Statistics Version 29.0.1.0 (IBM, Armonk, NY, USA).

Results

Accessibility and Understandability of the FFC Labelling in the User Testing

The median response times for each question for FFC labelling are shown in Table 7. When the 2-min cut-off was used, only one product (product B) among the five user-tested products met the threshold of 90% for all 10 questions (Table 8). However, the overall results were relatively positive. Products A and E achieved 90% or more correct responses for all questions but one (Q6). Product C missed the passing score for Q4 and Q5. Product D, which had the smallest font size on its labelling, showed the poorest performance among the five products, with two questions (Q1, Q2) not meeting the criterion. When participants successfully found an answer to a question, we confirmed their understanding of its content.

Table 7 Accessibility of the 10 Questions for the Five FFC Products in the User-Testing

Table 8 Proportion of Participants Who Correctly Identified the Answers for Each Question Within 2 min

When evaluated at 1 min, the overall performance was considerably poorer than that at 2 min, with no product meeting the 90% criterion for all questions (Table 9). Products A and B performed well, only missing the passing score for two questions (Q1, Q6). Product E failed to meet the passing score on four questions (Q2, Q6, Q7, Q8). Product D remained the worst-performing product, not reaching the 90% threshold for more than half of the questions.

Table 9 Proportion of Participants Who Correctly Identified the Answers for Each Question Within 1 min

Questions about diet, concomitant medications, illnesses, allergies, side effects, storage, and handling, which are essential for the safe use of FFC, were answered in the 70th percentile at 1 min.

A Spearman’s rank correlation coefficient was calculated to assess the relationship between participants’ age and the time taken to respond to each question. The analysis revealed a significant correlation between age and response time (r=0.177, p<0.001).

Qualitative Analysis of Participants’ Comments

For each question, comments were collected during the interviews, and similar comments were grouped (Table 10). Many participants had insufficient knowledge of FFC and perceived it as potentially beneficial for their health based on its image. Additionally, a significant number were unaware of the distinction between FFC and FOSHU, with 30% (15/50) of participants incorrectly believing that FFC was superior in effectiveness to FOSHU. The labelling included numerous technical terms such as ”Submitted Claim” and ‘Individual Evaluations.’ Some sentences posed challenges in comprehension and interpretation. For instance, it was difficult to discern the intended meaning of the statement, ‘This product is not a food developed for people suffering from diseases, minors, pregnant women (including those planning a pregnancy), or lactating women.’ As a result, the participants were unclear as to whether the relevant people were allowed to take it or not.

Table 10 Classification of Participants’ Perceptions and Comments on the Labelling of FFC

While the sales copy was easily understood because of its large font size and good design, the labelling itself presented difficulties in reading due to the small font size and the shape of its container or packaging.

Comparison Between the Current Version and the Revised Version of the Standardized Wording

In this user testing of FFC, the participants’ comments showed that the standardized wording developed by CAA (current version) is difficult to understand and is consistent with our previous findings. Therefore, we developed a revised version to enhance the original model created by CAA and conducted additional user testing for comparison (Figure 4). Due to limited space on the packaging, a QR code is provided to access more detailed information, including terminology, which can be obtained by scanning the QR code. Twenty participants were asked to compare them on a five-point scale for the revised labelling. The results of the evaluation are shown in Table 11. The revised version received higher ratings than the current version across all four elements (size, legibility, and length; terms and sentences; usefulness of the information; and overall evaluation). The differences were all statistically significant.

Table 11 Comparison Between the Current Standardized Version by the CAA and Its Proposed Revision in the Label

Discussion

To date, there has been a lack of studies investigating health food labelling from both HCPs and user perspectives, even in international contexts. In this study, we conducted user testing for five FFCs, which we had previously evaluated using the F-CCI. The results of this user testing showed that, in many cases, FFC labels provide inadequate explanations and are difficult to understand. In particular, the correct response rate was notably low for bottle labels, partly due to the small text and the round shape, which made the labels difficult to read. Additionally, the correlation between age and correct response rate was very weak, suggesting that the readability issue affected participants regardless of age.

In consumer research on food labelling, qualitative research should be conducted from the consumer’s point of view, using a range of approaches, including observation and semi-directive interviews in addition to questionnaire surveys.³⁷ This study marks the first user testing of FFC labelling, complemented by qualitative research, providing novel insights into the evaluation of FFC labelling. When assessing the labelling of FFC, it is crucial to approach the evaluation from the perspectives of both the HCPs and the end-users.

We previously evaluated FFC labelling from the HCPs’ perspective.¹³ None of the five products met the acceptance criteria for the F-CCI questions. Certain sentences indicated by the CAA included technical jargon that was not commonly used by the public.

Important information was often not immediately accessible because it was not summarized in the first section or was scattered throughout the label. Based on our previous user testing on drug-related information and considering the word count of the FFC label, we initially expected that 1 min would suffice for consumers to capture and comprehend the information accurately. However, the testing revealed consumers needed 2 min, twice as long as for drug-related information. One factor contributing to the longer time required for response is that the order of entry, position of entry, and font size in the FFC label are not specified, unlike those in drug labels. This underscores a clear need for improvements in FFC labelling to enhance information accessibility and consumer understanding. In light of the recent incident in Japan, it is particularly important to more clearly note the precautions for safe use. Furthermore, it should be emphasized that physical health care is crucial in self-care.

However, during the interviews, it became evident that the participants not only lacked sufficient knowledge about FFC but also held misconceptions. This could pose a fundamental challenge to the proper use of FFC. Taking into account consumers’ health literacy levels, information providers should offer easy-to-understand materials.

The evaluations of the two surveys yielded largely consistent outcomes, highlighting the need for improved labelling to enhance consumer safety and product usage. Overall, there is often a focus on product promotion through design and sales copy rather than facilitating consumer access to important information and comprehension of messages. In particular, precautions for safe use must be presented in a manner that is more easily understandable for consumers.

This study has certain limitations. First, we only evaluated five FFCs. In selecting the five products, we considered more than 100 FFC labelling cases and found a similar trend in labelling content, with the wording portion recommended by the CAA accounting for about half of the labelling cases. Nevertheless, future research should explore a broader spectrum of FFCs. Second, the representativeness of the sample should also be mentioned. The age and sex distributions of the participants closely aligned to those of the general population in Japan according to the national census of 2020.³⁸ However, educational attainment was skewed, with 100% of respondents having completed high school, compared to 85.9% of the total population. Furthermore, given the voluntary nature of participation, it is possible that individuals with a higher interest in functional foods or health-related topics would be more likely to participate, which may have introduced self-selection bias. Despite these limitations, valuable insights into consumer perceptions were garnered through live feedback from 50 interviewees.

We believe that the standardized language used in food labelling requires improvement as it currently contains numerous technical terms that pose difficulties for consumers to comprehend. In the development of health information materials, such as FFC, the newly established system facilitates the creation of optimal materials. This is achieved by enabling HCPs to assess and enhance the materials using a communication index as a specified indicator, followed by a validation process to ascertain their effectiveness and assess consumer comprehension. In the future, we intend to promote a website that we have developed to evaluate the usefulness of health-related information materials.³⁴

Conclusion

In this study, we undertook the first user testing of FFC labelling in Japan to ascertain users’ perceptions and comprehension. The results suggested that consumers encountered challenges in locating and understanding information within the current FFC labelling. The evaluation of the user testing underscores the need to improve the presentation of key information to ensure the safe and appropriate use of FFC, given that most consumers are not familiar with FFC.

A combined evaluation of HCPs’ and users’ perspectives identified common challenges in the safe use of FFCs. A critical evaluation is imperative from both viewpoints in the development and application of FFC labelling for effective risk-benefit communication. Establishing an integrated method for assessing usefulness becomes paramount in this context.

This initiative is significant as it has the potential to significantly contribute to consumer decision-making and the secure utilization of health food products, including FFC.