Associations of Vocal Features, Psychiatric Symptoms, and Cognitive Functions in Schizophrenia

Introduction

Schizophrenia is a complex and heterogeneous psychiatric disorder that affects millions of people worldwide, presenting significant challenges in both diagnosis and treatment.¹ Traditionally, schizophrenia has been viewed as a uniform condition; however, recent studies have uncovered significant variability in symptom presentation and individual responses to treatment.² This disorder is typically characterized by positive symptoms, such as delusions, hallucinations, and disorganized thinking, as well as negative symptoms, including reduced emotional expression, social withdrawal, and diminished speech output.³

Patients with schizophrenia are also known to exhibit widespread cognitive impairments.^4,5 Meta-analyses indicate that adult patients with schizophrenia experience a decline in IQ during the first episode, and this decline becomes more pronounced in chronic cases.⁶ Cognitive impairments, including deficits in attention, working memory, executive function, verbal memory, and processing speed are also common and contribute to the complexity of the clinical picture.^7,8 These cognitive impairments substantially contribute to poor clinical outcomes, such as unemployment and difficulty with independent living.⁹ In conjunction with positive and negative symptoms, these cognitive impairments significantly affect daily functioning and diminish quality of life.^10,11 Delayed treatment or prolonged course of the disorder can further exacerbate deterioration in executive function¹² and social functioning,¹³ highlighting the importance of early diagnosis, comprehensive psychiatric assessment, and timely intervention.

Effective management of schizophrenia requires continuous, multidimensional assessment of symptom severity and progression. Clinical assessment tools, such as the Positive and Negative Syndrome Scale (PANSS), are routinely used to evaluate various dimensions of symptomatology.¹⁴ Although PANSS and similar scales are useful, they may not fully capture the entire spectrum of schizophrenia’s heterogeneity, underscoring the need for a comprehensive approach that integrates psychiatric symptoms, cognitive functions, social supports, and comorbid conditions.¹⁰

In recent years, interest in utilizing advanced computational techniques to analyze vocal features in patients with schizophrenia has grown.¹⁵ Speech disturbances, including alterations in prosody, syntax, and semantics, are frequently observed in patients with schizophrenia and serve as valuable indicators for diagnosing and monitoring the progression of the disorder.^16,17 These disturbances are often associated with both positive and negative symptoms of schizophrenia, such as delusions, disorganized thinking, and emotional blunting.¹⁸ Vocal analysis provides insights that enhance the understanding of psychiatric symptoms and cognitive functions, revealing patterns that traditional clinical assessments may overlook.¹⁹ Vocal characteristics have been used to identify subtypes within psychotic disorders, with different vocal patterns linked to variations in symptom severity and cognitive functions.²⁰ A study evaluating the feasibility of collecting speech samples from patients with psychiatric disorders demonstrated that vocal features could serve as a useful tool for tracking clinical states over time, with individually trained models demonstrating strong correlations with clinician ratings.²¹ Given the complexity and heterogeneity of this disorder, vocal analysis has gained considerable attention as a potential tool to enhance clinical assessments and provide a more objective measure of symptom severity.²²

Among various computational techniques, vocal analysis using digital signal processing methods has emerged as one of the most promising approaches.²³ These methods allow for the quantification of acoustic features that may be associated with specific symptoms of schizophrenia.²⁴ Notably, techniques such as chromagram, mel spectrogram, and mel-frequency cepstral coefficients (MFCC) are being increasingly utilized for this purpose.^23,25 A chromagram provides a comprehensive representation of vocal signals in the time-frequency domain, capturing variations in pitch, intensity, and rhythm over time.²⁶ This approach has been employed in several studies to quantify prosodic changes in the vocal features of patients with schizophrenia.¹⁶ Similarly, mel spectrogram represents the frequency spectrum of an audio signal over time with a frequency scale warped to match the human auditory system’s response, highlighting subtle disruptions in vocal patterns, particularly regarding phonemic articulation.¹⁶ MFCC values are derived from mel spectrogram using a discrete cosine transform. They are widely used in both speech recognition and classification tasks, as they capture the spectral and temporal properties of vocal features, providing a compact representation of these characteristics.²⁷ The process of generating MFCC involves several steps. First, the input audio signal is transformed into a spectrogram, and mel-frequency filters are applied. These filters resample the frequency axis to mimic the logarithmic pitch sensitivity of human hearing, offering finer resolution at lower frequencies and coarser resolution at higher frequencies. The resulting mel-filtered data are then log-transformed and processed through a cepstral transform, which applies a series of cosine terms to the spectra. This transformation has proven valuable in reducing the dimensionality of the voice data while retaining essential acoustic features.²⁸

Voice analysis has emerged as a valuable tool for assessing neurological and psychiatric disorders, providing insights into clinical symptoms through acoustic features. A recent systematic review highlighted the increasing role of voice analysis in neurological and psychiatric disorders, including schizophrenia, with MFCC, jitter, shimmer, and fundamental frequency (F0) being the most commonly used speech features.²⁹ Similarly, another systematic review on speech recognition emphasized the widespread use of deep neural networks and identified MFCC as the most commonly used speech feature.³⁰ Some studies have adopted the average values of vocal features, including MFCC, to capture broader speaker characteristics and mitigate the influence of short-term variations in speech.³¹ Previous studies have demonstrated that alterations in MFCC values correlate with symptom severity and may offer a non-invasive, real-time method to track disease progression.²⁴

Convergent progress in speech research and computer sciences opens avenues for implementing speech analysis to enhance the objectivity of assessment in clinical practice.³² Machine learning approaches have been widely applied to classify speech patterns in schizophrenia and other psychiatric disorders, demonstrating their potential for improving the assessment of schizophrenia-related cognitive and psychiatric impairments.²⁹ Recent research has underscored the potential of these advanced vocal analysis techniques not only for assessing symptom severity but also for predicting treatment outcomes and relapse in schizophrenia.³³ This advancement would allow clinicians to incorporate more objective, quantitative tools to complement existing diagnostic methods such as clinical interviews and symptom rating scales.³⁴

We hypothesized that vocal analysis could serve as a promising approach for evaluating psychiatric symptoms and cognitive functions in patients with schizophrenia. As an assessment tool, vocal analysis could enhance objective symptom evaluation, particularly given the findings that indicate distinctions in vocal characteristics between schizophrenia and other psychiatric disorders.³⁵ However, to date, no study has reported on the utility of relatively new machine learning-based vocal features, such as the chromagram, mel spectrogram, and MFCC, for evaluating the psychiatric symptoms of schizophrenia in real-world clinical settings.

Therefore, we conducted a comprehensive correlational analysis of vocal features, psychiatric symptoms, and cognitive functions in patients with schizophrenia to enhance both understanding and management of the disorder. Our study uses a range of acoustic features to examine their associations with psychiatric symptoms and cognitive functions. This approach offers a broader perspective compared to previous studies by exploring multiple features to gain a better understanding of the relationship between vocal characteristics and schizophrenia.

Methods

This study was conducted by the University of Occupational and Environmental Health and the Kyushu Institute of Technology in accordance with the tenets of the Declaration of Helsinki (1975, revised in 2008) and the Ethical Guidelines for Medical and Health Research Involving Human Participants in Japan. All procedures involving human participants were approved by the Ethics Committee of the University of Occupational and Environmental Health Sciences (approval number UOEHCRB21-090).

Participants

Patients were recruited from the outpatient clinic of UOEH Hospital. Written informed consent was obtained from all participants, who were assigned anonymous identification numbers to ensure privacy. An experienced psychiatrist (T.N.) diagnosed schizophrenia based on a clinical interview following the criteria outlined in the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5). Participants were selected based on the following inclusion criteria: (1) diagnosed with schizophrenia according to DSM-5, (2) aged 20 years or older but younger than 65 years at the time of participation, (3) receiving outpatient treatment, (4) able to provide informed consent for study participation. Exclusion criteria included: (1) individuals who are not native Japanese speakers or do not have sufficient proficiency in Japanese to complete the assessments, (2) individuals with severe neurological disorders (eg, Parkinson’s disease, stroke, epilepsy), and (3) individuals with hearing impairments or speech disorders that could affect vocal analysis. The study initially included two male and 14 female participants; however, one female participant experienced an exacerbation of psychiatric symptoms between the date of consent and the cognitive assessment, hindering her ability to comprehend the test. Another male participant could not undergo the cognitive assessment because of difficulties in understanding the contents of the test. Consequently, 14 participants (1 male and 13 females) were included in the analysis.

Audio Measurement and Processing

The development of an automated questioning system and voice samples acquisition were carried out by a team from the Kyushu Institute of Technology. An automated questioning system was designed to collect voice samples under standardized conditions, minimizing factors that could influence responses, such as the relationship between the questioner and the participant, or the distance between the participant and the microphone array. This system not only presented questions but also recorded the participants’ spoken responses.

Participants were instructed to read 30 questions from the Japanese version of the Schizophrenia Quality of Life Scale to familiarize themselves with the task. The automated questioning system then presented these questions both audibly and visually, and participants responded with one of five possible answers: “Always”, “Often”, “Sometimes”, “Rarely”, or “Never”. The system recorded the participants’ spoken responses, captured by the microphone array positioned between the automated questioning system and the participant (see Figure 1), with a sampling frequency of 48 kHz. All voice samples were collected in a quiet, private room located in an area with no foot traffic, ensuring that external noise did not interfere with recordings. To further control environmental noise, the microphone and participant were kept at a fixed distance throughout the recordings, and noise reduction was applied during data processing to remove background interference and enhance recording clarity.

Figure 1 Preliminary experiment setup. The participants in (see Figure 1 have given written consent to be included in the Figure.

Voice samples for the five response categories (for example, “Never”) were manually extracted using Audacity^® 3.2.5 (https://support.audacityteam.org/additional-resources/changelog/older-versions/audacity-3.2/audacity-3.2.5), a voice editing software. This process facilitated the extraction of voice data for the aforementioned five types of responses. The vocal features, including 12-dimensional chromagram, 128-dimensional mel spectrogram, and 40-dimensional MFCC values, were then extracted using the Librosa audio library (https://librosa.org/doc/latest/index.html). The average values of these extracted features were then calculated for analysis.

Assessment of Psychiatric Symptoms and Cognitive Functions

On the same day as the audio measurements, an experienced psychiatrist (T.N.) assessed psychiatric symptoms using the Positive and Negative Syndrome Scale (PANSS). The positive subscale assesses symptoms such as delusions, hallucinations, and conceptual disorganization, reflecting the severity of positive symptoms. The negative subscale evaluates symptoms such as blunted affect, emotional withdrawal, and apathy, representing negative symptoms. The general psychopathological subscale examines symptoms such as anxiety, depression, and cognitive disturbances that do not fall under positive or negative categories. The total PANSS score is the sum of all item scores, providing an overall measure of symptom severity. Each item is rated on a scale from 1 to 7, with higher scores indicating more severe symptoms.

Cognitive functions were assessed using a computer-based tool, Cognitrax (CNS Vital Signs, Morrisville, NC, USA).³⁶ The test battery assesses several cognitive domains, including verbal memory, executive function, social cognition, working memory, sustained attention, and processing speed. Verbal memory was evaluated using the “Verbal Memory Test”, executive function using the “Shifting Attention Test”, social cognition using the “Perception of Emotions Test”, and working memory using the “4-Part Continuous Performance Test”. Higher scores generally indicate better cognitive performance; there are no standardized normal ranges for Cognitrax scores.

Statistical Analysis

All statistical analyses were conducted using SPSS software (IBM SPSS Inc., Armonk, NY, USA) and Python 3.0 (https://www.python.org/download/releases/3.0/). Descriptive statistics for demographic data are presented as means (standard deviation) or medians [interquartile range]. Multiple correlation analyses were performed to investigate the relationship between vocal features (chromagram, mel spectrogram, and MFCC) and psychiatric symptoms or cognitive functions, with age and sex included as covariates.

Acoustic features were extracted in small windows of audio data, resulting in multiple values for a given recording. To facilitate data analysis, we computed the average values of each vocal feature across all windows. For instance, the 40 dimensions of MFCC values were averaged across all windows to obtain a single representative value for each dimension.

Finally, correlation analyses were conducted with psychiatric symptoms as the independent variable and cognitive functions as the dependent variable, controlling for age and sex as covariates. P-values less than 0.05 were considered statistically significant.

Given the gender imbalance (92.9% female participants, n = 13), we conducted a secondary analysis using only the female participants to assess whether sex influenced the results. This analysis was executed using the same methods and vocal features as in the main analysis, with age included as a covariate in all statistical analyses.

Results

Demographic and Clinical Characteristics

Table 1 details the demographic and clinical characteristics of the participants (n = 14). The average age of the participants was 37.36 ± 12.08 years, with a female predominance (92.9%). The average age of onset of schizophrenia was 23 ± 7.05 years, with an average disease duration of 14.36 ± 7.12 years. Chlorpromazine equivalents for medication dosages ranged from 378.75 to 575.00 mg. The PANSS evaluation indicated mild to moderate severity of schizophrenia, with the following scores: PANSS positive scores (median: 15.0, IQR: 14.0–15.0), PANSS negative scores (median: 14.0, IQR: 14.0–14.75), PANSS general psychopathological scores (median: 32.0, IQR: 32.0–33.0), and PANSS total scores (median: 62.0, IQR: 61.0–63.0). Cognitive assessment scores varied across domains: verbal memory (median: 53.50, IQR: 50.50–57.50), executive function (median: 37.50, IQR: 27.25–44.00), social cognition (median: 7.00, IQR: 3.50–8.00), working memory (median: 11.00, IQR: 6.50–13.75), sustained attention (median: 32.00, IQR: 28.50–35.75), and processing speed (median: 105.50, IQR: 101.25–119.00).

Table 1 Demographic and Clinical Data of Study Participants

Associations of Vocal Features with Psychiatric Symptoms and Cognitive Functions

In the correlation analysis, significant negative correlations were observed between vocal features and cognitive functions, whereas no such correlations were found between vocal features and psychiatric symptoms. Specifically, significant negative correlations were found between working memory and both mel spectrogram (β = −0.645, p = 0.023) and MFCC (β = −0.669, p = 0.017). Furthermore, negative correlations were identified between sustained attention and mel spectrogram (β = −0.626, p = 0.029), as well as between sustained attention and MFCC (β = −0.578, p = 0.049), as shown in Table 2. However, no statistically significant correlations were found between any of the vocal features and psychiatric symptoms (PANSS scores). Significant negative correlations of the PANSS total score with both working memory (β = −0.604, p = 0.037) and sustained attention (β = −0.650, p = 0.022) were found. Additionally, sustained attention was negatively correlated with the PANSS positive score (β = −0.676, p = 0.016) (Figures 2 and 3).

Table 2 Correlation of Psychiatric Symptoms and Vocal Features with Cognitive Functions

Figure 2 Chord diagram of correlations between voice features, psychiatric symptoms, and cognitive functions.

Figure 3 Pearson correlations between vocal features, psychiatric symptoms, and cognitive functions, with bubble size indicating cognitive scores and color representing symptom scores.

A secondary analysis, conducted using only the female participants (n = 13), showed results consistent with those of the entire sample. Specifically, the significant negative correlations between vocal features (mel spectrogram and MFCC) and cognitive functions, such as working memory and sustained attention, remained unchanged. Similarly, the relationship between cognitive functions and psychiatric symptoms remained consistent in this subgroup.

Discussion

This study examined the relationships between psychiatric symptoms, cognitive functions, and vocal features in patients with schizophrenia. We found significant correlations between vocal features and cognitive functions in the 14 participants. Notably, negative correlations were observed between certain cognitive functions—specifically working memory and sustained attention—and vocal features extracted from mel spectrogram and MFCC analyses. Although no direct correlations between psychiatric symptoms and vocal features were identified, cognitive functions showed negative correlations with the severity of psychiatric symptoms, including both the PANSS total scores and PANSS positive scores.

Cognitive impairment—particularly deficits in attention, memory, and executive function—is often more debilitating than positive and negative symptoms concerning their impact on daily functioning.³⁷ Identifying objective vocal features that correlate with cognitive functions holds considerable promise for both clinical practice and research in schizophrenia.³⁸ In our study, the negative correlations between vocal features (mel spectrogram and MFCC) and cognitive functions (working memory and sustained attention) emphasize the potential of vocal features as markers of cognitive impairment in schizophrenia. Previous studies have suggested that abnormalities in specific neural oscillations affect language function. In schizophrenia, beta oscillations are particularly impaired and are closely associated with inhibitory control in the brain. Beta oscillations prevent irrelevant information from interfering with language processing. During working memory maintenance, increased beta activity is thought to protect stored information from external distractions. However, in patients with schizophrenia, the reduction in beta oscillations weakens this inhibitory function, leading to excessive activation of irrelevant information. Moreover, beta oscillations also play a role in motor control, including the coordination of speech production.³⁹ The observed associations between cognitive functions and vocal features in this study may be attributable to beta oscillation abnormalities.

There is a growing need for objective and reliable methods to assess cognitive impairment in schizophrenia. Traditional cognitive assessments typically rely on performance-based tasks and subjective evaluations, which can be influenced by factors such as patient motivation and examiner bias.⁴⁰ Therefore, objective vocal analysis methods, such as those based on mel spectrogram and MFCC, could complement existing cognitive assessments, providing clinicians with additional non-invasive tools for evaluating cognitive impairment. However, larger-scale studies are necessary to determine whether these objective vocal features can facilitate early detection of cognitive decline and enable more targeted interventions aimed at improving cognitive functions in schizophrenia.

Our study also identified significant negative correlations between cognitive functions and PANSS scores, suggesting that higher levels of psychiatric symptoms are associated with poorer cognitive performance across multiple domains. Specifically, sustained attention was negatively correlated with both PANSS total scores and positive scores, while working memory showed a negative correlation with PANSS total scores. These results align with prior research linking cognitive impairments to traditional psychiatric symptoms, including positive, negative, and general psychopathological symptoms.^41,42 However, previous studies have reported that the disorganization factor in the PANSS—which includes attention deficits, disorientation, impaired conceptualization, and difficulties with abstract thinking—shows the strongest association with cognitive scale scores.^41,42 Moreover, recent studies using network analyses have found no clear correlation between positive and negative symptoms and cognitive scale scores, but have shown associations between cognitive deficits and general psychopathological symptoms.^43,44 These discrepancies highlight the complexity of the relationship between cognitive impairment and psychiatric symptoms in schizophrenia, which may differ across patients.

Despite the absence of direct correlations between psychiatric symptoms and vocal features, our results suggest that cognitive functions may mediate the relationship between psychiatric symptoms and vocal characteristics. This finding contrasts with previous studies that have identified associations between certain psychiatric symptoms, such as disorganization in thought or behavior, and distinct vocal characteristics.^45,46 However, the lack of significant direct correlations does not rule out a potential relationship between psychiatric symptoms and vocal features. Factors such as sample size, symptom heterogeneity, and variability in vocal patterns may have influenced these results. Future studies with larger and more diverse samples are needed to clarify the relationship between psychiatric symptoms and vocal features.

Although cognitive impairments in schizophrenia are likely to contribute to vocal processing differences detected through acoustic analysis, our study has several limitations. First, the small sample size and the lack of a control group reduced the robustness of our findings. With only 14 participants, the statistical power of our analysis was limited, and some relationships may not have been detected due to insufficient sample size. A power analysis revealed a statistical power of 24.71%, which is below the recommended threshold of 80%. Thus, the generalizability of our findings is limited, necessitating future studies with larger and more diverse samples to validate our results and improve statistical robustness. Additionally, the psychiatric symptoms among our sample were relatively mild, as only outpatients were included. This may have introduced sampling bias, limiting generalizability. Furthermore, participant exposure to antipsychotic treatment and the lack of control for educational background may have influenced both vocal patterns and cognitive outcomes.

Second, sex differences in vocal features present a limitation. Males and females exhibit differences in vocal tract length, affecting vocal characteristics such as fundamental frequencies and MFCC. Our sample had a significant sex imbalance, with 92.9% of participants being female, which may have influenced the results. The predominance of female participants could have biased the acoustic measures, limiting the applicability of our findings to the broader schizophrenia population. Future studies should consider sex as a factor in vocal analysis to better understand its impact on the relationship between vocal features, cognitive functions, and psychiatric symptoms in schizophrenia. Ensuring a more balanced sex distribution in future research will be essential to assess whether sex-specific differences affect the observed associations.

Third, our study was limited by the methodology that utilized only single-word responses and short utterances of approximately 1 second. Although this approach was necessary to standardize experimental conditions, it does not capture the full complexity of natural speech. Moreover, it remains unclear whether our findings are applicable to languages other than Japanese. Future research should address these limitations by incorporating multi-word responses, larger and more diverse samples, control groups, and longitudinal designs to track changes in vocal features over time. Additionally, our study primarily focused on acoustic characteristics and did not take into account syntactic and semantic aspects of speech, which could further enhance our understanding of the relationship between vocal features and cognitive and psychiatric functions in schizophrenia.

Finally, recent research highlights the complexity of speech as both a biosocial marker and a neurobiological indicator of psychiatric symptoms. Speech is influenced not only by biological factors but also by social determinants such as socioeconomic background and interpersonal interactions.⁴⁷ Additionally, a study has shown that PANSS Disorganization (PANSS-P2) is more closely associated with linguistic style, particularly lower analytic thinking scores, than general cognitive impairments.⁴⁸ Another finding links conceptual disorganization to altered effective connectivity between the anterior cingulate cortex and anterior insula, highlighting its neurobiological basis.⁴⁹ However, our study did not separately examine the PANSS Disorganization factor, which may provide further insight into the relationship between vocal features and clinical symptoms. Moreover, our analysis was limited to a structured reading task rather than naturalistic or spontaneous speech, which may restrict the generalizability of our findings. Future research should incorporate spontaneous speech analysis to better capture the full scope of linguistic disturbances in schizophrenia and further explore the interplay between speech characteristics, cognitive functions, and psychiatric symptoms.

Conclusion

This study highlights the potential interplay between vocal features, psychiatric symptoms, and cognitive functions in patients with schizophrenia. By demonstrating correlations between vocal features and cognitive impairments, this study contributes to the growing body of research exploring vocal markers as potential indicators of cognitive impairments in schizophrenia. These results highlight the potential of vocal analysis in detecting early signs of cognitive impairments and psychiatric symptoms, which could lead to more timely and targeted interventions. Future research should expand the scope of vocal analysis to include a broader range of linguistic features such as prosody, syntax, and semantics. Larger, longitudinal studies with control groups are needed to further validate these findings and refine clinical applications, improving diagnostic accuracy and intervention outcomes for patients with schizophrenia.