Validity and reliability of the Turkish version of Athlete Psychological Strain Questionnaire (APSQ)

Abstract

Objective: The aim of this study was to test the validity and reliability of the Turkish version of the Athlete Psychological Strain Questionnaire (APSQ).

Material and Method: The APSQ, which consists of 10 items and 3 sub-dimensions (self-regulation difficulty, performance concerns, and externalized coping) and assesses athlete-specific psychological distress, was administered to a total of 565 elite athletes (247 females, and 318 males). Confirmatory factor analysis (CFA) was used to assess construct validity, and Cronbach alpha reliability analysis was used to determine internal consistency. Concurrent validity of the APSQ with the Depression Anxiety and Stress Scales-21 and the Kessler Psychological Stress Scale-10 was also tested.

Results: Findings for construct validity were consistent with three-factor structure of the original questionnaire (RMSEA=0.05, SRMR=0.03, NFI=0.94, CFI=0.96, GFI=0.97). The Cronbach’s α coefficient of the whole scale was 0.83, and the Cronbach’s α coefficients of three sub-dimensions were 0.75, 0.69, and 0.50 separately.

Conclusion: Findings demonstrate that the Turkish adaptation of the APSQ has sufficient psychometric properties for widespread use, assessing key dimensions of psychological strain experienced by elite athletes.

Introduction

There are many factors affecting athletes’ health and athletic performance (1), most of which are environmental and physical. However, mental health (MH) is also a key domain of elite athlete wellbeing. It has been emphasized that psychological factors can predict, prevent and aid recovery of sports injuries (2), and can affect not only the athletes’ health, but also the performance outcomes (3).

Elite athletes experience MH problems at least as much as the general population (4). In a recent meta-analysis, it was stated that 15% to 35% of elite athletes report various MH symptoms such as anxiety, depression, and alcohol misuse (5). When factors affecting the MH of athletes are examined, it can be seen that athletes are exposed to various sports-related stressors beyond those experienced by the general population, such as early career termination, serious injury, performance impairments, social media abuse, and concerns about sponsorship (6). The MH of athletes has also been affected by the coronavirus outbreak (7, 8), and associated impacts on competition. However, until recently, screening tools used for the general population were also used to screen the psychological states of athletes. Although some of these tools such as DASS-21 (9) have been reported to have adequate psychometric properties for athletes, there has been a lack of well-validated athlete-specific screening tools that can facilitate early detection of elite athletes experiencing psychological distress.

In this context, Rice et al. developed and validated the multidimensional Athlete Psychological Strain Questionnaire (APSQ) which was evaluated the sport-specific psychological strain in a sample of >1,000 elite athletes (10). The APSQ is the recommended trainee screening tool within the International Olympic Committee’s (IOC) Sport Mental Health Assessment Tool-1 to aid early detection of athletes' MH problems (4). While there are a number of non-English translations and validation studies of the APSQ has just been published (11, 12), the validity, and reliability of the Turkish APSQ is yet to be determined. In order to address this, the present study reports on the structural validity, reliability, convergent validity (concurrent, correlations) of the APSQ. Test-retest correlations were also performed to measure the temporal stability of APSQ scores.

Material and Methods

Participants and data collection

This study was approved by the ethical committee of the Faculty of Medicine of Ege University (E-43746 Date: 14.02.2021). In this study, the Turkish version of the APSQ was applied to the athletes by purposeful sampling in order to evaluate face validity (e.g., exploring if the questions we understandable, level of difficulty answering the questionnaire, and general opinions on the measure). Data obtained from this phase supported use of the questionnaire to the main study. According to Statistical power analysis (G-Power), the smallest sample size which reflects all athletes in sport disciplines included in the study was 370. Data collection was terminated when the recruitment target was achieved. A total of 565 elite athletes (318 males and, 247 females) from 21 different sporting disciplines participated in this study. Inclusion criteria were i) being aged between 18-38 years, ii) continue to participate in sports throughout the 2020-2021 season, iii) participating in sports discipline in the top league or national team level (e.g., Turkish Football Super League, Women's National Team, Badminton National Team), and iv) speaking Turkish as native language. Questionnaires were sent to the smartphones of athletes by sharing the questionnaire link with sports clubs, managers, and coaches. Informed consent was obtained from all athletes who volunteered to participate in this study. Confidentiality of all athlete data was ensured.

Measures

Athlete Psychological Strain Questionnaire (APSQ)

The APSQ consists of 10 items to measure sport-specific psychological strain (10). Each item is scored by a 5-point Likert-type (ranging from 1=none of the time to 5=all of the time) assessing self-regulation (e.g., “I was less motivated”), performance (e.g., “I found training more stressful”), and external coping (e.g., “I took unusual risks off-field”). While the minimum score is 10, maximum is 50. Rice et al. reported cut-off scores of APSQ for moderate ≥15, high ≥17, and very high ≥20 (13). Higher scores indicate greater psychological strain, and the factorial, convergent, and divergent validity of the APSQ’s has been established (10, 13).

Depression, Anxiety and Stress Scales (DASS-21)

The DASS-21 consists of 21 items, three sub-scales (depression, anxiety, and stress) and is scored by 4-point Likert-type scale (ranging from 0=never to 3=always) (14). By summing the scores of the items obtained from the sub-scales, the total score of each sub-scale was calculated. The higher scores indicate higher levels of either depression, anxiety, and stress that are defined by subscale. Vaughan et al. stated that the DASS-21 had sufficient psychometric properties in elite athletes (9). For depression, anxiety, and stress sub-scales, Cronbach’s alpha internal consistency coefficients were 0.89, 0.87, and 0.90, and test–retest coefficients were 0.93, 0.83, and 0.82, respectively (15). For this study, Cronbach’s alpha internal consistency coefficients for depression, anxiety, and stress sub-scales were calculated as 0.90, 0.85, and 0.86, respectively.

Kessler Psychological Distress Scale (K-10)

The K-10 consists of 10 questions that aims to measure the level of non-specific distress symptoms experienced within the last four weeks and currently (16). Each item scored by a 5-point Likert-type (ranging from 1= never to 5 = always). The minimum possible score is 10, while the maximum score is 50. Higher scores indicate higher levels of psychological distress. The internal consistency coefficient of the scale was found as 0.95 (17). For the present study, Cronbach’s alpha internal consistency coefficients for K-10 scale were calculated as 0.93.

Translation Phase and Procedure

The standard translation-back translation method suggested by World Health Organisation was used in the translation process of the scale (18). The questionnaire was translated from English to Turkish and then back into English. The English version of the questionnaire was translated into Turkish by one expert in both languages and three experts in the field of sports who speak English well. The items in the questionnaire were compared with each other, and the items with the same translation were determined. The items with the same and different translations were given to the individuals and translated back into English. The back-translated questionnaire items were compared with the original questionnaire items, and differences and errors were determined. By comparing the English translation with the original questionnaire, the final version of the Turkish questionnaire was created with the closest translations to each other. The final version of the questionnaire was shared with the scale developer, and it was confirmed that there was no loss of meaning in the items.

Since sufficient athletes who have good knowledge of both Turkish and English language and both cultures could not be reached, in order to determine the content validity index (CVI) (19), the "consulting an expert" or "expert opinion" method was used. Both the English and Turkish versions of the APSQ were submitted to 3 instructors (experts) working in the field of sports sciences to determine both language and content validity. Experts evaluated the questionnaire items according to their relevance (not suitable = 1 point, slightly suitable = 2 points, suitable but slight changes require = 3 points, very suitable = 4 points). Experts' assessments were evaluated using the content validity rate Davis technique (20). In the calculation of CVI, the number of experts evaluating the items as "suitable" and "very suitable" was divided by the total number of experts, and the CVI for each item was calculated (20). It is considered appropriate if the calculated CVI value is higher than 0.80. It was determined that the CVI value of the questionnaire items was 0.90 and, considering that all items in the questionnaire was appropriate, no item was removed from the questionnaire.

Data Analysis

Descriptive statistical methods (frequency, percentage, mean, standard deviation) were used. Reliability analysis was performed to test the reliability of the scales, and confirmatory factor analysis (CFA) to test the construct validity. CFA is a technique based on theoretical perspectives and is a validation method used in the adaptation of measurement tools developed in other cultures (21). Researchers propose CFA to test hypotheses about predetermined or constructed factor structures (22). It is common that modifications are required when testing CFA models. In CFA, the following indices were examined; Satorra and Bentler scaled chi-square value (CMIN) (23), root mean square error approach (RMSEA) (24), standardized root mean square error (SRMR), comparative fit index (CFI) (25), goodness of fit index (GFI), adjusted goodness of fit index (AGFI) (26), normed fit index (NFI) (27), un-normed fit index (NNFI) (28), incremental fit index (IFI) (29). In fit indices, 0 - 0.05 perfect fit of RMSEA and SRMR values, 0.05 - 0.10 acceptable fit; CFI value 0.97 - 1.00 perfect fit, 0.95 - 0.97 acceptable fit; the 0.95 - 1.00 range of NFI and GFI values indicates a perfect fit, and 0.90 - 0.95 an acceptable fit (30).

Concurrent validity is based on the principle of examining the relationship between the total scores obtained from the measurement tool and the total scores obtained from another measurement tool that measures the same feature or features that have been previously developed and proven to be valid (31). Concurrent validity and predictive validity were evaluated to determine the criterion-related validity of the study. Pearson correlation analysis was conducted to examine the relationships between the sub-dimensions of the APSQ and sub-dimensions of the DASS-21 and K-10 to test the criterion-related/concurrent validity of the APSQ. These correlation test results enabled that evaluation of DASS-21 and K-10 as predictors of the APSQ. Multiple regression analysis was used to determine the predictive degree of DASS-21 and K-10. The CFA was calculated to obtain information about the reliability of the questionnaire. The parametric tests were used to compare the APSQ and sub-dimensions according to variables (gender, age, etc.). The independent groups T-test was applied to the data set for the binary group comparisons, and one-way variance analysis used for groups of three and more.  G-Power 3.1.9.7 program was used to calculate the effect sizes (Cohen’s d) and power according to the variables (G-Power, 2017). Clinical differences and relationship between genders were also evaluated by chi-square test according to the APSQ cut-off values (≥15=Moderate, ≥17=High, ≥20=Very high) (13). A 95% (p < 0.05) criteria were taken as the confidence interval in the analyses. SPSS 25 and AMOS 20 statistical package programs were used to analyze the data.

Results

The characteristics of the athletes were presented in Table 1.

Table 1. Characteristics of athletes (n=565)

When examining the goodness of fit indexes of the APSQ according to the first-order CFA analysis, results indicated good model fit; χ²=80.63, p<0.01, sd=30 (χ²/sd=2.68), RMSEA=0.05, CFI=0.96, GFI=0.97, IFI=0.96, NFI=0.94, RFI=0.92 and SRMR=0.03 (Table 2). Modification indices were examined to determine covariance pathways among residual values. (e2-e3, e5-e8). The results of the CFA of the APSQ are presented in Figure 1.

Table 2. APSQ first-order multifactor model confirmatory factor analysis fit indices

Figure 1. Confirmatory factor analysis (CFA)

In order to evaluate the concurrent validity of the APSQ domains, the correlations between the self-regulation, performance concerns, and external coping sub-dimensions of the scale, and the K-10 and DASS-21 were evaluated. K-10 and DASS-21, APSQ total scores and sub-dimensions were found to be moderately, positively correlated. When K-10 and DASS-21 were considered as independent variables, they have a predictive feature as well as a relationship with the APSQ (ΔR²=0.47, p<0.01) (Table 3). The test-retest application was performed on the same sample (n=61) with 30 days interval to measure the stability of the APSQ, and the correlation between the two applications was reported as high and positive in the APSQ (r=0.88; p<0.01).

Table 3. Concurrent validity results for criterion-related validity

According to the evaluation of APSQ to gender, there was a significant difference between males and females on the external coping sub-dimension in favor of female athletes (t=-2.91; p<0.01), but not the other APSQ domains.  For age, the external coping sub-dimension scores of 18-22 years old athletes were significantly lower compared to 28-32 years of athletes (t=5.03; p<0.01).  According to the type of sport, assessing self-regulation sub-dimension scores of team sports athletes were significantly lower compared to individual sports athletes (t=-2.11; p<0.01) (Table 4).

Table 4. Comparison of APSQ total and subdimension scores according to variables

For clinical consideration, the proportion of moderate, high and very high scores of female athletes were 13.7%, 20.6% and 34.8%, respectively. Among male athletes, the proportion of moderate, high and very high scores were 16%, 14.4% and 41.5%, respectively (Table 5). There was no relationship between gender according to the APSQ cut-off values (Pearson Chi-square=5.55; p=0.13).

Table 5. Frequency and relationship between genders according to the APSQ cut-off values

Discussion

The current study aimed to examine the reliability and validity of the APSQ for implementation with Turkish athletes. To this end, the questionnaire was translated into Turkish, and its psychometric properties were comprehensively examined through the studies outlined above. The findings provide preliminary evidence that the APSQ is a psychometrically sound instrument to examine Turkish athletes’ MH from a screening perspective. Thus, by using the APSQ, which is adapted to the culture and environment of elite athletes, early manifestations of MH problems of Turkish athletes may be detected and preventive strategies initiated.

For language equivalence, the findings demonstrated that the correlation coefficient between the original measure and the Turkish form of the questionnaire was high, and the difference between scores obtained from the forms was not significant in any of the questionnaire items. Therefore, both forms were equivalent linguistically and the Turkish form of the APSQ can be used as a measurement tool (32).

While the Japanese version of the APSQ did not demonstrate the three-factor structure of the original questionnaire (11), the CFA results of the present study supported the three-factor structure of the questionnaire (10).  The validity of the APSQ structure was demonstrated with the CFA results, and the values were within acceptable and perfect reference ranges. For concurrent validity, the relationship between the APSQ and DASS-21, K-10 was revealed, and the predictive power of these scales for APSQ supported the relationship results. It’s proposed that a relationship between the studies is proven by the theoretical foundations in the original scale development study being similar with the findings from our study (10). In this context, the strength of the relationship can be presented as evidence which is equivalent in this adaptation. Although the present study coincided with the COVID-19 pandemic period, the fact that the relationship between the APSQ and sub-dimensions scores and the averages of responses obtained from the equivalent forms were consistent revealed that how robust the APSQ structure is (regardless of the period and conditions).  Nevertheless, it would be useful to evaluate the effects of the COVID-19 pandemic on the MH of athletes by using the APSQ.

According to the results of our study, the low Cronbach alpha coefficient of the external coping sub-dimension of the APSQ may be related that the low number of items in the sub-dimension, and the questionnaire evaluate a very specific group. Thus, in the Chinese version of the APSQ, Tan et al. reported that the external coping sub-dimension Cronbach alpha coefficient was 0.58 (12). Nonetheless, since there are studies in which the Cronbach α coefficient varies between 0.43 and 0.70 in scales that evaluate very specific structures such as psychology, personality, and attitude disorder, it can be said that our study results are acceptable (33).

According to the evaluation of the APSQ by gender, there was a significant difference between male and female athletes in the external coping sub-dimension in favor of females. Rosenfield et al. reported women experience more internalized mental disorders, whereas men are more affected and experience externalized factors (34). Researchers also indicated that men provide their coping strategies from their external motivation factors (35). It can be said that the results in our study support the literature revealing differences in external coping between genders.

It was reported that mental stamina is directly proportional in sports with age, with older athletes having higher mental stamina than youngers (36). On the other hand, career transition is a critical phase for athletes. Athletes at the end of their careers may exhibit maladaptive coping strategies, and therefore, have difficulties in creating coping with external factors. Additionally, since the conduction of this study coincided with the COVID-19 pandemic, older athletes might have been affected more by the situations associated with the pandemic (such as family, responsibilities, and financial losses) than youngers (37). Gulliver et al. also reported that younger athletes are more open to receive assistance than adults (38). Therefore, it can be said that external coping sub-dimension scores are lower in younger athletes due to their easier acceptance of external stimuli and supports.

According to the type of sport, self-regulation sub-dimension scores of team sports athletes were significantly lower compared to individual sports athletes. Research suggests that MH outcomes can change according to sporting type and can be affected by different dynamics (39). For example, in team sports, social support is provided by both the teammates and the units affiliated to the team, while the athlete may have difficulty in perceiving social support in individual sports. Since individual athletes were affected more negatively than the team athletes in the pandemic period (8), they might have had difficulty in self-regulation.

The study has some limitations. First, there is no clinical assessment, and all data was collected by self-report system. Secondarily, the study coincided with the COVID-19 pandemic. The effects of the COVID-19 pandemic might have differed depending on age, gender, or sports discipline. Lastly, although the present study comprised a sufficiently large sample, there was an imbalance in age and marital status distribution in the sample.

Despite the growing interest in athletes' MH, there are currently no systematic protocols for the early detection or treatment of MH issues in Turkey. Considering that the psychological states of Turkish athletes were negatively affected during the COVID-19 pandemic (8, 40), the assessment of MH issues in athletes has become more critical. Thus, our results show that most athletes (70.7%) reported symptoms of moderate, high, or very high levels of athlete-specific distress. Rice et al. reported similar rates of athlete-specific distress in Australia; however, they were scrutinized at moderate and high levels (13). Considering that Rice et al. conducted their study before the pandemic, the present study results suggest that the COVID-19 pandemic may affect elite-level athletes' MH. We also thought that the differences in coping strategies of athletes living in different cultures and societies may have an impact on the MH of athletes.  Additionally, our results showed that while only 23% of athletes reported symptoms of at least moderate severity of nonspecific stress, this proportion was 70.7% for athlete-specific distress. This remarkable result suggests that athlete-specific problems may not be noticed with the screening tools used for the general population. Lastly, Gouttebarge et al. showed that the APSQ allows for early-stage identification of those athletes with MH issues, including anxiety, depression, sleep disturbance, alcohol misuse, substance misuse, and eating disorders (4).

Conclusion

Findings from the present study illustrate that APSQ can be used as a screening tool to detect MH problems in Turkish athletes. The MH assessment using APSQ in elite Turkish athletes will help detect vulnerable athletes and enable appropriate and timely MH interventions.

Appendix. 1

Cite this article as: Lima Y, Oz N, Denerel N, Ozkaya O, Senisik S, Rice S. Validity and reliability of the Turkish version of Athlete Psychological Strain Questionnaire (APSQ). Turk J Sports Med. 2022, 57(3):147-54; https://doi.org/10.47447/tjsm.0637

The approval for this study was obtained from Ege University Clinical Research Ethics Committee (Decision no: E-43746 Date: 14.02.2021).

Concept YL, NDÖ, SR; Design YL, NDÖ, SR; Supervision All authors; Materials YL, ND, ÖÖ, SŞ; Data Collection and/or Processing YL, ND, ÖÖ, SŞ; Analysis and Interpretation YL, NDÖ, SR; Literature Review All authors; Writing Manuscript YL, NDÖ, SR; Critical Reviews All authors.

The authors declared no conflicts of interest with respect to authorship and/or publication of the article.

The authors received no financial support for the research and/or publication of this article.

References

1. Bahr R, Holme I. Risk factors for sports injuries—a methodological approach. Br J Sports Med. 2003;37(5):384-92. DOI: 10.1136/bjsm.37.5.384
2. Williams JM, Andersen MB. Psychosocial antecedents of sport injury: Review and critique of the stress and injury model. J Appl Sport Psychol. 1998;10(1):5-25. DOI: 10.1080/10413209808406375
3. Schinke RJ, Stambulova NB, Si G, Moore Z. International society of sport psychology position stand: Athletes’ mental health, performance, and development. Int J Sport Exerc Psychol. 2018;16(6):622-39. DOI: 10.1080/1612197X.2017.1295557
4. Gouttebarge V, Bindra A, Blauwet C, Campriani N, Currie A, Engebretsen L, et al. International Olympic Committee (IOC) Sport Mental Health Assessment Tool 1 (SMHAT-1) and Sport Mental Health Recognition Tool 1 (SMHRT-1): towards better support of athletes’ mental health. Br J Sports Med. 2020;55(1):30-7. DOI: 10.1136/bjsports-2020-102411
5. Gouttebarge V, Castaldelli-Maia JM, Gorczynski P, Hainline B, Hitchcock ME, Kerkhoffs GM, et al. Occurrence of mental health symptoms and disorders in current and former elite athletes: a systematic review and meta-analysis. Br J Sports Med. 2019;53(11):700-6. DOI: 10.1136/bjsports-2019-100671
6. Reardon CL, Hainline B, Aron CM, Baron D, Baum AL, Bindra A, et al. Mental health in elite athletes: International Olympic Committee consensus statement (2019). Br J Sports Med. 2019;53(11):667-99. DOI: 10.1136/bjsports-2019-100715
7. Kara OS, Buyukluoglu G, Buyukluoglu N, Gul S, Celebi M, Kaya H. Professional athletes have higher anxiety levels during COVID-19 outbreak compared to recreational athletes and sedentary people. Turk J Sports Med. 2021;56(2).73-80
8. Lima Y, Senisik S, Denerel N, Hursitoglu H, Balci GA, Bolat GU, et al. Effects of COVID-19 pandemic on the psychological states of youth and adult elite male athletes. Turk J Sports Med. 2021 Dec 11; https://doi.org/10.47447/tjsm.0597 DOI: 10.47447/tjsm.0597
9. Vaughan RS, Edwards EJ, MacIntyre TE. Mental health measurement in a post Covid-19 world: psychometric properties and invariance of the DASS-21 in athletes and non-athletes. Front Psychol. 2020;(11):590559.doi:10.3389/fpsyg.2020.590559.  DOI: 10.3389/fpsyg.2020.590559
10. Rice SM, Parker AG, Mawren D, Clifton P, Harcourt P, Lloyd M, et al. Preliminary psychometric validation of a brief screening tool for athlete mental health among male elite athletes: the athlete psychological strain questionnaire. Int J Sport Exerc Psychol. 2020;18(6):850-65. DOI: 10.1080/1612197X.2019.1611900
11. Ojio Y, Matsunaga A, Kawamura S, Horiguchi M, Yoshitani G, Hatakeyama K, et al. Validating a Japanese Version of the Athlete Psychological Strain Questionnaire. Sports Med Open. 2021;7(1):90. Doi: 10.1186/s40798-021-00385-9 DOI: 10.1186/s40798-021-00385-9
12. Tan Chen-hao YJ, Cao Guo-huan, Qiu Jun, Zhao De-feng. Revision and validation of Chinese version of athlete psychological strain questionnaire. Modern Preventive Med. 2021;48:2887-91.
13. Rice S, Olive L, Gouttebarge V, Parker AG, Clifton P, Harcourt P, et al. Mental health screening: severity and cut-off point sensitivity of the Athlete Psychological Strain Questionnaire in male and female elite athletes. BMJ Open Sports Exerc Med. 2020;6(1):e000712. Doi: 10.1136/bmjsem-2019-000712 DOI: 10.1136/bmjsem-2019-000712
14. Lovibond PF, Lovibond SH. The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther. 1995;33(3):335-43. DOI: 10.1016/0005-7967(94)00075-U
15. Yıldırım A, Boysan M, Kefeli MC. Psychometric properties of the Turkish version of the Depression Anxiety Stress Scale-21 (DASS-21). British Journal of Guidance & Counselling. 2018;46(5):582-95. DOI: 10.1080/03069885.2018.1442558
16. Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand S-L, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959-76. DOI: 10.1017/S0033291702006074
17. Altun Y, Ozen M, Kuloglu MM. Turkish adaptation of Kessler Psychological Distress Scale: validity and reliability study/Psikolojik Sikinti Olceginin Turkce uyarlamasi: Gecerlilik ve guvenilirlik calismasi. Anadolu Psikiyatri Derg. 2019;20(SI 1):23-32. DOI: 10.5455/apd.12801
18. who.int [Internet]. Process of translation and adaptation of instruments; 2017. [Cited 10.11.2021] Available from: https://www.who.int/substance_abuse/research_tools/translation/en/.
19. Lawshe CH. A quantitative approach to content validity. Personnel Psychology. 1975;28(4):563-75. DOI: 10.1111/j.1744-6570.1975.tb01393.x
20. Davis LL. Instrument review: Getting the most from a panel of experts. Appl Nurs Res. 1992;5(4):194-7. DOI: 10.1016/S0897-1897(05)80008-4
21. Brown T. Confirmatory factor analysis for applied research. New York, NY: Guilford. KA Bollen & JS Long. 2006.
22. Bollen KA. Measurement models: The relation between latent and observed variables. Structural Equations with Latent Variables. New York: John Wiley & Sons;1989.p.179-225. DOI: 10.1002/9781118619179.ch6
23. Satorra A, Bentler PM. A scaled difference chi-square test statistic for moment structure analysis. Psychometrika. 2001;66(4):507-14. DOI: 10.1007/BF02296192
24. Steiger, J. H., & Lind, J. C. Statistically Based Tests for the Number of Common Factors. Psychometric Society Annual Meeting, Iowa City, IA;1980.
25. Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990;107(2):238-46. DOI: 10.1037/0033-2909.107.2.238
26. Tanaka JS, Huba GJ. Structures of psychological distress: Testing confirmatory hierarchical models. J Consult Clin Psychol. 1984;52(4):719-21. DOI: 10.1037/0022-006X.52.4.719
27. Bentler PM, Bonett DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bull. 1980;88(3):588-606. DOI: 10.1037/0033-2909.88.3.588
28. Tucker LR, Lewis C. A reliability coefficient for maximum likelihood factor analysis. Psychometrika. 1973;38(1):1-10. DOI: 10.1007/BF02291170
29. sicotest.com [Internet]. Fit indices for structural equation modeling; 2009. [Cited 10.11.2021]. Avaible from: https://www.sicotests.com/psyarticle.asp?id=277.
30. Schermelleh-Engel K, Moosbrugger H, Müller H. Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. MPR-Online. 2003;8(2):23-74.
31. Adcock R, Collier D. Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review. 2001;95(3):529-46. DOI: 10.1017/S0003055401003100
32. Brislin RW, Lonner WJ and Thorndike RM. Questionnaire wording and translation. In:  Brislin RW, Lonner WJ and Thorndike RM. (Eds), Cross-Cultural Research Methods, 1st ed.New York:John Wiley & Sons, 1973. p.32-58.
33. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334. DOI: 10.1007/BF02310555
34. Rosenfield S. Gender and mental health: do women have more psychopathology, men more, or both the same (and why)? In: Horowitz A, Scheid T, eds. A Handbook for the Study of Mental Health: Social Contexts, Theories, and Systems. New York, NY: Cambridge University Press. 1999;348–360
35. Duncan MC, Messner MA. The media image of sport and gender. In: Wenner L. Editor, MediaSport. 1st ed. London:Routledge;1998.p.170-85.
36. Jones G. What is this thing called mental toughness? An investigation of elite sport performers. J Appl Sport Psychol. 2002;14(3):205-18. DOI: 10.1080/10413200290103509
37. Wilson JM, Lee J, Fitzgerald HN, Oosterhoff B, Sevi B, Shook NJ. Job insecurity and financial concern during the COVID-19 pandemic are associated with worse mental health. J Occup Environ Med. 2020;62(9):686-91. DOI: 10.1097/JOM.0000000000001962
38. Gulliver A, Griffiths KM, Christensen H. Barriers and facilitators to mental health help-seeking for young elite athletes: a qualitative study. BMC Psychiatry. 2012;12:157. doi: 10.1186/1471-244X-12-157. DOI: 10.1186/1471-244X-12-157
39. Bull SJ, Shambrook CJ, James W, Brooks JE. Towards an understanding of mental toughness in elite English cricketers. J Appl Sport Psychol. 2005;17(3):209-27. DOI: 10.1080/10413200591010085
40. Lima Y, Denerel N, Yilmaz ND, Senisik S. The psychological impact of COVID-19 infection on athletes: Example of professional male football players. Sci and Med Football. 2021;5(sup1):53-61. Doi: 10.1080/24733938.2021.1933156 DOI: 10.1080/24733938.2021.1933156