CC BY
26
mfts and aqsa scales validation in patients with multiple and concomitant foot fractures
Korolev MA ^ , Yarmak DO, Miroschnikova CA, Korobushkin GV
Department of Traumatology, Orthopedics and Field Surgery, Faculty of Pediatrics Pirogov Russian National Research Medical University, Moscow, Russia
To assess the effectiveness of treatment in traumatology, different scales and assessment questionnaires are used. This work presents the results of the validity test of the two scales designed by the authors, namely, Moscow Foot Trauma Scale (MFTS) and Abbreviated Questionnaire of Subjective Assessment (AQSA). The study enrolled 79 patients (59 male and 20 female individuals with a mean age of 42) with multiple or concomitant foot fractures. For the scales, coefficients of reliability, stability, constancy, internal consistency (Cronbach's alpha), split-half correlation (Guttman's lambda) and intraclass correlation were calculated. SF-36 (Short Form 36) and AOFAS (American Orthopaedic Foot and Ankle Society Score) were used as reference scales. The study revealed a high reproducibility of the new scales: stability coefficient was 0.85-0.96 for MFTS and up to 0.93 for AQSA. Their reliability and internal consistency were established.
Keywords: MFTS, AQSA, SF-36, AOFAS, rating scale, validity, convergent validity, internal consistency coefficient, split-half correlation coefficient, intraclass correlation coefficient, foot fractures, multiple injury
gg Correspondence should be addressed: Maksim Korolev
Leninskiy prospect, d. 10, korp. 7, Moscow, Russia, 117049; doctorkoroleff@mail.ru
Recieved: 30.03.2016 Accepted: 07.04.2016
валидизация шкал mfts и aqsa у больных с переломами костей стопы в составе множественной и сочетанной травмы
М. А. КоролёвД. О. Ярмак, Е. А. Мирошникова, Г. В. Коробушкин
Кафедра травматологии, ортопедии и военно-полевой хирургии, педиатрический факультет, Российский национальный исследовательский медицинский университет имени Н. И. Пирогова, Москва
Для оценки эффективности лечения в травматологии применяют различные шкалы и оценочные опросники. В работе представлены данные проверки валидности двух шкал, разработанных авторами: Moscow Foot Trauma Scale, MFTS (московская шкала оценки функции стопы после травмы) и Abbreviated Questionnaire of Subjective Assessment, AQSA (сокращенный опросник субъективной оценки). В исследовании участвовали 79 пациентов (59 мужчин, 20 женщин; средний возраст — 42 года) с переломами костей стопы в составе сочетанной и множественной травмы. Для шкал рассчитывали коэффициенты надежности, стабильности, константности, внутренней согласованности (альфа Кронбаха), раздельной корреляции (лямбда Гутмана) и внутригрупповой корреляции. В качестве шкал-эталонов использовали SF-36 (Short Form 36) и AOFAS (American Orthopaedic Foot and Ankle Society Score). Исследование выявило высокую воспроизводимость новых шкал: коэффициент стабильности был равен 0,85-0,96 для MFTS и до 0,93 для AQSA. Была отмечена их надежность и внутренняя согласованность.
Ключевые слова: MFTS, AQSA, SF-36, AOFAS, шкала оценки, валидность, конвергентная валидность, коэффициент внутренней согласованности, коэффициент раздельной корреляции, коэффициент внутригрупповой корреляции, переломы костей стопы, множественная травма
Для корреспонденции: Максим Александрович Королёв
117049, г Москва, Ленинский пр-т, д. 10, корп. 7; doctorkoroleff@mail.ru
Статья получена: 30.03.2016 Статья принята в печать: 07.04.2016
Modern requirements on provision of medical care to patients with severe injuries are now compelling physicians to focus not just on early diagnosis, active treatment and subsequent rehabilitation of the patient, but also on the cost-effectiveness of procedures, reduction of treatment and rehabilitation duration and reduction in disability. In addition to objective difficulties, such as short duration of stay by the patient at the hospital, lack of medical history data and general serious condition, diagnosis and treatment of concomitant and multiple injuries carry considerable financial and material costs for clinics and health insurance funds. This underlines the importance of objective evaluation of efficacy of treatment using rating scales.
There are so many scales and assessment questionnaires. There are non-specific scales (general assessment: VAS, NRS, SF-36, etc.) and specific scales (for a certain anomaly: AOFAS, FFI, DASH, etc.), which can be characterized by validity, compliance (friendliness), reliability, reproducibility of results in subsequent studies, and sensitivity to objective changes in indicators. A particular assessment tool for a specific group of patients is chosen based on these parameters [1-3].
However, the existing scales are imperfect. There is need to create new questionnaires for assessment of treatment results and correctly interpret the results; validation of these questionnaires is also required [4-7]. We developed two new
CTATbfl I TPABMATQflQrMfl
scales at the Department of Traumatology, Orthopedics and Field Surgery, Faculty of Pediatrics, Pirogov Russian National Research Medical University.
Moscow Foot Trauma Scale (MFTS) is a specific scale for evaluation of treatment outcome after foot injury. It consists of subjective and objective parts, each of which includes 3 multiple-choice questions (fig. 1). At the end of the scale are keys indicating the number of points for each answer. Possible score ranges from 0 to 90. Possible treatment effectiveness assessment results: 90-61 p — excellent; 60-41 p — good; 40-21 p — satisfactory; 20-11 p — bad; 10-0 p — very bad.
Abbreviated Questionnaire of Subjective Assessment (AQSA) is a non-specific scale that can be used to assess whether a patient has limited activity, whether he needs special orthopedic shoes or additional support, whether his pain syndrome needs to be relieved using analgesics, and to assess the change in the nature of work performed. The Questionnaire includes 6 questions (fig. 2). At the end of the scale are keys that indicate the number of points for each answer. The possible
To be completed by the doctor Objective part_
1) Function (range of motions)
1.1) Active motions
A) Full — 100 %
B) Moderately disabled — over 50 %
C) Strongly limited — less than 50 %
1.2) Passive movements
A) Full — 100 %
B) Moderately limited — over 50 %
C) Strongly limited — less than 50 %
2) Use of additional support and orthopedic products
A) No
B) Sometimes
C) Constantly
3) Extremity support ability
A) Full
B) Moderate
C) Low
score ranges from 0 to 30. The higher the score, the worse the result: 0-10 p — good; 11-20 p — satisfactory, above 20 p — bad.
Before using the newly developed scale, it is necessary to confirm its theoretical and pragmatic validity in the conditions of use. Confirmation of theoretical validity enables to establish whether this scale indeed assesses the indicator needed by us, while confirmation of pragmatic validity allows to determine how well does the scale perform its function in practice when dealing with patients. Validity is interpreted in different ways, depending on the task. Validity usually refers to the degree of confidence at which a test, measurement or experiment actually performs the function for which it is intended [8].
Checking the validity of the new assessment tool is quite a challenging task. In traumatology, the SF-36 (Short Form 36) and AOFAS (American Orthopaedic Foot and Ankle Society Score) scales are most commonly used to verify theoretical validity. These scales have confirmed their stability in population-based studies on large and relatively homogeneous samples [9-13].
During analysis, correlation between the attributes assessed by the scale and similar attributes of the reference scale should be revealed, and there should be no correlation with symptoms that have other theoretical grounds. Meeting these conditions means that the scope of the scale has been chosen correctly.
Pragmatic validity was evaluated by an external attribute, which should be relevant (i.e. correspond to the test attribute by meaning), free from interference (this is usually ensured by formation of a fairly homogeneous sample) and reliable [14].
Before validity checking, it is required to establish the level of system reliability. Reliability is a relative constancy, stability, consistency of test results in initial and repeated use on the same group of patients. Gurevich recommends interpreting reliability as: 1) reliability of the measuring instrument itself; 2) stability of the test attribute; 3) constancy, i.e., relative independence of results from the experimenter's identity.
To be completed by the patient Subjective part_
4) Pain
A) None
B) Moderate
C) Severe
D) Very severe
E) Unbearable
5) Socialization
A) Previous work without limitations
B) Previous work with limitations
C) Changed to an easier work
D) I don't work because of a foot injury
E) I don't work for other reasons
6) Satisfaction with outcome
A) Excellent
B) Good
C) Satisfactory
D) Bad
E) Very bad
Keys (in points)
1.1) A 20, B 10, C 0
1.2) A 3, B 2, C 0
2) A 2, B 1, C 0
3) A 4, B 1, C 0
4) A 15, B 10, C 5, D 1, E 0
5) A 40, B 30, C 20, D 10, E 0
6) A 6, B 3, C 2, D 1, E 0
Interpretation of results
90-61 p — excellent; 60-41 p — good; 40-21 p — satisfactory; 20-11 p — bad; 10-0 p — very bad.
Fig. 1. Questionnaire for assessing the effectiveness of treatment of foot fractures using the Moscow Foot Trauma Scale (MFTS)
Question
1) Are you working now?
A) Yes
B) No
2) What kind of work?
A) Previous
B) Lightweight
C) Disability
3) Do you take painkillers?
A) Yes
B) No
4) Do you have any physical disability?
A) Yes
B) No
5) Do you use additional means of support?
A) Yes
B) No
6) What kind of shoes do you wear?
A) Regular
B) Orthopedic
C) Insoles
Keys (in points)
1) A 0, B 5
2) A 0, B 3, C 5
3) A 5, B 0
4) A 5, B 0
5) A 5, B 0
6) A 0, B 5, C 3
Interpretation of results
0-10 p — good; 11-20 p — satisfactory, above 20 p — bad.
Fig. 2. The questionnaire for assessing the effectiveness of treatment of foot fractures using the Abbreviated Questionnaire of Subjective Assessment (AQSA)
He proposes calculating three corresponding coefficients: reliability, stability and constancy [15].
Reliability coefficient was calculated by split-half reliability method, in which the test was divided into equal parts and the correlation between their values calculated. The technique is considered reliable if the correlation coefficient is above 0.75.
The stability coefficient of the attribute under study is determined through test-retest. Its meaning lies in the retesting of the group being investigated using the methodology under study. The correlation coefficient between the initial and repeated test characterizes the stability of the attribute.
Constancy is checked by the testing of one test group based on one technique but by different researchers. The correlation coefficient should be greater than 0.80 provided the same conditions apply.
Apart from calculation of these coefficients, it is possible to assess the reliability of the system by equivalent-form technique, which requires creation of similar forms of one test and testing that test on a large group of patients with correlation calculated. We didn't use it due to large labor input and relatively small sample involved.
In addition to calculating the above indicators, the coefficient of internal consistency (Cronbach's alpha, aK), split-half correlation coefficient (Guttman's lambda, ÁG) and Kuder-Richardson coefficient (KR20) should be calculated [16-18].
The aim of the study was to test the validity of the MFTS and AQSA scales for patients with multiple and concomitant foot fractures.
METHODS
The study included 79 patients (59 men and 20 women; mean age - 42 years) at the trauma unit of Pirogov City Clinical Hospital No 1. The patients were treated for multiple and concomitant foot fractures in 2007-2016. Surgical treatment was carried out in 32 patients, conservative treatment in 47. The study was approved by the Ethics Committee of Pirogov Russian National Research Medical University (Minutes No 139 of 10 November 2014). All the patients gave informed consent to participate in the study.
By localization, right foot fractures occurred more frequently (n = 39) than left foot fractures (n = 27) and were bilateral in 13 patients. Initial examination revealed 69 fractures, which is 54.3 % of the total number of fractures. Subsequent examinations by traumatologists and other specialists detected other 24 fractures (26 %). Another 28 fractures were late-diagnosed and 12 of these cases needed surgical treatment. The more severe a patient's condition was, the higher the likelihood of diagnosing the fractures late. In severe condition assessed on the ISS scale (Injury Severity Score), 11 and 17 fractures were undiagnosed for scores less than 16 p and above 16 p respectively. Before injury, 55 people were able to work fully. After treatment, 37 patients retained their working ability.
After multiple-stage or single-stage treatment, the patients were discharged from the hospital and observed at an injury care center near where they lived. When necessary, the patients were sent to major hospitals for consultations.
The patients were divided into two groups: group of patients with retrospective observation (n = 36) and group with prospective observation (n = 43). The treatment results were evaluated 1, 3, 6 and 12 months after injury or the treatment was repeated using the developed scales MFTS and AQSA and scales SF-36 and AOFAS. The coefficients of reliability, stability,
constancy, aK, ÁG and the intraclass correlation coefficient were calculated for all the scales. The Kuder-Richardson coefficient was not calculated due to the non-dichotomous nature of all the scales. While using the test-retest approach, repeated testing was performed after 11 ± 3.2 days.
These numerical values of treatment results for patients with foot fractures showed the importance of careful attention to diagnosis and treatment of foot bones, as well as timely and thorough examination of the patient with further treatment tactics.
The Statistica 10.0 software (StatSoft, USA) was used for statistical data analysis. With a relatively small sample, a significance level of p <0.05 was taken. For data analysis, non-parametric statistics methods were used due to the presence of data distribution in most cases that is different from normal distribution.
RESULTS
Table 1 presents the treatment effectiveness assessment results after 12 months using reference scales and the studied scales as an example of the use of scales. Like assessment using the SF-36 and AOFAS scales, the MFTS and AQSA assessments confirmed a pattern: the later the fractures are diagnosed, the lower the treatment effectiveness.
The MFTS scale recorded the highest Cronbach's coefficient that indicates the internal consistency of a scale, while the physical component summary of the SF-36 scale yielded the biggest Guttman coefficient (PCS) (tab. 2). The ÁG value was also high in MFTS.
For the MFTS scale, in different periods of testing, the values of the constancy coefficient ranged from 0.81 to 0.93, for the AQSA scale — from 0.57 to 0.69.
The intraclass correlation coefficient defined through the test-retest approach was equal to 0.85-0.96 for MFTS and 0.76-0.93 for AQSA. For both scales, p <0.05. This result indicates there is high dependency of indicators within the MFTS and AQSA scales.
Table 3 shows convergent correlations between the reference scales and studied scales for both groups of patients. As can be seen, they have low level of significance. Correlation between MFTS and AOFAS and correlation between MFTS and the physical component summary of the SF-36 scale were identified, which is logical, given the one-way specialization of these questionnaires. However, all the values had a low level of significance. Therefore, the existence of a real relationship between the scales can only be assumed.
In view of its low specificity, AQSA correlated with both physical and mental components of the SF-36 scale. It correlated negatively with AOFAS.
DISCUSSION
The high Cronbach's coefficient recorded in the MFTS scale indicates there is optimal construction of questions in the scale. With the help of the Statistica 10.0 software, the need to exclude similar questions was assessed. For the MFTS scale, it was recommended to remove 2 questions (in order to reduce the value of aK). For the AOFAS and AQSA scales, it was recommended to add from 1 to 3 questions to improve the internal consistency of the evaluating tool.
Guttman coefficient confirmed the effectiveness of assessment with the use of physical component summary
СТАТЬЯ I ТРАВМАТОЛОГИЯ
Table 1. Treatment effectiveness assessment (12 months) using the SF-36, AOFAS, MFTS and AQSA scales
Scale Foot fractures
Diagnosed early Diagnosed late Undiagnosed
SF-36 (PCS/MCS) 51/47 38/46 34/28
AOFAS 54 43 31
MFTS 45 22 15
AQSA 7 18 16
Note: PCS — Physical Component Summary of the SF-36 scale, MCS — Mental Component Summary of the SF-36 scale. The results are presented as arithmetic mean.
Table 2. Clinical and metric properties of the SF-36, AOFAS, MFTS and AQSA scales
Scale SF 36 AOFAS MFTS AQSA
PCS MCS
Number of questions 21 15 9 6 6
Cronbach's coefficient (aK) 0.982 0.957 0.989 0.993 0.99
Guttman coefficient (AG) 0.986 0.951 0.983 0.985 0.981
Intraclass correlation coefficient 0.896 0.769 0.93 0.961 0.936
Note: PCS — Physical Component Summary of the SF-36 scale, MCS — Mental Component Summary of the SF-36 scale.
of the SF-36 scale (0.986). The high values of the test-retest indicator — 0.85-0.96 for MFTS and up to 0.93 for AQSA — reflected the good reproducibility of these scales.
The data obtained indicate there is a fairly high level of individual validity of the MFTS and AQSA scales. However, the convergent validity values were lower than those of the reference scales. One cannot tell exactly what will be the correlation between the indicators under the conditions of another experiment, but it is important that a relationship is still there. Perhaps in future studies involving different groups of patients, we will be able to confirm that there is a relationship.
CONCLUSIONS
Statistical data analysis confirmed that the MFTS scale has a high validity and compliance for the doctor, and high sensitivity and reliability. The disadvantages are average reproducibility and low compliance for the patient. The AQSA scale showed high reliability, reproducibility and compliance for the doctor and patient, but low validity and sensitivity. The convergent validity values of these scales with the SF-36 and AOFAS scales showed there is a weak correlation between the scales.
The MFTS and AQSA scales can be used to assess the effectiveness of treatment of patients with multiple and concomitant foot fractures. In this case, the peculiarities indicated for them should be taken into account.
Table 3. Convergent validity values for the SF-36, AOFAS, MFTS and AQSA scales
Scale SF-36 PCS SF-36 MCS AOFAS
R P R P R P
MFTS R 0.183 - -0.206 - 0.22 -
P - 0.215 - -0.719 - 0.104
AQSA R 0.227 - 0.170 - -0.301 -
P - 0.378 - 0.351 - -0.292
Note: PCS — Physical Component Summary of the SF-36 scale, MCS - Mental Component Summary of the SF-36 scale; R — group of patients with retrospective observation, P — group of patients with prospective observation; p <0.05.
References
1. Viskarra ME, Zimina EV, Gurina SA, Yarygin NV. Validizatsiya shkal i voprosnikov dlya otsenki funktsional'nogo sostoyaniya i kachestva zhizni patsientov s perelomovyvikhami golenostopnogo sustava. Rossiyskie meditsinskie vesti. 2011; 16 (2): 61-71. Russian.
2. Novik AA, lonova TI. Rukovodstvo po issledovaniyu kachestva zhizni v meditsine. Moscow: lzd-vo "OLMA Media Grupp"; 2007. 313 p. Russian.
3. Knop C, Oeser M, Bastian L. Development and validation of the visual analogue scale (VAS) spine score. Unfallchirurg. 2001; 104 (6): 488-97. German.
4. Evsina OV. The quality of life in medicine — an important andicator pf patient health status (review). Lichnost' v menjajushhemsja mire: zdorov'e, adaptacija, razvitie. 2013; (1): 119-33. Russian.
5. Viskarra ME, Yarygin NV, Zimina EV. Analiz faktorov, vliyayushchikh na kachestvo zhizni i funktsional'noe sostoyanie patsientov s perelomovyvikhami golenostopnogo sustava. Problemy standartizatsii v zdravookhranenii. 2011; (3-4): 9-15. Russian.
6. Viskarra ME. Izuchenie kachestva zhizni i funktsional'nogo sostoyaniya patsientov s perelomovyvikhami golenostopnogo sustava s pomoshch'yu shkal i oprosnikov [dissertation]. Moscow: M. I. Evdokimov Moscow State University of Medicine and Dentistry; 2011. Russian.
7. Glantz S. Mediko-biologicheskaya statistika. Moscow: lzd-vo "Praktika"; 1998. 459 c.
8. Golovin SYu. Slovar' prakticheskogo psihologa. Minsk: lzd-vo "Kharvest"; 1998. 800 p.
9. Amirdzhanova VN, Goryachev DV, Korshunov Nl, Rebrov AP, Sorotskaya VN. SF-36 questionnaire population quality of life indices. Nauchno-prakticheskaya revmatologiya. 2008; (1): 36-48.
10. Jenkinson C, Coulter A, Wright L. Short form 36 (SF-36) health survey questionnaire: normative data for adults of working age. BMJ. 1993; 306 (6890): 1437-40.
11. Kitaoka HB, Alexander U, Adelaar RS, et al. Clinical rating systems for the ankle hindfoot, midfoot, hallux and lesser toes. Foot Ankle lnt.1994. 15: 349-53.
12. SooHoo NF, Shuler M, Fleming LL; American Orthopaedic Foot and Ankle Society. Evaluation of the validity of the AOFAS Clinical Rating Systems by correlation to the SF-36. Foot Ankle lnt. 2003 Jan; 24 (1): 50-5.
13. Ware JE, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). l. Conceptual framework and item selection. Med Care. 1992 Jun; 30 (6): 473-83.
14. Schuh A, Hausel M. Difficlties in evaluating follow-up outcome in calcaneus fracture managed with plate osteosinthesis. ls there a reliable score? Unfallchirurg. 2000; 103 (4): 295-300. German.
15. Gurevich KM, Borisova EM. Psihologicheskaja diagnostika. Moscow: lzd-vo "URAO"; 1997. 304 p. Russian.
16. Cronbach LJ. Coefficient alpha and the internal structure of tests.
Psychometrika. 1951; 16: 297-334.
17. Kline P. A handbook of test construction (psychology revivais): introduction to psychometric design. Abingdon, UK: Routledge; 2015. 274 p.
Литература
1. Вискарра М. Э., Зимина Э. В., Гурина С. А., Ярыгин Н. В. Ва-лидизация шкал и вопросников для оценки функционального состояния и качества жизни пациентов с переломовывиха-ми голеностопного сустава. Российские медицинские вести. 2011; 16 (2): 61-71.
2. Новик А. А., Ионова Т. И. Руководство по исследованию качества жизни в медицине. М.: Изд-во «ОЛМА Медиа Групп»; 2007. 313 с.
3. Knop C, Oeser M, Bastian L. Development and validation of the visual analogue scale (VAS) spine score. Unfallchirurg. 2001; 104 (6): 488-97. German.
4. Евсина О. В. Качество жизни в медицине — важный показатель здоровья пациента (обзор литературы). Личность в меняющемся мире: здоровье, адаптация, развитие. 2013; (1): 119-33.
5. Вискарра М. Э., Ярыгин Н. В., Зимина Э. В. Анализ факторов, влияющих на качество жизни и функциональное состояние пациентов с переломовывихами голеностопного сустава. Проблемы стандартизации в здравоохранении. 2011; (3-4): 9-15.
6. Вискарра М. Э. Изучение качества жизни и функционального состояния пациентов с переломовывихами голеностопного сустава с помощью шкал и опросников [диссертация]. М.: Московский государственный медико-стоматологический университет имени М. А. Евдокимова; 2011.
7. Гланц С. Медико-биологическая статистика. М.: Изд-во «Практика»; 1998. 459 c.
8. Головин С. Ю. Словарь практического психолога. Минск: Изд-во «Харвест»; 1998. 800 с.
9. Амирджанова В. Н., Горячев Д. В., Коршунов Н. И.,
18. Schmitt N. Uses and abuses of coefficient alpha. Psychological Assessment. 1996; 8 (4): 350-3.
Ребров А. П., Сороцкая В. Н. Популяционные показатели качества жизни по опроснику SF-36 (результаты многоцентрового исследования качества жизни «МИРАЖ»). Научно-практическая ревматология. 2008; (1): 36-48.
10. Jenkinson C, Coulter A, Wright L. Short form 36 (SF-36) health survey questionnaire: normative data for adults of working age. BMJ. 1993; 306 (6890): 1437-40.
11. Kitaoka HB, Alexander U, Adelaar RS, et al. Clinical rating systems for the ankle hindfoot, midfoot, hallux and lesser toes. Foot Ankle Int.1994. 15: 349-53.
12. SooHoo NF, Shuler M, Fleming LL; American Orthopaedic Foot and Ankle Society. Evaluation of the validity of the AOFAS Clinical Rating Systems by correlation to the SF-36. Foot Ankle Int. 2003 Jan; 24 (1): 50-5.
13. Ware JE, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med Care. 1992 Jun; 30 (6): 473-83.
14. Schuh A, Hausel M. Difficlties in evaluating follow-up outcome in calcaneus fracture managed with plate osteosinthesis. Is there a reliable score? Unfallchirurg. 2000; 103 (4): 295-300. German.
15. Гуревич К. М., Борисова Е. М. Психологическая диагностика. М: Изд-во «УРАО»; 1997. 304 с.
16. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951; 16: 297-334.
17. Kline P. A handbook of test construction (psychology revivals): introduction to psychometric design. Abingdon, UK: Routledge; 2015. 274 p.
18. Schmitt N. Uses and abuses of coefficient alpha. Psychological Assessment. 1996; 8 (4): 350-3.
