ABSTRACT

CONCLUSION:

The scale was considered a valid and reliable instrument. However, it is recommended to test it in other groups in order to increase its reliability criteria.

Cronbach’s alpha coefficients were calculated for the whole scale and its factors (total scale α=0.68, Factor 1 “perceived severity” α=0.81, Factor 2. “perceived barriers” α=0.782 and Factor 3 “perceived benefits” α=0.70). CFA was also evaluated with a different sample (n=120). According to these results; the Degrees of Freedom (DF) value was found to be 101 (p<0.001), the Root mean square error approximate value (RMSEA) was 0.08, the Goodness of Fit Index (GFI) value was 0.89; the Normalized Fit Index (NFI) value was 0.62 and the Comparative Fit Index (CFI) value was 0.67.

RESULTS:

Content and construct validity of the items were evaluated (n=477). The 16-item scale had a KMO test result of 0.808 and a Bartlett’s test result of 2,308,179. In the determination of the invariance of the scale with respect to time (n=85), there was no statistical difference (p>0.05).

MATERIALS AND METHODS:

The COVID-19 Vaccine Knowledge and Attitude Scale was produced via two different studies (a cross-sectional study and a methodological study). The first study was the cross-sectional (n=396) study. This study’s results were used for the first draft candidate scale, 50 items, with the literature. In this study, it was aimed to evaluate the validity and reliability of the COVID-19 Vaccine Knowledge and Attitude Scale. The study population consisted of individuals who were over the age of 18 years living in Northern Cyprus, who could speak Turkish, used social media platforms, and had a smart phone or a computer. In this study,firstly, the researchers evaluated the first candidate scale (n=50 items) and then this was reduced to 25 items. According to the literature, the study sample size (25x10) should be at least 250 participants. In this study, sampling selection was achieved via a Convenience Sampling method and 477 participants who met the study criteria and agreed to participate in this study as a volunteer took part. Also, in this study, confirmatory factor analysis (CFA)was performed with a different sample (n=120).The data was collected via Google Form (age, gender, eight socio-demographic questions and the COVID-19 Vaccine Knowledge and Attitude Scale) on internet platforms. SPSS were used for the statistical evaluation of the study. The Content Validity Index method was used for the content validity of the scale. Kaiser–Meyer–Olkin (KMO) and Bartlett’s tests were applied to evaluate the sampling adequacy and suitability for factor analysis. Pearson correlation analysis was used for item analysis and the Cronbach alpha reliability coefficient was used to test internal consistency. Subsequently, approximately 2 weeks later, the scale was reapplied to the participants (n=85) to test-retest reliability using the paired dependent sample t-test.No statistically significant difference was found(p>0.05).The results are shown as mean ± standard deviation, number(n) and percentage (%). Aconfidence interval (CI) of 95% and p<0.05 were accepted as statistical significance. The IBM SPSS V22 (IBM Corp., Armonk, NY, USA) Amosprogram was used for CFA.

BACKGROUND/AIMS:

The coronavirus disease-2019 (COVID-19) virus has spread to many countries in a short time since its emergence in December 2019 and it has been declared as a pandemic. It is important to wear a mask and comply with physical distance rules to protect against COVID-19. However, the COVID-19 vaccine is important for breaking the infection chain. This study was a methodological type of validity and reliability study on the COVID-19 Vaccine Knowledge and Attitude Scale conducted in the Northern Cyprus between October 2020 and January 2021.

INTRODUCTION

The coronavirus disease-2019 (COVID-19) virus spread very rapidly and turned into a long-term pandemic. The rate of morbidity and mortality (3.4%) of the virus causes fear and panic in society, causes economic losses, and psychological and social problems.^1,2 The virus causes serious health problems and death in all age groups, but especially for those who are elderly, have chronic diseases, or have immune system problems. The virus also creates a heavy burden on the health system.^3,4

There is no specific treatment for the virus to date, so prevention has become even more essential.² It is aimed to gain both individual and social immunity with the vaccines developed.⁵ In the world, seven different vaccines have been developed so far.⁶ Their level of protection ranges from 40% to 90% and it is predicted that they will protect for at least six months.⁷ However, there are various problems and concerns about obtaining the vaccines, their application, their side effects, and the preventiveness of these vaccines.⁸ In some sections of society, these concerns remain even for vaccines developed many years ago (measles, polio, rubella, etc.), and this leads to “anti-vaccination”or “vaccine-hesitancy” attitudes.^9,10

Health belief is defined as the individuals’ beliefs and attitudes towards health behaviors.¹¹ If a person thinks that a disease has fatal or dangerous health consequences (perceived severity) and believes that the current method of protection/treatment will protect/cure them (perceivedbenefits), they will seek health care. However, the same person may also experience some difficulties (perceivedbarriers) in accessing the treatment.¹² Valid, reliable measuring tools are needed to determine the community’s knowledge regarding the vaccine, and the associatedperceived barriers, perceived severity and perceived benefits.

This study was conducted as a methodological study to determine the validity of the newly developed “COVID-19 Vaccine Knowledge and Attitude Scale”.

MATERIALS AND METHODS

This study was a methodological type of validity and reliability study conducted in Northern Cyprus between October 2020 and January 2021.

Population

The study population consisted of individuals over the age of 18 years living in the Northern Cyprus, who could speak Turkish, used social media platforms, and had a smart phone or a computer.

Sampling

In this study, the first candidate scale with 50 items was produced from the first study which was a cross-sectional (n=396) study¹³ and subsequently, it was reduced to 25 items. According to the literature,^14-18 the sample size should be at least 5–10 times of the number of scale items. In this study, the second candidate scale was 25 items, and so the study sample size was calculated to be at least 250 participants.

In this study, sample selection was achieved with a Convenience Sampling Method and 477 participant who met the study criteria and agreed to take part as volunteers were enrolled in this study.  Subsequently, approximately 2 weeks later, the scale was applied to 85 participants again to test-retest its reliability. The confirmatory factor analysis (CFA) was performed with a different sample of 120 (Figure 1).

Figure 1

Data Collection

The data was collected via Google Form (age, gender, eight socio-demographic questions and the COVID-19 Vaccine Knowledge and Attitude Scale) on internet platforms.

Ethical Aspects of the Study

Ethics committee approval for this study was obtained from Near East University Scientific Research Ethics Committee (decision number: 2020/85, date: 26.11.2020), and written informed consent was obtained from all participants before the study.

Inclusion/Exclusion Criteria

Those who could read and understand Turkish and who volunteered to participate in this study were included, and those who could not access the internet via their computer or smart phone were excluded from this study.

Limitations of the Study

The research data was collected with the participants’ self-declaration and applies only to this sample group. It cannot be generalized to other groups.

Data Collection Tools

Socio-demographic Questionnaire

It consists of eight questions that investigate the age, gender, and educational status of the individuals, and the COVID-19 transmission status and chronic disease history of themselves and their family.

COVID-19 Vaccine Knowledge and Attitude Scale

COVID-19 Vaccine Knowledge and Attitude Scale was developed by the researchers. The final version consists of16 items inthreesub-scales. All scale items were calculated positively and the sub-scales can used individually. There is no cut-off point of the scale. High scores obtained indicate that the participant has a high level of “severity” (five items= 1;2;3;4;5),“barriers” (seven items= 6;7;8;9;10;11;12), and “benefits” (four items= 13;14;15;16) perception regarding COVID-19 Vaccine Knowledge and Attitude. The scale was a Likert type scale. Items were evaluated as1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, and 5=Strongly Agree. It takes about 10 minutes to complete the scale.

Statistical Analysis

Statistical analyses were performed using the Statistical Package for Social Sciences software version 21.0 (IBM SPSS Corp., Armonk, NY, USA).While testing the reliability of the scale, the Content Validity Index (CVI) was used to evaluate the Content Validity, and exploratory factor analysis was used to evaluate the Construct Validity. Kaiser–Meyer–Olkin (KMO) and Bartlett’s tests were applied to evaluate the sampling adequacy and suitability for factor analysis. Pearson correlation analysis was used for item analysis and the Cronbach alpha reliability coefficient was used to test internal consistency. Test-retest reliability was evaluated via the paired dependent t-test.The results are shown as mean ± standard deviation, number (n) and percentage (%). Confidence interval (CI) 95% and p<0.05 were accepted as statistical significance. The Amos SPSS V22 (IBM Corp, Armonk, NY, USA) was used for CFA.

RESULTS

The Study Participants Sociodemographic Characteristics

The average age of the participants was 23±39.3 years, 67.1% (n=320) were women, and 81.8% (n=390) were university graduates. It was determined that 27.9% of the participants (n=133) had a family member infected with the COVID-19 virus, and 5% (n=24) lost one of their family members due to COVID-19. In addition, it was determined that 9% (n=43) of the participants had been infected with COVID-19.

COVID-19 Vaccine Knowledge and Attitude Scale Validity Assessment

Content Validity Index Analysis

The first draft of this scale (50-item scale) was prepared on the basis of a health belief model, according to the results of a literature review and before the Cross-Sectional type study (n=396) results. Following this, the first draft scale items (50-item scale) were evaluated independently by the researchers, and they reached a consensus on the second draft scale (25-item draft scale).

The second candidate scale (25 items) and its items were evaluated by expert researchers (n=5) according to the CVI. CVI evaluation for each item was as follows: 1 = the item is not relevant, 2 = the item is not relevant and a major change is required to become relevant, 3 = the item is relevant but a minor change is required, 4 = the item is very relevant. Those itemsassessed as 3 or 4 were considered sufficient in terms of Item Contend Validity and remained in the draft scale. At this stage, five items were removed from the scale; so that the third draft scale (20-item scale) was created. The evaluation of the experts for this scale was found to be CVI= 80%–90%.

Construct Validity Analysis

According to the results of the first analysis (n=477), four items with Eigenvalues <1 were removed from the draft scale. Statistical evaluation was continued with the remaining fourth draft scale (16-item scale).

In the Table 1, KMO and Bartlett tests were used for exploratory factor analysis on the data obtained from the fourth draft scale (16-item scale). KMO was found to be 0.808, and the Bartlett test result was found to be 2308.1 (p<0.001) (Table 1).

Table 1

The Varimax orthogonal rotation method was used to rotate the factor loadings matrix and explain the factor variances with fewer variables in a maximum way. An Eigenvalue of 1 was accepted to determine the number of factor items. A Scree Plot diagram was used to determine the number of factors. According to the Scree Plot diagram, the last point before falling below a nEigenvalue of 1determines the number of factors.¹⁹

It was determined that among the scale items included in the analysis, those with an Eigenvalue >1 explained 51.55% of the total variance. The variance ratio explained by the first factor with an Eigenvalue of 4.00 was 25.00%; the variance ratio explained by the second factor with an Eigenvalue of 2.64 was 16.55%; the variance ratio explained by the third factor with an Eigenvalue of 1.80 was 11.28%. The total variance ratio explained in the scale was found to be 51.55%.

When the Scree Plot graph was examined, it was determined that a sharp decline continued until the fourth point, and after the fourth point, the slope of the line became horizontal (Figure 2). When the dot intervals up to the fourth point were counted, it was determined that it was three, and this suggests that a useful model for these data may have three factors. Accordingly, the first factor is perceived severity, the second factor is perceived barriers, and the third factor is perceived benefits (Table 1).

Figure 2

Table 1

COVID-19 Vaccine Knowledge and Attitude Scale Reliability Analysis

Reliability is the degree to which the items of the measurement tool are consistent with each other, the degree to which their results are free of random errors.¹¹ Internal Consistency Cronbach Alpha, Spearman and Guttman Coefficients, Item analysis, and test-retest confidence analyses were used to determine the reliability of the scale developed.

Internal Consistency Reliability (Cronbach Alpha) Analysis

Cronbach’s alpha coefficient, one of the methods of testing the internal consistency reliability in Likert-type scales, was calculated for the whole scale and its sub-scales (Table 2). The Scale Total Cronbach alpha value was a = 0.68, Factor 1/perceived severity Cronbach Alpha value was a=0.81, Factor 2/perceived barriers Cronbach Alpha value was a=0.78 and Factor 3/perceived benefits Cronbach Alpha value was determined to be a=0.70. A Cronbach alpha item deleted test was performed, but it was determined that the Cronbach alpha value did not increase if any items were removed.

Table 2

Determination of Spearman-Brown and Guttman Values Analysis

As can be seen in Table 3, the split-half reliability coefficients obtained by dividing the scale items into two equivalent halves were calculated. Accordingly, the Spearman value (Equal-length Spearman–Brown) was found to be S=0.349, and the Guttman value (Guttman split-half) was G=0.347.

Table 3

Item Correlation Analysis

Correlation Analysis

In Table 4, “Pearson-moment correlation analysis” was performed to determine the relationship between the scale score and factor scores. A correlation was found between Factor 1-perceivedseverityand Factor 2-perceivedbarriers (r=0.310), Factor 3-perceived benefits (r=1.000), and the scale total score (r=0.816) (p<0.001). A correlation was found between Factor 2-PerceivedBarriersand the scale total score (0.782), Factor 1-Perceived Severity (r=-0.105), Factor 3-Perceived Benefits (r=1.000) (p<0.001). A correlation was found between Factor 3-Perceived Benefits and the overall score of the scale (r=0.697), Factor 1-PerceivedSeverity (r=1.00), and Factor 2-PerceivedBarriers (r=-0.108) (p<0.001).

Table 4

Item Loads of Factors Analysis

In cases where the Eigenvalue was below 0.40, the relevant item was removed from the scale. The factor loads are shown in Table 5.

Table 5

The factor loads of the items in the first factor vary between 0.522 and 0.807, the factor loads of the substances in the second factor vary between 0.459 and 0.713, and the factor loads of the substances in the third factor vary between 0.472 and 0.721.

Test-Retest Analysis

In Table 6, to determine the relationship between the overall and sub-scales of the scale, the test was applied to a group of the study participants (n=85) again. Subsequently, the data was analyzed with the paired dependent sample t-test. There was no statistically significant difference between either the total mean point of the scale (pre-test=52.28±4.30, post-test=45.95±5.08) and the mean points of Factors 1, 2 and 3 between the pre-test and post-test (p>0.05).

Table 6

Confirmatory Factor Analysis

The CFA performed in the AMOS SPSS version 22 (IBM Corp, Armonk, NY, USA) statistical program examined the relationship between the different participants (n=120) and the factors and co-variance values (Figure 2).

Figure 2

The structure examined contains five items for severity sub-scale, seven items for barrierssub-scale, and four items for benefits sub-scale. The results of the CFA are shown in Figure 3. According to these results, the Degrees of Freedom (DF) value was found to be 101 (p<0.001),the root mean square error of approximation (RMSEA) value was found to be 0.08, the Goodness of Fit Indices value (GFI) was 0.89, the Normed Fit Index (NFI) value was 0.62, and the Comparative Fit Index value (CFI) was 0.67.

Figure 3

DISCUSSION

The COVID-19 virus affected billions of lives shortly after its emergence and caused many deaths worldwide. There have been more than three million deaths from COVID-19 worldwide to date, and the number of deaths due to the virus in Northern Cyprus has exceeded 30.^20,21 In this study, 5% of the participants had lost a family member due to COVID-19, and approximately 10% had been infected with COVID-19 (Table 1). The data in this study are similar to the literature.

Table 1

In the literature, it is stated that draft scale questions in scale development studies be created by scanning the literature or by qualitative interviews.^22-24 In this study, the literature was reviewed and the first draft scale (50-item draft scale) was created using cross-sectional study data conducted with a different sample group in the first step of the research as part of the scale development study.²³ In this study, firstly, the researchers evaluated the first candidate scale (50 items) and then the second draft candidate scale was created by reducing the number of items to 25.

Validity is the conformity of the measurement tool to the feature required to be measured and the degree of measurement of the feature it intends to measure.²⁵ A developed measurement tool is expected to meet validity. Validity is evaluated as content validity and construct validity.²⁶ Testing the content validity of a scale is carried out to determine whether the newly developed scale measures the concept that it is intended to measure and whether it contains unrelated concepts.²⁷ The scale was presented to experts in this field in order to eliminate items that are not related to the condition to be measured. The scale was edited in line with the comments and assessments of these experts. In the literature, it is stated that the number of experts to be consulted to test the content validity can vary between 5 and 40.^25,28-30 The purpose of the validity test is to evaluate the draft scale items by determining whether the draft items represent the behaviors to be measured. This second draft scale (25-item scale) was presented to experts (n=5) to test its content validity. As a result of the evaluations of the experts, five items scoring below three were removed from the scale, and it was determined that the CVI value of the scale was above 80%. The research was then carried out with the third draft scale (20items).

The construct validity determines to what extent the items in the scale accurately measure what it is intended to measure.³¹ The test of construct validity is done by using the factor analysis method and scoring the answers given to the items in the measurement tool. As a result of this analysis, items with low factor loads are excluded from the scale. Factor analysis is continued until an appropriate result is reached, which includes a sufficient number of items to measure the desired area.^32,33 With the KMO test and Bartlett’s tests, data on the scale are tested to ensure their suitability for factor analysis. The lower limit of KMO testing is 0.50, and factor analysis should not be continued in case of the result being lower than this value. The KMO result is expected to be above 0.70 and close to 1 to perform a good factor analysis.^20,25,34 In this study, the Kalmogrow–Simirnow test was used to determine the normality distribution of the data. Accordingly, whether or not the sample size was sufficient to develop a scale was tested with the KMO test and Bartlett’s test among the exploratory tests. As a result of these tests, it was determined that the items were sufficient to develop the scale. The KMO coefficient of the study (0.808) and Bartlett’s test (2308.179) were found to be sufficient for analysis (Table 1).

Table 1

Among principal component analysis, the Varimax Rotation method is one of the most used methods to determine the factor structure of a scale.³² In this study, Varimax rotation was used and the factor structure of the scale was determined. In the literature, the value of factor load is used to explain the relationship between items and the factor and when deciding on the substances to be included in the scale. The lower limit specified for the factor load value is 0.30, and load values between 0.30 and 0.59 are considered to be medium and values 0.60 and above are considered high. It is recommended that values above 0.40 should be taken as the factor load value.^26,32 In this study, four items with a factor load below 0.40 were excluded from the scale. Thus, the number of items on the scale decreased to n=16 items (the fourth draft of the scale).

As a result of factor analysis, the higher variance rates mean a stronger factor structure. However, it is not possible to reach a high variance rate in many areas; variance rates varying between 40% and 60% are considered ideal.³¹ In this study, the total variance explained after factor analysis was determined to be 51.55%. This scale is within acceptable limits in terms of the exploratory factor load value.

The Scree Plot graph is used to determine the number of factors.³⁴ In this graph, the vertical axis expresses Eigenvalues and the horizontal axis expresses factors. Points giving a steep slope in the graph are included in the study. Points giving a superficial, flat slope are not included in the study. A horizontal line is drawn from the point where the graph shows a horizontal slope, and the distances between the points above this line are accepted as a scale.³⁵ In the literature, it is recommended to take the opinions and comments of experts in the naming of the sub-scales.³⁶ In this study, according to the factor analysis, the three factors were named as the perceived severity sub-scale, the perceived barriers sub-scale and the perceived benefits sub-scale.

The consistency of all the items in the measurement tool and the degree to which the measurement results are free from random errors is called Reliability. A test accepted as valid should also be reliable.^25,26 Internal consistency, split-half, test-retest, and factor analysis methods are used in the reliability analysis of a developed scale.

Internal consistency is determined by calculating the Cronbach alpha coefficient.³⁷ Internal consistency is the reliability method that indicates the items included in the measurement tool can measure the variable that is desired to be measured. A high Cronbach alpha coefficient means that the items in that scale are consistent with each other.³⁸ In the literature, it is reported that a scale is reliable if the Cronbach alpha coefficient is in the range of 0.60–0.70, while values between 0.70–1.00 are considered as high reliability.²⁶ In this study, the total scale the Cronbach’s alpha value of the scale was found to be a=0.68. This value was considered as meaning the reliability of the scale was within acceptable limits. Removing any item from the scale while evaluating Cronbach alpha may increase the Cronbach alpha value.³⁹ However, in this study, it was determined that the Cronbach alpha value did not increase with the deletion of any item. For the Cronbach alpha value of the sub-scales, the perceived severity sub-scale with a=0.81, the perceived barriers sub-scale with a=0.78, and the perceived benefits sub-scale with a=0.70, were determined, and it was considered a reliable measurement tool.

In determining the internal consistency of the scale, in addition to the Cronbach alpha coefficient, the split-half method is used and the Guttman and Spearman–Brown reliability coefficients are calculated.⁴⁰ When calculating the internal reliability coefficient using the division in half method, the coefficient value should be at least 0.70.²⁹ In this study, the Spearman–Brown value of the scale was calculated to be S=0.349 and the Guttman value as G=0.347. The Spearman–Brown and Guttman values were found to be low in this study.

Testing consistency-against-time is another scale of reliability. The scale is applied to part of the same sampling group after 2 to 4 weeks, and the mean scores between the two measurements are compared.^25,26 In this study, the relationship between the overall and sub-scale of the scale was evaluated in the test-retest method with a group of the study participants (n=85), and there was no statistically significant difference between them (p>0.05). This result was evaluated as a consistent measurement of the scale against time.

Item analysis is carried out to test whether the items in the whole or sub-scales of the measurement tool are significantly included in the whole or sub-scales of the scale. In item analysis, the variance of each scale item and the variance of the total scale score are compared with Pearson-moment correlation analysis, and the relationship between them is examined.⁴⁰ If the items of the scale are of equal weight and in the form of independent units, it is expected that the correlation coefficient between each item and the total value will be high, and the item-total correlation results will also show statistical significance.^25,34,20 In this study, there was a statistically significant relationship between all items in the scale and the total score according to the results of Pearson-moments correlation analysis calculated to determine item-total correlations (p<0.05). As a result of the Pearson-moment correlation analysis performed for the item-total correlations of the sub-scales, a significant correlation was found between the perceived severity, perceived barriers,perceived benefits items, and their sub-scales total score (p<0.05).These results indicate that the items in the scale are distinctive in terms of the properties they measure.

CFA is a frequently used analysis method in developing a new measurement tool, the evaluation of the psychometric properties of the measuring instrument, examining the effectiveness of the method, determining whether the validity of the measurement tool created varies according to time, population and groups, and determining the correlation between measurement errors.⁴¹ CFAof the scale developed in this study was carried out with a different sample (n=120) (Figure 3).

Figure 3

In the CFA, if the RMSEA value is less than 0.08 and the Goodness of Fit Indices value (GFI) is above 0.90, it indicates that the scale has a “good” fit.⁴² If the Normed Fit Index (NFI) value is above 0.90 and the Comparative Fit Index value (CFI) is equal to 0.95, it means that the scale has a “perfect” fit.³³ According to this study’s results, the degrees of freedom (DF) value was found to be 101. RMSEA value was found to be 0.08, the Goodness of Fit Indices value (GFI) was 0.89, the Normed Fit Index (NFI) value was 0.62, and the Comparative Fit Index value (CFI) was 0.67. In line with these data, it was found that the CFA of the scale was within the reference values given.

CONCLUSION

The COVID-19 Vaccine Knowledge and Attitude Scale, which was analyzed for validity and confidence in this study, is a valid and reliable tool. However, it is recommended to test it in other groups in order to increase its reliability criteria.

ACKNOWLEDGEMENTS

The authors would like to thank all the lecturers who provided expert opinion during the scale development phase during this research.

MAIN POINTS

● This is the first scale regarding COVID-19 Vaccine Knowledge and Attitudes in the literature.

● This scale may help to plan health education in the future regarding a community’ sperceived fears, barriers and benefits with respect to the COVID-19 vaccine.

● As this scale is validate and reliable, it might be implemented in different studies in a standardized way.

ETHICS

Ethics Committee Approval: Ethics committee approval for this study was obtained from Near East University Scientific Research Ethics Committee (decision number: 2020/85, date: 26.11.2020).

Informed Consent: Written informed consent was obtained from all participants before the study.

Peer-review: Externally peer-reviewed.

Authorship Contributions

Concept: K.E., H.B., Design: K.E., H.B., Data Collection and/or Processing: K.E., H.B., Analysis and/or Interpretation: K.E., H.B., Literature Search: K.E., H.B., Writing: K.E., H.B., Critical Review:K.E., H.B.

DISCLOSURES

Conflict of Interest: No conflict of interest was declared by the authors.

Financial Disclosure: The author declared that this study had received no financial support.