Does giving class-time improve SET response rates?

Last month we surveyed all faculty who used Blue for online student evaluations of teaching (SET) during the fall 2019 semester regarding their use of class time for completing their online SETs. Because results from published studies at other institutions suggest a positive relationship between providing in-class time and response rates, we sought to quantify the magnitude of the relationship at UMaine, if one exists.  We received responses for 168 unique course sections (thank you to all those who responded!). We limited our analysis to in-person sections with 10 or more students enrolled, in an attempt to avoid extreme response rates that are more likely in low-enrollment courses. After applying the course and enrollment filters we were left with 142 unique course sections. The percentage of these sections where in-class time was given (“yes”) or was not given (“no”) to complete online SETs are indicated in Figure 1.

Figure 1: Proportion of course sections where instructors gave students in-class time to do their online SETs (yes), versus those that were not (no).

We compared the section response rates of these two groups. The distribution of response rates within the groups are shown in Figure 2. The histograms suggest that response rates were better if students were given in-class time (albeit with a lot of overlap).

Figure 2: Distribution of response rates for course sections where students were given in-class time to do their online SETs (Yes) and those that were not (No).

The mean response rates for the two groups are provided in Table 1.  We found the mean difference between the two groups to be 15.1 percentage points. This difference is statistically significant (Welch’s T-Test(83.3)=4.369, p<0.000) and indicates a 15.1 percentage point improvement in response rates for class sections that provided students in-class time to do their SETs.  It should be noted that our survey did not include questions about other strategies being used instead of or in addition to providing class time, and these results only apply to the select subset of faculty who responded to the survey. That said, these results do suggest that providing class time can positively impact response rates.  

Table 1: Mean response rates for course sections where students were and were not given in-class time to do their online SETs.

Our results are in agreement with published literature that suggests giving in-class time increases response rates, but we acknowledge that additional study is needed to understand what other incentives or strategies are used and how they could impact response rates.  Toward that end, we asked the faculty of those courses that did not give in-class time, but also had response rates at or above 70%, to describe their approaches. A summary of strategies and incentives used by those faculty who responded are provided in Table 2. Using such strategies as reminding students, conducting a mid-term feedback survey, and showing students how the feedback is used, in addition to giving in-class time for SETs, might result in even greater improvements in response rates.   

Another, somewhat unexpected, result was the number of low response rates for sections where in-class time was given for online evaluations to be filled out. We do not have data to address what might have been happening in these situations, but it does imply that simply giving in-class time might not always be enough to result in an adequate number of responses. We recommend that faculty consider using the strategies described in Table 2 regardless of the way SETs are being administered. We provide similar recommendations on the OIRA website.

Table 2: Strategies and incentives used by faculty that had high SET response rates but did not give in-class time for the evaluations. Note: Please check with your department before giving points as an incentive.