Hong, G., Corter, C., Hong, Y., & Pelletier, J. (2011). Differential effects of literacy instruction time and homogeneous grouping in kindergarten: Who will benefit? Who will suffer? To appear in Educational Evaluation and Policy Analysis.
Online Supplementary Materials
Appendix
Marginal Mean Weighting through Stratification for Estimating Differential Effects
Here we explain the procedure of estimating the marginal mean weight for each child through propensity score stratification. This part of the analysis involves five major steps.
Step 1. Estimating propensity scores
After identifying all the observed classlevel and schoollevel pretreatment covariates for each of the six treatments, we created missing indicators to capture different missing patterns among categorical covariates, and then imputed missing data in the continuous covariates via maximum likelihood estimation. We then analyzed a multinomial logistic regression model at the class level to estimate a vector of six propensity scores for each kindergarten class denoted by and corresponding to the six possible treatments. The six estimated propensity scores summed up to 1.0 for every kindergarten class. Each propensity score summarizes the observed pretreatment information including class composition, teacher characteristics, and school characteristics predicting the probability that the kindergarten class would adopt the corresponding treatment. As proven by Rosenbaum and Rubin (1983, 1984), the experimental units and the control units that have the same propensity score for a certain treatment are not systematically different on average in how they would respond to that treatment. Due to the richness of the observed pretreatment information that we used to estimate each propensity score, it seemed reasonable to assume, for example, that the kindergarten classes in the L0 group and the classes not in the L0 group that have the same estimated propensity score are not systematically different in how their students would have achieved in literacy growth on average had they all been assigned to the L0 treatment. This is the socalled “weak ignorability assumption” for evaluating multiple or multivalued treatments (Imbens, 2000).
Step 2. Identifying the analytic sample
Next, we compared each treatment group and the rest of the sample on the distribution of the logit of the estimated propensity of being assigned to that particular treatment. This procedure enabled us to empirically identify and exclude classes that did not have counterfactual information in the data.
Step 3. Stratifying the sample on each propensity score
For each of the six treatments, we then divided the analytic sample into either five or six strata on the basis of the corresponding propensity score. According to Cochran (1968), stratifying a sample into five subclasses typically removes at least 90% of the bias associated with a pretreatment covariate.
4. Computing marginal mean weight
Here we use the L0 group to illustrate. Once we have stratified the whole sample on the basis of the logit of the estimated , within each stratum, the kindergarten classes in the L0 group and the classes not in the L0 group have the same distribution of the logit of . When the weak ignorability assumption holds, we expect that within each stratum the observed mean outcome of children attending the kindergarten classes in the L0 group is an unbiased estimate of the population average potential outcome associated with L0 for all the children in that stratum regardless of the actual treatment assignment of their classes. The same result holds for all three subpopulations of children. This is because child prior ability indicated by child relative standing in class is independent of the classlevel treatments. For example, the observed mean outcome of highability children in the L0 classes within each stratum provides an unbiased estimate of the subpopulation average potential outcome associated with L0 for all the highability children in that stratum. Hence, we can estimate the marginal mean potential outcome associated with L0 for the entire subpopulation of highability children through computing a weighted mean of the observed outcome of highability children in the L0 group.
Let a = 1, 2, 3 denote the lowability,
mediumability, and highability subpopulations, respectively. As derived by Author (2010),
in general, for a child from subpopulation a whose kindergarten class
adopted treatment z and was found in stratum , the marginal mean weight is
MMWS, (a1)
where is the number of subpopulation children in stratum under the stratification on q_{z}; is the number of sampled children from subpopulation a whose classes in stratum actually adopted treatment z; is the proportion of children in subpopulation a who attended kindergarten classes in treatment group z. Intuitively speaking, for each subpopulation of children, we assign weight to those in a certain treatment group such that the weighted composition of this treatment group approximates the pretreatment composition of the entire subpopulation.
Table A1 illustrates the construction of the weighted sample of highability children in the L0 group. For example, because the kindergarten classes in stratum 1 had a relatively low propensity of adopting L0, the highability children attending L0 classes in this stratum had a relatively low representation in the L0 group (15 out of 207) when compared with their representation in the whole sample (400 out of 1,203). The estimated marginal mean weight for these 15 children in stratum 1 was . Hence, the weighted L0 group would have highability children in stratum 1. Highability children attending kindergarten classes in a higher stratum had a relatively higher representation in the L0 group and thus would receive a relatively lower weight. As a result, the composition of highability children in the weighted L0 group would resemble that of the entire analytic sample as if their classes had been assigned at random to L0.
Applying Equation (a1) to the childlevel data, we computed a marginal mean weight for each child as a function of the child’s prior ability level, treatment group membership, and stratum membership. We applied the same strategy to each of the six treatments for each subpopulation of children. Table A2 displays the computed marginal mean weights for the six treatment groups within each subpopulation of children.
Step 5. Checking balance in pretreatment composition among treatment groups in the weighted sample. We examined the difference in each pretreatment covariate among the six weighted treatment groups for children at each prior ability level. Adopting a significance level of .05, we expected to see about 5% of the covariates showing significant differences under the null hypotheses. Indeed, no more than 5% of the hypotheses testing showed a statistically significant difference. We therefore concluded that, under the weak ignorability assumption, all the six treatment groups became comparable for children at the same prior ability level.
Table 3
Weighted Analysis of
Differential Treatment Effects on Literacy Scale Score
Fixed Effects 
Coefficient 
SE 
t 
Literacy Pretest 



High Ability 



Intercept, 
30.44 
0.90 
33.81*** 
L0,

0.60 
1.32 
0.45 
L1,

0.02 
1.12 
0.02 
H0,

0.17 
1.32 
0.13 
H1,

0.49 
1.03 
0.48 
H2,

2.77 
2.01 
1.38 
Medium Ability 



Intercept, 
17.68 
0.38 
47.09*** 
L0,

0.57 
0.50 
1.14 
L1,

0.52 
0.51 
1.02 
H0,

0.69 
0.51 
1.34 
H1,

0.48 
0.45 
1.08 
H2,

0.66 
0.57 
1.16 
Low Ability 



Intercept, 
12.08 
0.58 
20.74*** 
L0,

0.25 
0.81 
0.31 
L1,

0.67 
0.75 
0.89 
H0,

0.51 
0.76 
0.67 
H1,

0.60 
0.67 
0.89 
H2,

0.48 
0.85 
0.56 
General Knowledge () 
0.27 
0.01 
19.18*** 
SES () 
1.01 
0.13 
8.01*** 




Literacy Growth 



High Ability 



Intercept, 
12.53 
1.07 
11.68*** 
L0,

0.36 
1.45 
0.25 
L1,

2.66 
1.51 
1.76 
H0,

0.84 
1.24 
0.67 
H1,

0.62 
1.26 
0.50 




H2,

1.39 
1.53 
0.90 
Medium Ability 



Intercept, 
13.99 
0.51 
27.53*** 
L0,

0.01 
0.66 
0.01 
L1,

0.39 
0.73 
0.54 
H0,

0.73 
0.69 
1.06 
H1,

1.39 
0.61 
2.27* 
H2,

2.26 
0.86 
2.62** 
Low Ability 



Intercept, 
13.43 
0.83 
16.18*** 
L0,

2.05 
0.99 
2.07* 
L1,

1.85 
1.14 
1.63 
H0,

1.12 
1.05 
1.07 
H1,

2.51 
1.01 
2.48* 
H2,

1.60 
1.59 
1.01 
General Knowledge () 
0.13 
0.02 
6.21*** 
SES () 
1.04 
0.20 
5.12*** 




Random Effects 
Variance Component 
df 

Level 2 



Student literacy pretest, 
15.62 
6,966 
18,038.41*** 
Student literacy growth rate, 
38.22 
6,966 
18,891.35*** 
Level 3 



Class literacy pretest 
10.22 
1,697 
5,055.70*** 
Class literacy growth rate, 
11.36 
1,697 
3,403.54*** 
Note: * p <
.05; ** p < .01; *** p < .001
Table 4
Estimated
Endof Year Proficiency Probability in Literacy Subdomains by Instructional
Treatment and Prior Ability
Literacy Subdomains 
Instructional Treatment 

L0 
L1 
L2 
H0 
H1 
H2 

Letter Recognition 






High ability 
1.00 
1.00 
1.00 
1.00 
1.00 
1.00 
Medium ability 
.99 
.99 
.99 
1.00 
1.00 
1.00 
Low ability 
.98 
.97 
.96 
.97 
.98 
.98 
Beginning Sounds 






High ability 
.99 
.99 
.99 
.99 
.99 
.99 
Medium ability 
.84 
.83 
.82 
.86 
.87 
.88 
Low ability 
.59 
.58 
.44 
.51 
.60 
.60 
Ending Sounds 






High ability 
.95 
.96 
.94 
.96 
.94 
.96 
Medium ability 
.48 
.51 
.45 
.51 
.51 
.56 
Low ability 
.19 
.22 
.10 
.16 
.21 
.20 
Sight Words 






High ability 
.47 
.55 
.29 
.50 
.42 
.51 
Medium ability 
.01 
.01 
.01 
.02 
.02 
.02 
Low ability 
.00 
.00 
.00 
.00 
.00 
.00 
Comprehension 






High ability 
.11 
.14 
.08 
.12 
.11 
.13 
Medium ability 
.00 
.01 
.00 
.01 
.01 
.01 
Low ability 
.00 
.00 
.00 
.00 
.00 
.00 
Table 5
Weighted Analysis of Differential Treatment Effects in Literacy Subdomains

Letter Recognition 
Beginning Sounds 
Ending Sounds 
Sight Words 
Words in Context 


Coefficient 
SE 
Coefficient 
SE 
Coefficient 
SE 
Coefficient 
SE 
Coefficient 
SE 
High Ability 










Intercept 
8.40*** 
0.30 
4.52*** 
0.274 
2.70*** 
0.26 
0.88** 
0.33 
2.38*** 
0.23 
L0 
0.25 
0.43 
0.27 
0.36 
0.28 
0.35 
0.75 
0.45 
0.40 
0.31 
L1 
0.26 
0.41 
0.58 
0.38 
0.58 
0.34 
1.08** 
0.41 
0.63* 
0.29 
H0 
0.11 
0.36 
0.34 
0.33 
0.42 
0.32 
0.89* 
0.36 
0.53* 
0.25 
H1 
0.22 
0.36 
0.06 
0.32 
0.02 
0.30 
0.57 
0.38 
0.40 
0.27 
H2 
0.27 
0.48 
0.72* 
0.35 
0.46 
0.34 
0.92* 
0.42 
0.57* 
0.29 
c^{2} (df) 
4.82 (5) 
10.90 (5) 
10.35 (5) 
14.40* (5) 
7.12 (5) 








Medium Ability 










Intercept 
5.04*** 
0.15 
1.49*** 
0.14 
0.20 
0.12 
4.58*** 
0.20 
5.42*** 
0.19 
L0 
0.12 
0.20 
0.16 
0.17 
0.12 
0.16 
0.35 
0.25 
0.11 
0.24 
L1 
0.25 
0.21 
0.12 
0.19 
0.23 
0.16 
0.32 
0.27 
0.15 
0.25 
H0 
0.53** 
0.21 
0.31 
0.18 
0.24 
0.16 
0.51* 
0.25 
0.38 
0.24 
H1 
0.61** 
0.19 
0.38* 
0.17 
0.23 
0.15 
0.62* 
0.24 
0.49* 
0.22 
H2 
0.71** 
0.27 
0.53* 
0.22 
0.43* 
0.17 
0.75** 
0.26 
0.47 
0.24 
c^{2} (df) 
32.80*** (5) 
16.88** (5) 
9.31 (5) 
17.98** (5) 
18.29** (5) 








Low Ability 










Intercept 
3.10*** 
0.22 
0.25 
0.24 
2.18*** 
0.21 
7.57*** 
0.55 
7.69*** 
0.29 
L0 
0.84** 
0.30 
0.60* 
0.28 
0.73** 
0.28 
1.62*** 
0.42 
1.40*** 
0.36 
L1 
0.54 
0.28 
0.57 
0.30 
0.89** 
0.28 
1.45*** 
0.41 
1.17** 
0.36 
H0 
0.50 
0.28 
0.31 
0.30 
0.53 
0.31 
1.21** 
0.43 
0.91* 
0.37 
H1 
0.71** 
0.26 
0.65* 
0.27 
0.86*** 
0.26 
1.57*** 
0.37 
1.19*** 
0.34 
H2 
0.95* 
0.39 
0.65 
0.37 
0.82* 
0.34 
1.41** 
0.51 
0.88* 
0.41 
c^{2} (df) 
15.51** (5) 
12.26* (5) 
19.53** (5) 
34.35*** (5) 
32.47*** (5) 
Note: * p < .05; ** p < .01; *** p < .001
Table 6
Weighted Analysis of
Differential Treatment Effects on General Learning Behaviors
Fixed Effects 
Coefficient 
SE 
t 
High Ability 



Intercept 
3.47 
0.03 
113.07*** 
L0 
0.03 
0.05 
0.63 
H0 
0.02 
0.04 
0.37 
H1 
0.04 
0.04 
0.96 
H2 
0.03 
0.06 
0.49 



Medium Ability 



Intercept 
3.12 
0.02 
156.69*** 
L0 
0.06 
0.03 
1.82 
H0 
0.02 
0.03 
0.85 
H1 
0.02 
0.03 
0.86 
H2 
0.08 
0.04 
2.21* 




Low Ability 



Intercept 
2.74 
0.03 
79.43*** 
L0 
0.10 
0.05 
1.92 
H0 
0.06 
0.06 
1.11 
H1 
0.11 
0.04 
2.51* 
H2 
0.16 
0.07 
2.46* 
Note: * p < .05; ** p < .01; *** p < .001
Table A1
Marginal Mean
Weight for HighAbility Children Attending Kindergarten Classes with Low
Reading Time and No Grouping (L0)
Stratum 
Unweighted Sample

MMWS 
Weighted Sample 

L0 = 1 
L0 = 0 
Total 
L0 = 1 

1 
15 
385 
400 
4.59 
68.85 
2 
19 
216 
235 
2.13 
40.47 
3 
60 
241 
301 
0.86 
51.60 
4 
78 
145 
223 
0.49 
38.22 
5 
35 
9 
44 
0.22 
7.7 
Total 
207 
996 
1,203 
 
207 
Table A2
Marginal Mean
Weight for High, Medium, and LowAbility Children
High Ability 

Stratum 
L0 
L1 
L2 
H0 
H1 
H2 
1 
4.59 
4.02 
2.45 
4.05 
2.53 
3.17 
2 
2.13 
1.52 
0.98 
1.96 
2.27 
1.35 
3 
0.86 
1.09 
0.96 
0.82 
0.95 
0.48 
4 
0.49 
0.47 
0.29 
0.53 
0.81 
0.31 
5 
0.22 
0.27 
0.23 
0.39 
0.60 
0.13 
6 
 
 
 
0.42 
0.40 
 
Weighted n 
207 
233 
136 
221 
300 
106 
Medium Ability 

Stratum 
L0 
L1 
L2 
H0 
H1 
H2 
1 
3.71 
3.27 
2.96 
3.82 
3.39 
3.09 
2 
1.65 
1.26 
1.05 
1.68 
1.73 
1.39 
3 
0.93 
0.95 
0.70 
0.86 
1.23 
0.45 
4 
0.51 
0.44 
0.31 
0.48 
0.86 
0.29 
5 
0.24 
0.30 
0.22 
0.42 
0.59 
0.14 
6 
 
 
 
0.29 
0.42 
 
Weighted n 
981 
934 
626 
916 
1,376 
493 
Low Ability 

Stratum 
L0 
L1 
L2 
H0 
H1 
H2 
1 
2.95 
5.08 
2.48 
2.98 
3.60 
3.34 
2 
2.00 
1.32 
1.10 
1.89 
1.76 
0.87 
3 
1.06 
0.85 
0.81 
0.73 
1.26 
0.56 
4 
0.56 
0.43 
0.31 
0.61 
0.73 
0.32 
5 
0.23 
0.27 
0.20 
0.52 
0.54 
0.15 
6 
 
 
 
0.37 
0.38 
 
Weighted n 
414 
378 
213 
383 
538 
213 