Hong, G., Corter, C., Hong, Y., & Pelletier, J. (2011). Differential effects of literacy instruction time and homogeneous grouping in kindergarten: Who will benefit? Who will suffer? To appear in Educational Evaluation and Policy Analysis.
Online Supplementary Materials
Appendix
Marginal Mean Weighting through Stratification for Estimating Differential Effects
Here we explain the procedure of estimating the marginal mean weight for each child through propensity score stratification. This part of the analysis involves five major steps.
Step 1. Estimating propensity scores
After identifying
all the observed class-level and school-level pretreatment covariates for each
of the six treatments, we created missing indicators to capture different
missing patterns among categorical covariates, and then imputed missing data in
the continuous covariates via maximum likelihood estimation. We then analyzed a
multinomial logistic regression model at the class level to estimate a vector
of six propensity scores for each kindergarten class denoted by 
and 
corresponding
to the six possible treatments. The six estimated propensity scores summed up
to 1.0 for every kindergarten class. Each propensity score summarizes the
observed pretreatment information including class composition, teacher
characteristics, and school characteristics predicting the probability that the
kindergarten class would adopt the corresponding treatment. As proven by
Rosenbaum and Rubin (1983, 1984), the experimental units and the control units
that have the same propensity score for a certain treatment are not
systematically different on average in how they would respond to that
treatment. Due to the richness of the observed pretreatment information that we
used to estimate each propensity score, it seemed reasonable to assume, for
example, that the kindergarten classes in the L0 group and the classes
not in the L0 group that have the same estimated propensity score 
are not
systematically different in how their students would have achieved in literacy
growth on average had they all been assigned to the L0 treatment. This
is the so-called “weak ignorability assumption” for evaluating multiple or
multi-valued treatments (Imbens, 2000).
Step 2. Identifying the analytic sample
Next, we compared each treatment group and the rest of the sample on the distribution of the logit of the estimated propensity of being assigned to that particular treatment. This procedure enabled us to empirically identify and exclude classes that did not have counterfactual information in the data.
Step 3. Stratifying the sample on each propensity score
For each of the six treatments, we then divided the analytic sample into either five or six strata on the basis of the corresponding propensity score. According to Cochran (1968), stratifying a sample into five subclasses typically removes at least 90% of the bias associated with a pretreatment covariate.
4. Computing marginal mean weight
Here we use the L0
group to illustrate. Once we have stratified the whole sample on the basis of
the logit of the estimated , within each stratum, the kindergarten classes in the
L0 group and the classes not in the L0 group have the same
distribution of the logit of . When the weak ignorability assumption holds, we expect that within
each stratum the
observed mean outcome of children attending the kindergarten classes in the L0
group is an unbiased estimate of the population average potential outcome
associated with L0 for all the children in that stratum regardless of
the actual treatment assignment of their classes. The same result holds for all
three subpopulations of children. This is because child prior ability indicated
by child relative standing in class is independent of the class-level
treatments. For example, the observed mean outcome of high-ability children in
the L0 classes within each stratum provides an unbiased estimate of the
subpopulation average potential outcome associated with L0 for all the
high-ability children in that stratum. Hence, we can estimate the marginal mean
potential outcome associated with L0 for the entire subpopulation of
high-ability children through computing a weighted mean of the observed outcome of high-ability
children in the L0 group.
Let a = 1, 2, 3 denote the low-ability,
medium-ability, and high-ability subpopulations, respectively. As derived by Author (2010),
in general, for a child from subpopulation a whose kindergarten class
adopted treatment z and was found in stratum 
, the marginal mean weight is
MMW-S
, (a1)
where 
is the number
of subpopulation 
children in stratum 
under the stratification on qz; 
is the number
of sampled children from subpopulation a whose classes in stratum actually
adopted treatment z; 
is the
proportion of children in subpopulation a who attended kindergarten classes
in treatment group z. Intuitively speaking, for each subpopulation of
children, we assign weight to those in a certain treatment group such that the
weighted composition of this treatment group approximates the pretreatment
composition of the entire subpopulation.
Table A1
illustrates the construction of the weighted sample of high-ability children in
the L0 group. For example, because the kindergarten
classes in stratum 1 had a relatively low propensity of adopting L0, the
high-ability children attending L0 classes in this stratum had a
relatively low representation in the L0 group
(15 out of 207) when compared with their representation in the whole sample
(400 out of 1,203). The estimated marginal mean weight for these 15
children in stratum 1 was 
. Hence, the weighted L0 group would have 
high-ability children in stratum 1.
High-ability children attending kindergarten classes in a higher stratum had a
relatively higher representation in the L0 group and thus would receive
a relatively lower weight. As a result, the
composition of high-ability children in the weighted L0 group would resemble that of the entire analytic sample as
if their classes had been assigned at random to L0.
Applying Equation (a1) to the child-level data, we computed a marginal mean weight for each child as a function of the child’s prior ability level, treatment group membership, and stratum membership. We applied the same strategy to each of the six treatments for each subpopulation of children. Table A2 displays the computed marginal mean weights for the six treatment groups within each subpopulation of children.
Step 5. Checking balance in pretreatment composition among treatment groups in the weighted sample. We examined the difference in each pretreatment covariate among the six weighted treatment groups for children at each prior ability level. Adopting a significance level of .05, we expected to see about 5% of the covariates showing significant differences under the null hypotheses. Indeed, no more than 5% of the hypotheses testing showed a statistically significant difference. We therefore concluded that, under the weak ignorability assumption, all the six treatment groups became comparable for children at the same prior ability level.
Table 3
Weighted Analysis of
Differential Treatment Effects on Literacy Scale Score
|
Fixed Effects |
Coefficient |
SE |
t |
|
Literacy Pretest |
|
|
|
|
High Ability |
|
|
|
|
Intercept, |
30.44 |
0.90 |
33.81*** |
|
L0,
|
0.60 |
1.32 |
0.45 |
|
L1,
|
0.02 |
1.12 |
0.02 |
|
H0,
|
0.17 |
1.32 |
0.13 |
|
H1,
|
0.49 |
1.03 |
0.48 |
|
H2,
|
2.77 |
2.01 |
1.38 |
|
Medium Ability |
|
|
|
|
Intercept, |
17.68 |
0.38 |
47.09*** |
|
L0,
|
0.57 |
0.50 |
1.14 |
|
L1,
|
0.52 |
0.51 |
1.02 |
|
H0,
|
0.69 |
0.51 |
1.34 |
|
H1,
|
0.48 |
0.45 |
1.08 |
|
H2,
|
0.66 |
0.57 |
1.16 |
|
Low Ability |
|
|
|
|
Intercept, |
12.08 |
0.58 |
20.74*** |
|
L0,
|
0.25 |
0.81 |
0.31 |
|
L1,
|
0.67 |
0.75 |
0.89 |
|
H0,
|
0.51 |
0.76 |
0.67 |
|
H1,
|
0.60 |
0.67 |
0.89 |
|
H2,
|
0.48 |
0.85 |
0.56 |
|
General Knowledge ( |
0.27 |
0.01 |
19.18*** |
|
SES ( |
1.01 |
0.13 |
8.01*** |
|
|
|
|
|
|
Literacy Growth |
|
|
|
|
High Ability |
|
|
|
|
Intercept, |
12.53 |
1.07 |
11.68*** |
|
L0,
|
0.36 |
1.45 |
0.25 |
|
L1,
|
2.66 |
1.51 |
1.76 |
|
H0,
|
0.84 |
1.24 |
0.67 |
|
H1,
|
0.62 |
1.26 |
0.50 |
|
|
|
|
|
|
H2,
|
1.39 |
1.53 |
0.90 |
|
Medium Ability |
|
|
|
|
Intercept, |
13.99 |
0.51 |
27.53*** |
|
L0,
|
0.01 |
0.66 |
0.01 |
|
L1,
|
0.39 |
0.73 |
0.54 |
|
H0,
|
0.73 |
0.69 |
1.06 |
|
H1,
|
1.39 |
0.61 |
2.27* |
|
H2,
|
2.26 |
0.86 |
2.62** |
|
Low Ability |
|
|
|
|
Intercept, |
13.43 |
0.83 |
16.18*** |
|
L0,
|
2.05 |
0.99 |
2.07* |
|
L1,
|
1.85 |
1.14 |
1.63 |
|
H0,
|
1.12 |
1.05 |
1.07 |
|
H1,
|
2.51 |
1.01 |
2.48* |
|
H2,
|
1.60 |
1.59 |
1.01 |
|
General Knowledge ( |
0.13 |
0.02 |
6.21*** |
|
SES ( |
1.04 |
0.20 |
5.12*** |
|
|
|
|
|
|
Random Effects |
Variance Component |
df |
|
|
Level 2 |
|
|
|
|
Student literacy pretest, |
15.62 |
6,966 |
18,038.41*** |
|
Student literacy growth rate, |
38.22 |
6,966 |
18,891.35*** |
|
Level 3 |
|
|
|
|
Class literacy pretest |
10.22 |
1,697 |
5,055.70*** |
|
Class literacy growth rate, |
11.36 |
1,697 |
3,403.54*** |
Note: * p <
.05; ** p < .01; *** p < .001
Table 4
Estimated
End-of Year Proficiency Probability in Literacy Subdomains by Instructional
Treatment and Prior Ability
|
Literacy Subdomains |
Instructional Treatment |
|||||
|
L0 |
L1 |
L2 |
H0 |
H1 |
H2 |
|
|
Letter Recognition |
|
|
|
|
|
|
|
High ability |
1.00 |
1.00 |
1.00 |
1.00 |
1.00 |
1.00 |
|
Medium ability |
.99 |
.99 |
.99 |
1.00 |
1.00 |
1.00 |
|
Low ability |
.98 |
.97 |
.96 |
.97 |
.98 |
.98 |
|
Beginning Sounds |
|
|
|
|
|
|
|
High ability |
.99 |
.99 |
.99 |
.99 |
.99 |
.99 |
|
Medium ability |
.84 |
.83 |
.82 |
.86 |
.87 |
.88 |
|
Low ability |
.59 |
.58 |
.44 |
.51 |
.60 |
.60 |
|
Ending Sounds |
|
|
|
|
|
|
|
High ability |
.95 |
.96 |
.94 |
.96 |
.94 |
.96 |
|
Medium ability |
.48 |
.51 |
.45 |
.51 |
.51 |
.56 |
|
Low ability |
.19 |
.22 |
.10 |
.16 |
.21 |
.20 |
|
Sight Words |
|
|
|
|
|
|
|
High ability |
.47 |
.55 |
.29 |
.50 |
.42 |
.51 |
|
Medium ability |
.01 |
.01 |
.01 |
.02 |
.02 |
.02 |
|
Low ability |
.00 |
.00 |
.00 |
.00 |
.00 |
.00 |
|
Comprehension |
|
|
|
|
|
|
|
High ability |
.11 |
.14 |
.08 |
.12 |
.11 |
.13 |
|
Medium ability |
.00 |
.01 |
.00 |
.01 |
.01 |
.01 |
|
Low ability |
.00 |
.00 |
.00 |
.00 |
.00 |
.00 |