Thus, a whole-school reform effort, for example, may seek to change the everyday practices of all administrators, teachers, staff, and students in a school; an effort to boost employment in a public housing development may be aimed at all of the development’s residents, not just selected residents.
Second, even if the services are not directed toward everyone, they may have “spillover” effects that would make a fair test of the services impossible.
In this methodology issue focus, the first in a series, we explain one such design, has been at the forefront of both the theoretical refinement and practical use of this methodology.
As the name suggests, cluster random assignment means the random assignment of whole groups, or clusters, of people.
Larger standard errors make it harder to determine that an impact is statistically significant — that is, reflective of a real difference between the treatment and control groups, rather than likely to have arisen as a result of chance.
One way to reduce the standard error is to increase the number of individuals in the sample, since the larger the sample, the more likely it is that the treatment and control groups will be substantially identical.
Unless both kinds of sampling error are included in the standard error, investigators may wrongly decide that a program is making a significant difference when, in fact, it is not.
The strategy that Bloom and other leading social scientists employ in cluster randomized trials — referred to as “multilevel modeling” or as “hierarchical modeling” — takes account of both sources of sampling error in producing impact estimates.
Using an unusually rich set of individuals’ earnings data, employment histories, and socio-economic characteristics, the authors address two questions: (1) Which nonexperimental comparison group methods provide the most accurate estimates of the impacts of mandatory welfare to work programs; and (2) do the best methods work well enough to substitute for random assignment experiments?
The nonexperimental groups are compared with experimental control groups from a large-sample, six-state random assignment experiment — the National Evaluation of Welfare-to-Work Strategies.