UPDATES Sign up to receive periodic updates from the Student Experience Research Network.
DESIGN
The National Study of Learning Mindsets (NSLM) is the largest-ever randomized controlled trial of a growth mindset program in the United States in K-12 settings. The study combined a test for cause-and-effect (a randomized experiment) with a sample that enables claims about an entire population (a nationally representative probability sample). This unique design enabled the study team to understand which kinds of students in which kinds of schools and classrooms would benefit most from the short online program designed to foster a growth mindset. It also generated a rich longitudinal data set that can be used for many future research projects.
SCHOOLS IN THE SAMPLE
The schools that participated in the NSLM were selected using sophisticated sampling techniques so the results of the study could be generalized to the more than 12,000 regular public high schools in the U.S. 139 schools were invited to administer the program and provide student records, and 76 schools agreed to participate. 65 schools provided all requested records, including both survey data and administrative records for students. The remaining 11 schools provided only the student and teacher survey data.
For more information on the sample of schools in the NSLM, see Yeager et al., 2019 and Gopalan & Tipton, 2018.
The sampling frame was built using the U.S. Education Department’s Common Core of Data (NCES). Schools that were excluded from the sampling frame include charter and private schools, schools with specialized missions or populations (e.g., schools for students who are visually impaired), schools with fewer than 25 9th graders, and schools that do not have 9th grade as the lowest grade.
The schools that participated in the study were randomly sampled, rather than being a convenience sample of schools that volunteered to opt in to the study. This random site selection is important because schools that opt into a study might be different in ways that could ultimately influence the effectiveness of the intervention, potentially biasing the results.
139 schools were randomly selected from over 12,000 regular American public high schools. These schools were randomly selected from five different categories of schools that varied in their achievement level and concentration of students of color, to ensure that high- and low-achieving schools as well as schools with high- and low-concentration of students of color were sufficiently represented in the sample. This improved the study’s ability to understand how the effects varied across student groups.
A total of 76 schools of the 139 that were sampled agreed to participate in the study. 65 of the 76 schools provided both student and teacher survey data as well as student records. The remaining 11 participated in the study but only provided student and teacher survey data.
The schools that participated were a mix of rural, urban, and suburban. In fact, one strength of the study relative to much of the education literature is the strong representation of rural schools. Many evaluations of educational programs occur in districts that offer many large schools and easy access to data, to improve sample size. The NSLM included over 30% rural schools. Overall, the schools also represented varying levels of achievement as measured by school test scores, average PSAT and SAT scores, and NAEP math scores.
Schools were categorized into one of three performance levels: lower-achieving schools in the bottom 25% nationally, medium-achieving schools in the middle 50% nationally, and high-achieving schools in the top 25% nationally. These groupings were determined by combining several sources of data. The first was GreatSchools.org, which provided within-state information on schools (and rankings) based upon test scores. The second was average PSAT scores, which were provided by the College Board. The third was AP test-taking data, including the proportion of students at the school who take AP tests and their test scores. These three data sources were combined in order to create an underlying school achievement measure.
STUDENTS IN THE SAMPLE
Over 16,000 9th grade students from 76 U.S. public high schools participated in the National Study of Learning Mindsets. 17% identified as Black/African American, 6% Asian American, 21% Latinx, 44% White, and 12% another race or ethnicity. 35% of participants reported that their mother had a bachelor’s degree or higher.
The analyses of the growth mindset program’s effects on academic outcomes published in Nature included over 12,000 9th grade students from the 65 schools that provided all requested records: 11% Black/African-American, 4% Asian-American, 24% Latinx, 43% White, and 18% another race or ethnicity. 29% of participants reported that their mother had a bachelor’s degree or higher.
Students with GPAs below the median GPA of students at their school in the term prior to random assignment (spring of 8th grade for most participating students) were defined as “lower achieving.” In other words, the bottom 50% of students in the sample were defined as “lower achieving” and the top 50% of students in the sample were defined as “higher achieving” for purposes of the analysis.
STUDENT RESPONSE RATE
The student response rate was 92% and the median school had a student response rate of 98%. This response rate was obtained by extensive efforts to recruit students into make-up sessions if students were absent, and it was aided by the student-tracking software system developed by PERTS. A high within-school response rate was important because lower-achieving students are typically more likely to be absent.
MEASURES COLLECTED
The NSLM includes a wide array of student-, classroom-, and school-level measures. Student-level measures included academic outcomes and demographic information from administrative records; survey measures completed by students about their background, beliefs, and classroom and school contexts; and a behavioral task measuring students’ challenge-seeking. Mathematics teachers completed a survey about their beliefs, instructional practices, and background. A wide range of publicly available data on schools was also gathered.
LONGITUDINAL DATA COLLECTION
The first wave of the NSLM followed 9th grade students through the beginning of 10th grade. The study team will be collecting additional longitudinal data on the students who participated in the study to examine the longer-term effects of the program on academic and life outcomes. These future waves of data collection will also yield a rich, longitudinal data set that follows students through high school and beyond.
