Mindsets and the learning environment: Open-ended responses in college applications contain valuable insights about mindsets and college success
The Student Experience Research Network launched an interdisciplinary initiative in Fall 2016 to explore how learning environments shape the mindsets students develop about learning and school. The project’s aim is to generate scientific evidence about how educators, school systems, and structures can convey messages to students that they belong and are valued at school, that their intellectual abilities can be developed, and that what they are doing in school matters.
Fourteen projects were awarded over two rounds of the initiative. Funding was generously provided by the Bill & Melinda Gates Foundation, Joyce Foundation, Overdeck Family Foundation, and Raikes Foundation. Seventeen different Network scholars are participating along with over 20 external collaborators. The projects span a wide range of topics, from exploring how teacher practices cultivate learning mindsets and identity safety in K-12 classrooms, to the relationships between learning mindsets and neural processes throughout adolescent development.
This is part of a series of blog posts in which we will hear from the leaders of each of the projects funded in the second round of the initiative to find out more about the questions they are exploring, what they are learning, and how their work is advancing the field of mindset science.
Prior research has found that high school students who participate in extracurricular activities are more likely to experience college success than their peers. The linkage is thought to be due, in part, to the fact that extracurricular activities can develop beliefs and skills that can be relevant in college, including students’ sense of confidence, identity, and purpose.
The project, Language as Thought: Using Natural Language Processing to Investigate Mindsets, Learning Environments, and College Success, builds on these findings by exploring the connections between extracurricular activities, learning mindsets, and academic outcomes. Using sophisticated analytic approaches to assess college applications, the researchers identify language related to learning mindsets in students’ descriptions of their extracurricular activities and work experiences. The researchers then examine whether these language-derived measures of mindsets predict students’ college success (as measured by graduating within four to six years). Finally, the researchers investigate whether students’ high school learning environments influence the link between mindsets and college success.
Who are the members of the research team?
The interdisciplinary research team is comprised of Mindset Scholar Sidney D’Mello, Cathlyn Stone, and Stephen Hutt from the University of Colorado Boulder, and Mindset Scholar Angela Duckworth, Margo Gardner, Donald Kamentz, and Abigail Quirk from Character Lab. Together, the team has research expertise in cognitive science, computer science, psychology, education, human development, as well as practical experience implementing college initiatives.
What is the purpose of the project and how will it fit into the field of mindset science?
This project will shed light on how students’ extracurricular activities and work experiences – as well as the language students use to describe those activities and experiences – reflect learning mindsets. As Sidney put it, “language reflects thought, so an analysis of language should provide insight into how people think – i.e., their mindsets.”
The researchers are also examining whether high school and home learning environments shape learning mindsets and college graduation outcomes.
The team is using data from the Common Application, the primary application portal for nearly 700 colleges, and the National Student Clearinghouse, where students’ college enrollment and completion information is reported (together, the CommonApp-NSC).
Specifically, the team is analyzing 150-word, open-ended written responses, in which applicants elaborate on one extracurricular activity or work experience, to identify language related to learning mindsets. They are also looking at how quantitative dimensions of extracurricular activity (number of activities, participation duration, and participation frequency) relate to college outcomes. To analyze the open-ended responses at scale, they use natural language processing and machine learning techniques.
“We are in an exciting age where these artificial intelligence techniques, which are apt at analyzing incredibly large volumes of data, can complement more traditional psychological methods,” explained Sidney.
Ultimately, this project will be able to provide valuable insights about which features of extracurricular activities and work experiences matter most for success in college.
About the data
The project leverages six years of CommonApp-NSC data, which includes college application data and graduation outcomes for 273,196 U.S. high-school students. The CommonApp includes students’ demographic data and information about their high school. The NSC includes college enrollment and graduation status.
The study makes an important contribution to the field by using pre-collected data; that is, no additional information was collected directly from students. This approach is valuable as a complement or alternative to survey-based self-reporting or even teacher reports, which can be biased because respondents may guess at the aim of the survey and cater their responses to the objective.
In their college applications, students are certainly putting their best foot forward, but they likely are not thinking explicitly about learning mindsets. Importantly, the option to report on work experiences provides students from low-income families, who are less likely than students from higher-income families to participate in extracurricular activities, with a forum for reporting their experiences.
Using a random sample of 550 responses, the researchers identified seven constructs related to learning mindsets that could be identified from the open-ended responses: accolades, goal orientation, growth mindset, leadership, perseverance, psychological connection, self-transcendent purpose, and team orientation.
Then, the team sampled 4,000 responses and manually coded them for these seven dimensions of mindset. Next, they studied the relationship between mindset and quantitative dimensions of extracurricular participation, including number of activities and duration and frequency of participation.
The team found that students who used more mindset-related language in their college applications also spent more time on high school extracurricular activities. These results varied by the dimension of students’ participation. For example, the researchers found significant positive associations between number of extracurricular activities and all but one mindset indicator (psychological connection) but found that participation duration was positively related only to accolades and leadership. Overall, however, these findings support the hypothesis that extracurricular activities and work experiences during high school may shape or reflect the development of learning mindsets.
All of the initial coding and analysis was done manually by the research team, so their next step was to automate the coding. This was done using a combination of natural language processing and deep machine learning techniques that identify which features of language are diagnostic of the various mindset dimensions.
As Sidney explains, “the computer basically learns which words in the essays are most effective at replicating the human codes in a manner that generalizes across students.” For example, the program learned that words like “learned,” “taught,” and “helped” were positively correlated with growth mindset, while terms related to overcoming hardship like “difficult,” “tough,” “constant,” and “trying” correlated with perseverance.
The team found that mindset-related language surfaced by both the human and computer coding predicted college graduation, with accolades and leadership being the two mindset-related concepts most predictive of 4- and 6-year graduation.
The team found consistent results for students with different racial/ethnic backgrounds, English Language Learner (ELL) status, parent marital status and education, and high school environment (Title I eligibility, racial/ethnic composition).
What are the next steps for the project?
After additional refinement of their automated language coding, the research team will apply their models to the complete sample of more than 270,000 student responses. The full analysis will illuminate the role of learning mindsets in the relationship between extracurricular activities and work experiences and college graduation, and will provide insights about how these connections vary based on students’ identities and learning environments.