BUILDING EWS DATA SYSTEMS

RECOMMENDATIONS

Click the icon to view recommendations

SUPPORT

RECOMMENDATION 6

A PAS workgroup of leading education data, research, and technology experts from school districts, state departments of education, technology companies, and nonprofit organizations developed the following guidance for EWS 2.0 data systems, and the considerations for districts, states, and other entities as they develop and improve their data systems.

Key Characteristics of Effective EWS 2.0 Data Systems:

~ Any early warning data system should provide educators with easily accessible data on validated indicators that are predictive of their students’ high school graduation and postsecondary success.

~ The indicators provided should give educators timely, actionable insight into the students who may need additional support — and in which ways — toward high school graduation and postsecondary success. Ideally, EWS data systems should identify individuals and groups of students who need particular support, and help provide information for broader school and district policy decisions that will result in greater support overall.

~ Early warning data systems should enable educators to record, track, and analyze the impact of the actions — also called interventions and responses — they take in response to information from the indicators.

~ EWS 2.0 data provided should be able to be aggregated at the individual, classroom, grade, school, and district level (and state level for statewide systems), and disaggregated by different student sub-groups, including customized sub-groups created by schools and districts.

To achieve these characteristics, PAS makes the following six recommendations below.

RECOMMENDATION 1:  VALIDATE

Validate indicators and thresholds for accuracy and usefulness in supporting students’ success in middle and high schools and students’ readiness for postsecondary success.

~ EWS 2.0 data should be based on validated indicators that are strongly predictive of high school graduation and postsecondary success.
For these indicators, data systems should provide thresholds for action that show users whether a student is on track, falling off track, or entirely off track.

~ While there are national recommendations around specific thresholds for attendance, behavior, and course performance that suggest when students are off track for high school graduation and postsecondary preparation, our research and experience also suggests that the predictive power of the ABCs can vary by district.
Given this, we recommend that districts locally validate their indicators wherever feasible. Alternatively, states can validate a set of EWS 2.0 indicators based on statewide data while giving districts the flexibility to adjust thresholds based on local circumstances. In states, districts, and schools where neither option is viable, start with the nationally recommended thresholds and adjust them as needed, based on local experience over time.

Additional considerations:

~ Indicators, or thresholds for action, should be based on accurate data sets.
Data collected by schools and districts can be messy, with many different definitions, standards, collection methods, entry and processing procedures, time stamps, and more. Thus, district or state data analysts will need time to organize and clean up data before schools and districts conduct analysis. School staff involved with data reporting and entry also may need training in the use of common data definitions and accurate data entry.

~ District and state capacity and state roles in serving districts vary based on size of district and the nature of each state:
U.S. school districts range in size from about 100 students to 1.1 million. Thirteen states each have fewer students than the country’s four largest school districts. Local and state decisions must be made about the location and design of EWS 2.0 data systems and training for their use. States with large numbers of small or rural districts may need to take responsibility for validating data and action points, technical details of setting up systems, and helping schools use the systems. States with large variation in size and nature of school districts will need to determine their areas of greatest need and how to deploy technical and human capacities and resources.

~ States should explore whether to develop a cloud-based, self-service model that will allow districts to upload their data for analysis.
The program could then validate the indicators, set action points, and provide real-time reports back to the school or district. This type of solution may be critical for small and rural districts. It would also allow districts to experiment with different variables that may influence graduation or postsecondary readiness rates and provide better information on student success. Since cloud-based technology requires sending data back and forth, these channels should be rigorously verified for security. We also strongly recommend that such data be used for student-support purposes only — not for states’ school accountability systems.

~ Some researchers have found different thresholds for different groups of students, such as English-language learners.
Ensuring indicators and thresholds work well for all student populations will help users target interventions more carefully and efficiently.

RECOMMENDATION 2:  DEVELOP COMPOSITE INDICATORS

As technology and digital storage capacity continue to evolve, so do possible strategies for gathering and monitoring EWS data. Initial EWS were based on data in teachers’ grade books, attendance rolls, and disciplinary referrals. Later, Excel-enabled digital sorting and analysis were used.

Such approaches were based on only a single or few easily collected and validated indicators (like those that formed our ABC system). Later, more advanced programs allowed schools and districts to consider many different indicators of students’ needs, as well as composite indicators that rate or issue scores on students’ status toward graduation.

~ Ideally, EWS 2.0 data systems should contain both valid individual indicators to help schools identify effective actions to keep students on track to postsecondary success, and composite measures that provide guidance for prioritizing student-support strategies.

~ Whether schools or districts have a composite indicator or not, EWS 2.0 data systems should provide easily accessible data on individual indicators that show directly whether students are on track to high school graduation and postsecondary success.

~ These guidelines can help districts, states, and those who will build or adjust data systems determine how to organize their EWS 2.0 indicators

RECOMMENDATION 3:  TRACK INTERVENTIONS

Track interventions for students:

~ EWS 2.0 data systems must enable monitoring of interventions.

~ EWS 2.0 data systems should help users clearly see which students need additional support, which interventions or supports they have received, and whether student outcomes improve as a result.

~ EWS 2.0 data systems should also include the type of intervention, how often a student participates, and the indicator the intervention will address, so that results can be aggregated and analyzed to determine which interventions help students the most.

~ In addition to common types of interventions, systems should also allow school or district teams of educators to add customized interventions.

RECOMMENDATION 4:  DATA DISPLAYS & REPORTS

Data displays and reports.

~ Data displays should be developed for different user groups, since school-based teams that will determine support for individual or groups of students may need different reports from those needed by principals or district/state officials for entire schools or districts.

~ Additional reports for students and families could help communicate students’ areas of need and strength, provide students more agency in their own improvement, and help them monitor their own progress.

~ For each report, the team should only provide the needed data for each group, ensuring the reports are useful and compliant with student privacy laws.

Additional consideration:

~ Align EWS 2.0 with other district priorities so that reports can serve a variety of purposes.

Data Visualization Samples:

RECOMMENDATION 5:  DISSEMINATION

Disseminate reports and meaningful uses for the data.

~ EWS data system developers should consider how they will disseminate reports and encourage meaningful use of the displays. A collaborative approach should include an iterative process of asking educators for their needs in student support monitoring, leading to mutual ownership of the system design.

~ Careful consideration should be given to how data are disaggregated. Professional development should be provided for school/district teams with respect to the discussion of diverse groups of students.

Additional considerations:

~ Build champions in the district who can advocate for EWS 2.0 systems as an essential resource. Design systems to be sustainable through staff turnover or technology changes.

~ Ground the data in individual stories to make the work compelling. EWS 2.0 also may motivate students to take actions or seek help if they can see the indicators for themselves. They should be partners in identifying solutions and their own education goals.

RECOMMENDATION 6:  SUPPORT TEAM

Assemble an effective state-/district-level EWS 2.0 data system development and support team

~ District/state-level EWS 2.0 data system teams should consist of technology experts who will build the system and can make changes based on the team’s feedback; administrators who can support the initiative and align the work with other school/district initiatives; and an EWS 2.0 manager or coach(es) who will train schools on the system and EWS 2.0 more broadly.

~ In large school districts and at the state level, the divide between leaders of information technology and research/assessment is a common challenge. Build relationships across departments and have a district/state leader prioritize the work and help build collaboration.

Additional considerations:

~ Build constituency and usability. Provide presentations across the state/district to get feedback and build educator support and interest.

~ Help districts tie EWS 2.0 into existing priorities and develop metrics so that teams can identify when reports need to be updated and data improved. Align technology with student support needs rather than allowing technology to shape the work.

~ Many times, only a few professionals in a district/state can provide programming for the EWS 2.0, and these experts often must address other urgent issues. Developing high-level champions for the project and a direct line of communication with the superintendent or another leader will help make EWS 2.0 sustainable. Provide clear links between EWS 2.0 and all other data initiatives such as RTI, PBIS, and MTSS; do not make EWS 2.0 a standalone project.