Research+Design+Articles

Experimental research –provides most credible evidence of effectiveness of a practice.
 * Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71(2), 165-179. **
 * Instead of using groups to make comparisons, single subject participants provide their own comparison.
 * Baseline and intervention conditions. Essential features: To establish whether a functional relationship exists between a practice and student outcomes at the level of individual participants. Each participant’s behavior is compared to his or her own behavior across multiple conditions.
 * Three Key components: 1) Target behaviors must be assessed repeatedly. 2) Intervention is systematically introduced and withdrawn and 3) The effects across baseline and intervention conditions must be analyzed for each participant
 * Get a stable baseline (nned several data points to get current level)
 * Single subject research requires that at least one replication of the functional relationship is included within the design
 * Reversal: ABAB
 * Multiple Baseline: AB across participants
 * Mean performance calculated and compared for each condition in the design to determine impact of intervention
 * Level-change in performance that occurs just after intervention is implemented or withdrawn—referred to as change in level
 * Researchers look at the **immediacy and magnitude** of change as indicators of the strength of intervention’s effect.
 * Trend- the trendline of the treatment data
 * Results are always graphed
 * See Horner for Checklist

=== ** Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71 (2), 149-164. ** ===

** If the intervention is applied to new participants, settings, of context, there should be clear links based on argument and/or research for this new use. **
=== ** Comparison groups: Most common reason—comparing innovative approach to a traditional one. “designs that are more elegant include multiple groups, some involving subtle contrasts (eg. Identical intervention with or without progress monitoring). ** ===

** -Research question stated clearly: A succinct statement of the key questions addressed is critical. Linked in an integrated fashion to the purpose statement. **
=== ** -Hypothesized results should be specified. But should be candid if there is no specific prediction—“reviewers need to be aware that it is equally appropriate to frame clear research questions without a specified hypothesis” ** === === ** -Describing participants: Must move beyond school district provided labels. Provide definition of disability and also include assessment results documenting individual in the study met the definition requirements. ** ===

** Include some measure of interobserver reliability **
=== ** “Far too often, the weakest part of an intervention study is the quality of the measures used to evaluate the impact of the intervention”… a good part of the effort must be on development of dependent measures. Evidence for validity and reliability must be provided, too. ** ===

** -Effect sizes in the .40 or larger range are considered minimum levels for educational or clinical significance **
=== ** Determining when a practice is “evidence based”: There are at least 4 acceptable quality studies, or two high-quality studies that support the practice AND the weighted effect size is significantly greater than zero. ** ===

** Angell, M. E., Stoner, J. B., & Shelden, D. L. (2009). Trust in education professionals: Perspectives of mothers of children with disabilities. Remedial and Special Education, 30(3), 160-176 **
====** Face to face interviews: **** Trust identified as a first step in creating collaborative relationships. Importance of trust –critical: “Most critical thing for advancing education and welfare of children”..found to be a factor that de-escalates conflict within Due Process ** ====

** State Problem: **
====** “To date, however, researchers have not fully examined the nature of trust between parents of children with disabilities and schools”… ** ==== ====** -has emerged as important in examining the collaboration between parents of children with disabilities and school, but trust has not been the central issue examined ** ====

** * How do mothers of children with disabilities describe their trust in educational personnel? **
====** * What factors do mothers of children with disabilities identify as contributing to or detracting from their trust in educational personnel? ** ====

** Minimally effective **
====** Second phase: asked individual education professionals to assist in recruiting. The participants distributed recruitment materials to their friends. “snowballing method” ** ====

** -checked demographics of participants **
====** -conducted cross-case analysis to study each case as a whole and comparative analysis to study the cases against one another ** ====

** -Constant comparative method **
====** -Achieved triangulation, respondent validation (process by which researchers ask participants to check accuracy of the findings in the areas of descriptors, themes, and interpretations), and member checking (providing participants with the opportunity to review material—shared transcripts with participants) ** ====

** -Discussion section: **
====** -In this study, after analysis of the categories, the researchers placed the results within a Family Systems framework. ** ====

** Generalizability is an issue **
====** “recruitment through school personnel may also limit generalizability in that school personnel may not be comfortable identifying mothers with whom they or others in the school had negative relationships or incidents of distrust” ** ====

===<span style="font-family: 'Times New Roman',Times,serif;">**Brantlinger, E., Jimenez, R., Klingner, J., Pugach, M., & Richardson, V. (2005). Qualitative studies in special education. Exceptional Children, 71 (2), 195-207.** === ===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt;">Definition—Qualitative research is a systematic approach to understanding qualities of a phenomenon within a particular context. ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· To document or discover phenomena
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Typically collect their own data by observing in the field and or interviewing participants ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Finally, “tell the story”
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Data collection is most productively done in creative ways eg., using interview protocol flexibly ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt 0.5in; text-indent: -0.25in;">· Objectivity and subjectivity
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt;">Qualitative researchers are on the more positivist end of a qualitative to quantitative continuum –Quantitative see subjectivity as a problem that interferes with validity, Most qualitative researchers recommend being explicit about personal positions, perspectives, etc. ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt;">Example: Mercer’s work on IQ tests & African American children---requirement for an adaptive scale
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt;">Personal histories are prominent forms. Qualitative typically include an insider-to-phenomenon perspective (in contrast to quantitative) === ===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt;">Interpretation: Studies are interpretive when they contain critical element that entails intense interrogation of the meanings that undergrid daily life occurrences, common sense assumptions, trends in the field, power imbalances in institutions, etc. ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Researcher Reflexivity—Researchers self-disclose
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Member checks—having participants review accuracy of interview transcriptions or field notes ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Peer debriefing
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Audit trail—keeping track of interviews conducted and/or specific times and dates spent observing as well as who was observed ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Thick, detailed descriptions
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt 0.5in; text-indent: -0.25in;">· Particularizability—documenting cases with thick description so that readers can determine the degree of transferability to their own situations ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Researcher’s personal position is reflected
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">· Conclusions substantiated by sufficient quotes from participants, observations, and evidence ===

<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt 0.5in; text-indent: -0.25in;">· Connections with related research
===<span style="font-family: 'Times New Roman',Times,serif; margin: 0in 0in 10pt 0.5in; text-indent: -0.25in;">Not done for generalization but to produce evidence based on exploration of specific contexts & particular individuals ===

<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Concern about the quality of scientific research in education <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Some agencies: What Works Clearinghouse—have focused on the “gold standard”—randomized experimental group designs (2003) <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">CEC-task force (2003) identified 4 types of research a) experimental group b) correlational c) single subject d) qualitative <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Methodology matched to question arising from different points in continuum <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">National Academy of Sciences—proposed most research questions in education can be grouped into 3 types: a) description (what is happening?) b) cause (is there a systematic effect?) c) process or mechanism (why or how is it happening?) <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Complexity of sp ed as a field <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">-extreme variability of participants <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Greater ethnic and linguistic diversity doe to overrepresentation <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">-educational context more complex and varied <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Specify for whom and in what context practice is effective <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Can’t always have random assignment to a non-treatment group due to ethical issues. Students with disabilities often clustered in classrooms and in experimental group design, the classroom rather than the student becomes the unit on which researchers base random assignment, data analysis, etc. <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Focus research on question of effectiveness <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">RCT-random assignment to experimental groups is 1 indicator of high quality group design <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Descriptive & process oriented research may require qualitative <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Value of mixing methods <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">WWC- design and implementation Assessment Device—rater can conduct very thorough evaluation of an article <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Problems of deciding what studies to include in a research synthesis and how to determine ES <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Levin et al. (2003) suggest program of educational research might be thought of as occurring in 4 stages: 1) observation—exploration and flexible methodology which qualitative and correlational methods allow 2) controlled lab or classroom experiments, observational studies of classroom, teacher-researcher collaborative experiments 3) design well-documented intervention and prove effectiveness through RCT studies implemented in classrooms or naturalistic settings by the natural participants 4) determine factors that lead to adoptions of effective practices in typical school systems under naturally <span style="font-family: 'Calibri','sans-serif'; font-size: 11pt; line-height: 115%;"> existing conditions.
 * Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R. (2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 71 (2), 137-148.**

**Tankersley, M., Harjusola-Webb, S., & Landrum, T. J. (2008). Using single-subject research to establish the evidence base of special education. Intervention in School and Clinic, 44(2), 83-90**
<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">To establish that an instructional practice is effective, researchers seek to find a systematic or functional relationship between variables <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Experimental regarded as providing the most credible evidence of the effectiveness of a practice. Two types of methodological approaches: Group experimental and Single Subject <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Group: a) Use meaningful comparison groups b) actively and systematically implement the instructional practice being tested. <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Single Subject: allow researchers to draw causal conclusions <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Differs from group experimental: 1) Instead of using groups to make comparisons, participants provide their own comparison. 2) Their performance is compared across conditions in which they are and are not participating in the intervention under study · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Baseline and intervention conditions · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Observations occur frequently to determine reliable baseline · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Once baseline is stable & predictable the intervention is introduced and performance is again measured frequently while it is in place during a period of time · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Comparisons of data across baseline and intervention provide the basis for determining whether there is a Functional Relationship between the indepdendent and dependent variables · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Horner et al. (2005) recommend that at least three demonstrations of a functional relationship should be included · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Purpose of Single Subject is to establish whether a functional relationship exists between a practice and student outcomes at level of individual participants · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Each participant’s behavior or performance is compared to h/her own behavior or performance across multiple conditions **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">3 Key Components: ** <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">1) Target behaviors must be assessed repeatedly using trustworthy measures <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">2) Interventions must be systematically introduced and withdrawn <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">3) Effects across baseline and intervention conditions must be analyzed for each participant

· <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Progress monitoring model—frequent and repeated measurement over time · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Interrater agreement—2 independent observers record same behaviors · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Operational definitions of targets · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Use at least two conditions <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Requires at least one replication of the functional relationship included in the design <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.75in; text-indent: -0.5in;">I) ABAB: One of the most powerful because it can clearly show relationship between implementation and changes in the target behavior <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.75in; text-indent: -0.5in;">II) Multiple Baseline: when there are concerns with withdrawing the treatment (eg. SIB) or when something (such as “learning”) cannot be reversed. Incorporates a baseline and an intervention (AB) across participants, behaviors (more than one target), or settings (more than one environment). Intervention introduced in a staggered sequence for each participant, behavior, or setting. Sequential introduction continues until the intervention is introduced in succession for each participant, behavior, or setting **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Analysis of Effects: ** <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.25in;">Within-participant changes in performance are typically evaluated according to the strength or magnitude of the target behavior (mean and level) across conditions and the rate of those changes. · **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Mean **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">: mean performance calculated and compared for each condition to determine impact—if mean performance during intervention is meaningfully better than baseline, the intervention shows evidence of effectiveness. Graphed · **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Level: **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">The change in performance that occurs just after intervention is implemented or withdrawn is referred to as a change in level. Target behaviors that show immediate or abrupt changes demonstrate strong reaction (immediacy and magnitude) · **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Trend: **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;"> Change in trend or direction of the data as the intervention is applied and withdrawn · **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Latency **<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">: The quickness with which the behavior changes at the termination of one condition (baseline or intervention) and onset of another. The shorter the timeframe the clearer the effect <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Practice of Sp Ed in schools lends itself to single subject—Hallmark of which is the assessment of individual performance and behavior change over time <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Results must be replicated across several studies before generalization can be made · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Single Subject—Appropriate for answering research questions that allow comparisons to be made within participants and for which interventions can be made within participants and for which interventions can be introduced and withdrawn systematically · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Target behaviors must be assessed repeatedly using trustworthy measures · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Researchers must measure target same way on repeated occasions · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Measurement of target behavior must occur multiple times during each baseline and intervention condition · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Interventions must be systematically introduced and withdrawn repeatedly · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Effects of intervention are evaluated when conditions change · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">The effectiveness of the intervention evaluated with participants · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Changes between baseline and intervention conditions are assessed within individuals rather than between individuals or groups. · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Changes in a participant’s performance across conditions are evaluated according to the mean, level, trend, and latency of the changes in h/her observed behavior · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Designs include explicit evaluation of causality and thus can be used to determine whether a practice is evidence-based · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Given typically small sample sizes, multiple single subject studies are needed to determine definitively whether a practice is effective.

<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Correlational studies quantitative multi-subject designs in which participants have not been randomly assigned to treatment conditions <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Analytic methods commonly applied are multiple regression analysis, canonical correlation analysis, hierarchical linear modeling, and structural equation modeling <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Crucial to match research questions and designs <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Questions involving school or classroom cultures may require qualitative <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Clinical trials may raise ethical questions regarding denial of needed services to control groups <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Correlational designs do not provide the best evidence re: causal maechanisms. There are two ways to causal inferences: <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">1) Statistical Testing of Rival Causal Models <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in;">Statistical equation modeling—factor analysis <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in;">Within the structural model, analysts may test whether 1) 2 latent constructs ( x + y) covary or are correlated 2) x causes y 3) y causes x 4) x + y reciprocally cause each other. Rival models can be tested & if there is only one model that fits the data, then there is some evidence bearing on causality <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.5in; text-indent: -0.25in;">2) Logically Based Exclusion Methods: we might investigate the pre-intervention differences in the students on everything we consider relevant (eg., pre-intervention reading scores). Confirm no extraneous contaminants of treatment influences <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.25in;">Limitation of Nonexperimental research <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.25in;">-All of the correct variables are used in testing models—the true model is seldom known <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.25in;">-“Stepwise analysis”- do not correctly identify best subset of predictors <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.25in;">-Yield results that tend to be non-replicable <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.25in;">-temptation toward Type I errors <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.25in;">Effect sizes reported in less than ½ published articles __<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Quality Indicators: __ · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Measurement: reliability—do scores measure what they are supposed to? Most articles don’t even mention reliability. Should report coefficients for instruments used · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">“It is unacceptable to induct the score reliability coefficients from prior studies or test manuals if there is no explicit evidence presented that the sample compositions and standard deviations from the prior study and a current study are both reasonably comparable · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Score reliability coefficients are reported for all measured variables · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Score reliability coefficients reported for all measured variables based on analysis of the data in hand in the particular study · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">…or test manual that suggests scores are valid for the inferences being made in the study · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Score validity is empirically evaluated based on data generated within the study · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">The influences of score reliability and validity on study interpretations are explicitly considered in reasonable detail <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Practical Significance: evaluates the potential –quantifying the degree to which sample results diverge from the null hypothesis. Quantifications referred to as effect sizes. Standardized differences (eg., Cohen’s d) most common. <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Clinical Significance: extent to which intervention recipients no longer meet diagnostic crieteria and thus no longer require intervention. <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">They mention Cohen’s Power Analysis, in which he provided benchmarks for small, medium, and large effect sizes. <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Common Mistakes: <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Effect size reporting is rare. Some report but do not interpret. Often they fail to identify which effect size is being reported. __<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Quality Indicators: __ · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">One or more effect size statistics is reported for each outcome. Statistic used is identified · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Interpret effect sizes explicitly and compare to prior or related studies · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Explicitly consider study design and effect size statistic limitations __<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">Four Types of Errors: __ <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">1) Failure to interpret structure coefficients: GLM weights are interpreted reflecting correlations of predictors with outcome variables only in the exceptional case that the weights indeed are correlation coefficients <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in; text-indent: -0.25in;">2) Converting Intervally Scaled Variables to Nominal Scale <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 0pt 0.5in;">Researchers may convert independent variables into nominal scale to run “OVA’s”. eg. take intervally scaled IQ data and convert to “low” and “high”. This attenuates reliability—throws data away. Quality—interval data not converted to nominal unless such choices are justified and results interpreted. Univariate methods are inappropriate with multiple outcomes variables (inflates the probability of Type 1, does not honor the reality that outcome variables can interact with each other to define unique outcomes that are more than their constituent parts). <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt 0.5in; text-indent: -0.25in;">3) Failure to test statistical assumptions: <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Confidence Intervals for Reliability Coefficients, Statistics, and Effect Sizes: <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">CI can be used to determine whether a given null hypothesis would be rejected. If a hypothesized value is not within the interval, the null hypothesis positing the parameter value is rejected. Confidence intervals inform judgment regarding all the values pf the parameters that appear to be plausible, given the data . By comparing the overlaps of CI across studies, can evaluate the consistency of evidence across studies. The widths of CI within a study or across studies provides critical information regarding the precision of estimates. When intervals are wide, evidence for a given point estimate being correct is called into question. · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">confidence intervals are reported for the reliability coefficients derived for study data · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">confidence intervals reported for sample statistics · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%;">confidence intervals reported for study effect sizes
 * <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 115%; margin: 0in 0in 10pt;">Thompson, B., Diamond, K. E., Mcwilliam, R., Snyder, P., & Snyder, S. (2005). Evaluating the quality of evidence from correlational research for evidence-based practice. Exceptional Children, 71(2), 181-194. ﻿- **