May 29…Scientifically-based Educational Research
How did the Eisenhart
and Towne article leave you feeling about the possibility of educational
research to be scientific? What obstacles do you see to realizing the vision of
a scientifically-based ed. research and are they insurmountable?
I appreciated Eisenhart and Towne’s clarification of the differences in the definitions of scientifically based research. Their historical account of the efforts education researchers made to improve bad legislation is important. We need stories that document the significance and the successful outcomes of scholarly activism. However, the article left me wondering about the current allocation of federal funding for education research. If by FY 2002 74% of all IES projects were causal claims using random assignment, what is that percentage today? Has the pressure from the community of researchers waned and if so, what message does that send? In my experience, when faced with the money and politics that Berliner attributes to the push for a narrowly defined, scientifically based research, a win doesn’t stay a win for long. Policymakers often allow a dissenting voice at the table, and even include language to satisfy the dissenters, with the full intent of pursuing their original agenda when the activists pack up and go home.
ReplyDeleteThe debate about the realization of a vision of scientifically based education research seems to hinge on whether the community of education researchers will be satisfied with inclusive policy language. This appears to me to be a discussion about the world as it should be as opposed to a discussion about the world as it is. The world as it is (referencing the dominant, hegemonic power structure) in this present time in history, recognizes, values, and is bent on pushing education research towards the emulation of hard science methodology. I left the “world as it should be” land a long time ago, because we live in the world as it is. If an overwhelming majority of education research federal funds are awarded to projects adhering to the narrow definition, that definition will drive the education research and subsequent policy agenda. Even if dissenting voices write within their circles about the inadequacies of a narrowly defined approach, the vision will have been realized, and will continue to be played out on the stages of America’s classrooms.
I don’t know if it’s because I have heard the debate about whether educational research is scientific so many times or if I’m just jaded by what Tami referred as the “world as it is.” Call me naïve, but I think research should be evaluated individually based on quality, but agree that the “sorry state of education research is not a matter of science at all, but of money and politics…” (p.31). I hate that so much of our time in education, especially as researchers, is spent having to defend the field and our research. Unfortunately, I don’t think this will change anytime soon.
ReplyDeleteI think the biggest challenge will be the incentivizing of certain types of research through the awarding of grants to preferred research and methodologies. With the tight economy and the reward structure at universities, many scholars might be led to pursue a narrow definition of research, even if it goes against their research question. In comparison to other articles illustrating the same debate, I think the Eisenhart and Towne article did leave me with a little bit of as the authors demonstrated that public input, especially from scholars, did result in some changes.
Articles like the Eisenhart & Towne piece leave me feeling concerned and frustrated. I don't know how my preferred research methods (phenomenological inquiry) can fit in a world where "replication" and "explicit chain or reasoning" are requirements. Like Stefani, I was struck by Tami's phrase: "This appears to me to be a discussion about the world as it should be as opposed to a discussion about the world as it is." This is so true. For example, I teach in a jail. It's pretty chaotic. I never know who is going to be present in my class each time I teach. Sometimes I only teach one or two students, and sometimes I teach fifteen. I don't know how a valid quantitative or empirical research project could be conducted in that kind of environment. However, interviews and narrative strategies are flexible and could reveal a lot about the learning experience in this type of setting.
ReplyDeleteOf course, the Eisenhart & Towne article, the NRC report, and NCLB are only focused on the P-12 population. I work with adult learners, and I intend to do research with that population. Will qualitative researchers be left on our own to scrape together the resources to figure out what went wrong with the "children" who get "left behind" and grow up to be adult learners without basic skills?
Ginger -
DeleteThis is also the case for special education for P-12 and beyond (who unfortunately can be the adults you speak of in jail). Sometimes students in special education have good instructional days and bad instructional days...the teachers may be forced to veer from the curriculum those days. Ultimately, in many educational contexts, the ability to be strictly scientific, create a replicable study, and explicit reasoning, is questionable. This is the main reason single-subject design is used so much in special education (and behavior analysis), a few case studies are used to obtain quantitative data. Many people do not always generalize single subject studies to the larger whole or see it as scientific enough. Is single subject research enough to get us out of the mush zone? Will there ever truly be 'education envy' from other disciplines?
Taryn
Like Ginger, I am usually frustrated when confronted with the proliferation of experimental research in social fields, like education. I appreciated how Eisenhart and Towne come at this issue. I like the chronological approach to telling this story with only a smidge of editorializing. While I still feel that the fervor over experimental designs in education is well and alive, it’s encouraging to read that, resulting from input from the public, education research definitions are broader. That said, “the WWC does not believe that quantitative methods are the only methods that can be called ‘scientific’” is hardly a glowing endorsement of alternative research methods. I wonder how much of the modest accommodations made as a result of public dissent are real in practice and are not just promises for more inclusive research without any backing whatsoever.
ReplyDeleteChanneling Ginger again, I also work in adult education where there is very little research available about what works, and the structure of adult education programs is not conducive to controlled experiments at all. Attendance is sporadic; teacher training and preparedness is often iffy; funding is unreliable; and many adult ed students may have undiagnosed learning disabilities which could greatly confound quantitative research results. This article leaves me with the feeling that, although ESRA claims to endorse research designs “as appropriate to the research being conducted”, there is not enough respect for the kinds of research that would work best with the adult population. If experimental research is what’s being funded, and adult education doesn’t fit the bill, then this already very marginalized population will just continue to fall further and further behind because we don’t have the structure or the monitory investment to do appropriate research that could help transform the field.
Before I get started, I feel like I must have a “rant” moment. Since learning about philosophies in education, I feel like we are constantly told to remain open-minded and learn about other perspectives. I feel like I have done that and I have an appreciation for most other theories (sorry ABER, you didn’t make the cut). With that said, I almost feel like postpositivism and “scientific-based research” is constantly being bashed in education (or in my exposure to education). Yes, I know this perspective yields methodology that is not always appropriate, but must everyone associate this theory to the many faults of education and policy? It’s not just the theory, people! End rant.
ReplyDeleteNow onto the Eisenhart and Towne article. First, I still don’t really feel comfortable saying that “causal” relationships occur in educational research even with a randomized controlled experiment. This brings me to the point that contextual factors play a monumental role in educational research and what might be “true” for one group of kids might not be true for all. Does this mean educational research is not scientific? Ehhhh, I don’t know. I guess it all depends on your definition and from this article there seems to be discrepancies. Personally, I feel like mixed methods research might be the best method to achieve scientifically based research in education. As a result, definitions should be altered to accept quantitative and qualitative methods of research. Numbers and randomization can assist with generalizability, while participant perspectives can shape the context. I am not saying this should be adopted as the “golden rule” of educational research, but I feel like research that addresses both methodologies gets a more holistic view of the study. However, methods will be dependent on your research question. I think the point about alignment in the article is one that is often overlooked because people are so concerned with methodology.
Unlike others, I am not very surprised nor disappointed or frustrated by this article, only because I feel 1) that things are shifting and have shifted over the decade+ since this has been published 2) most disciplines outside of the "hard sciences" all experience this issue.
ReplyDeleteComing from a psychology background, I am able to say that we grappled with the same issues of trying to make sure our research is scientifically based. Picturing the 4-quadrant grid that was drawn yesterday in class, any discipline that falls into Q3 or Q4, anything on the softer ends, will experience the questioning of the whether their type of research is scientifically based. It is more so the "outsiders" that question our research. As we talked about yesterday, we have to stop fighting to be in Q1, stop having physics envy, and just do what we know how to do and do what we know works!
What also makes me even more optimistic is that this article was published 11 years ago, along with the legislations that it discusses. Eleven years later, even though they are still being used, what journals, funding agencies, and universities consider research has changed drastically. With that being said, I do not believe that the obstacles that educational researchers experience are insurmountable. Changes have occurred, qualitative research is more widely accepted, educational research seems to be respected more. In a perfect world, OUR research is what effects policy, like the SRE influenced the Castle Bill, staying in line with our own goals as educational researchers will eventually pay off. Guess I'm a glass is half full type person!
As postpositivist as my pals may insist I am, I immediately go ballistic when I read things positing that randomized control trials and experimental designs are what we should all strive for. They do realize how incredibly difficult and resource-instensive these studies are, yes? More importantly, they do realize that despite using the most "perfect" methodologies, you aren't saving yourself from all of the other toxins that could be poisoning your results, yes? (i.e., poor validity and reliability). I could have a flawless (in some eyes) experimental design but my results could still be crap. It's like using a MANOVA to protect against Type I error when your research question is clearly a univariate research question. These things can't protect you from anything and they certainly can't cover up bad science! Even if they could, I would STILL be hesitant to say X caused Y because life isn't that simplistic (hence why we have discussion sections).
ReplyDeleteI'm with Lisa. I'm not upset by this article and I do think that things are SLOWLY shifting to encompass a more broader definition of scientific research. I do believe that too much emphasis is placed on methods than on just plain ole' good science. Everyone needs to stop being methods police and start evaluating quests for knowledge as quests for knowledge. If it answers the research question, is ethical, valid and reliable, who cares if I asked students a few questions or had them complete a survey? Both have advantages and disadvantages. Don't tell me how to do MY research. Judge me for the quality of my research. I'm not sure how what we all learn in our programs is not being clearly communicated to the fat boys in Washington (or Richmond, for that matter) but clearly they aren't hiring the right people. That is something I've struggled with for a long time. If we know all that we know, how are things like this still an issue?
I pulled this quote from a book and even though it is specific to psychology, I think it is applicable to education as well:
"Partly because many people do not know what science is, many people do not understand that psychology is a science--and even more people do not understand that it being a science is a good thing for both psychology and for the world."
Disclaimer: I completely lost my train of thought and now no longer remember why I added this quote. It's a good quote regardless so I'm just going to leave it!
I am going to run with Sarah's quote. I have always been taught that science is simply a systematic way of approaching something, so this whole debate around "scientifically based" research is a little pointless. Something is either scientific or not, regardless of methodologies or intentions. Many people do not understand education in this way, and fall into physics envy, instead of realizing that our fields are different but both can be scientific. This is good for the world because we try to answer different questions! Yes, certain kinds of work tend to be favored, because historically the case has been made for their effectiveness. However, it is clear that room is being made for new ways to see problems that still fall under the umbrella of science. It falls on us to make the cases for these new perspectives so that people can understand the way science works in our field and in the world.
ReplyDeleteI had a hard time getting my thoughts about this issue in writing until primed by the thoughts of my peers. I attribute a portion of my mental resistance to a general sense of frustration about this issue. I agree with Sarah and Lisa and am optimistic about the future of educational research and am frustrated that we continue to quibble about methodology. After reading Eisenhart and Towne (2003), I still believe that it is possible and desirable for educational research to be scientific. Multiple fields have struggled to define themselves and gain acceptance as valid areas of study. Education is still a relatively young field of study and I agree with Lisa that progress demonstrated since the publication of this article is promising
ReplyDeleteEisenhart and Towne bring to light struggles to define scientific research within education and the origins of current principles that influence the field. For the sake of personal clarity, I took a detour while reading the article and started to read the NRC report. I actually found the NRC report to be informative and absorbing, a real page-turner, and plan to continue reading it. The definition of scientific research that resonated most with me is from the NRC report: “To be scientific, the design must allow direct, empirical investigation of an important question, account for the context in which the study is carried out, align with a conceptual framework, reflect careful and thorough reasoning, and disclose results to encourage debate in the scientific community” (NRC, 2002, p. 6). Eisenhart and Towne (2003) assert that the NRC report outlined a set of principles rather than a prescription for research methodology. I think that we need to move away from placing too much emphasis on methods and more on the quality of the design and the knowledge produced. This was more succinctly stated by Sarah’s argument that quality research is quality research and a quest for knowledge should be evaluated as such.
The primary obstacle that I see in realizing an inclusive vision of educational research that leaves room for multiple perspectives and methodology is money. This view has been expressed by my peers and Eisenhart and Towne (2003) paraphrase David Berliner arguing that the “current state of educational research is not a matter of science at all, but of money and politics” (p. 31). School administrators, the researchers they employ, and those depending on government funding to survive are not going to endanger essential dollars by stepping outside of narrow definitions of “fundable” research projects. I believe that this is the heart of the problem and where we as educational researchers, leaders, and advocates need to focus efforts to advance the field.
This article did a lot to fill in some of the gaps in my own knowledge about these pieces of legislation and committees. At times it felt like Eisenhart and Towne were playing alphabet soup, but that's nothing new or unique in pieces highlighting educational policy.
ReplyDeleteI see one major obstacle to realizing the vision of a scientifically-based approach to educational research within the current fiscal-political climate that has been developed by these laws and policies.
The emphasis on randomized trials, like Sarah said, seems a little ridiculous. There are a few reasons for this. One, from what I've started to see, educational researchers sometimes overlook or fail to consider incentive structures when talking about external motivators for behavior. Yes, I hear people recognizing that "funding determines research", but I would love to see a more explicit piece that even used some economic theories (behavioral or otherwise) to explain and examine how federal funding effects our work. The obvious case that is shown in this article is the preference for randomized trials in receiving funding. No matter what researchers believe in their heart of hearts, if grant money goes to postpostivist studies, then there are going to be mostly postpositivist studies. This isn't groundbreaking, but I would love to see an article like this one that focuses more on that. Another major problem with a preference toward randomization in experimental method is the sheer logistical nightmare it can be to have thousands of ed. researchers conducting their work using randomization and clinical trials. Who is speaking up for the students, teachers or districts in this case? II'm not a flag-waver, I just think it would be a major impediment to both the educational process on the ground and it would slow the research community immensely. There is of course a place for methodology mirroring clinical trials, but to give it a prime spot seems irresponsible.
This article does, however, show a gradual shift towards a more balanced approach to the definition of scientifically based research in education, and I agree- it's a good thing. My one question, before I can get too critical of the Castle Bill, SRE or even NCLB is: how bad was it before? Was educational research in the 80s and 90s laughable? I honestly don't know the answer, and I think that might be an area in which I do some individual inquiry. We were talking about the importance of context in quality analysis and questioning last class. I think that piece might really help understand where these bills sit in historical and philosophical context.
It took me a few minutes to realize that the unfamiliar feeling I had tingling inside me after reading the article was actually…optimism. One of my biggest personal concerns is that the best I can do for my area of concern is rearrange deck chairs on the Titanic, so it was encouraging to read a story where those "in charge" were actually influenced by community efforts in a seemingly positive direction. I even found the standards laid out in the ESRA reasonable. Granted, the language sets a positivist tone (and I don't like that the methodology should be "objective"), but I don't see how it excludes any effective ways of gaining knowledge…
ReplyDelete…and it’s right about here that my coffee kicked in and the critic in me woke up. As a member of a traditionally oppressed racial group (bear with me- I’m still looking for a succinct replacement for “minority,” which I’ve recently stopped using) I’m acutely aware of how policy can appear open-minded to the group in power yet condescending and misguided to a powerless group. Here, I’m one of the quantitatively-oriented people patting myself on the back for being more inclusive while failing to grasp how this policy – even if in tone, alone - still implicitly privileges quantitative research and treats qualitative research as second-class.
Anyway, as for the possibility of “scientific” educational research – once again I find myself unable to address the question because I’m caught up on a term that I fail to attach sufficient significance to. I get that there should be metrics against which the quality of a piece of research is measured, I just reject that the degree to which it is “scientific” should serve as some gold-standard. I think there’s an analogy to McMillan’s emphasis on the difference between “reliability of an instrument” versus the “reliability of the scores obtained from an instrument” – here, the “research” itself shouldn’t be what’s subjected to scrutiny, but rather the process of drawing conclusions from the research. The methodology is only important insomuch as it helps the consumer conclude whether certain findings are relevant to him or her. In other words, to me, whether a piece of knowledge is “right” or “wrong” trumps how that piece of knowledge was developed. Can ed research be scientific? Sure, why not. Science has been a pretty reliable way of developing knowledge so it’s probably useful here, too. But is it important whether it’s scientific? Honestly, I don’t know that I care, so long as the knowledge developed by ed research is useful and “accurate.” Of course, there are definitely obstacles in the way of my utopian vision of research, but having just skimmed through other people’s comments and realizing I’m not necessarily alone in my thoughts here, I’m back to being optimistic again…
…for now.
I am in agreement with a lot of my colleagues on this matter. I, too, have heard a lot of "we should be open to all types of research" - which is a statement with which I would mostly agree. I admittedly struggle with arts-based work, as well as autoethnographies. However, many of the same voices that repeat this mantra often do not extend the same openness towards postpositivist work.
ReplyDeleteAs someone who draws from postpostivism and pragmatism, I have come to find value in other types of work. Some of the best articles I've read recently about experimental and quasi-experimental studies took the time to examine critiques of their topic from many other types of research (critical, narrative, case studies, etc). They addressed these critiques in their methodology and their studies were only strengthened as a result. Without other types of work being done, these postpostivist studies would have yielded findings that would have told us less. Different perspectives are good and definitions of what constitutes appropriate research should reflect that.
Like Drew, I would also like to have an economist (or a team of economists) look at the impact that these definitions have had on what types of research gets funded and produced. While it seems to make sense that these definitions, established a decade ago, would have made everyone a postpostivist, one could also argue that there is greater diversity than ever in terms of the type of work that is being produced in education. This leads me to question whether these definitions have had as chilling of an effect as some might suggest.
It is hard for me to believe all educational research can be "scientifically based" because we are dealing with humans that are by design learning and evolving before our eyes! The definitions stated in the article don't allow for this type of subject very well. As we discussed in class, our discipline is soft and applied which makes it hard to replicate studies and lends itself to different types of research - research that is meaningful and that we (the education community) can learn from and expand on.
ReplyDeleteI am not a huge fan of policy. There is a lot I don't understand and I see many people not in education making decisions that should be at the county or school level. So, this article was painful for me. Just as Drew pointed out, wherever the money is going will in the end influence the type of research that will be done. I know this is not going to change so maybe the researchers need to change and figure out ways to do meaningful work that in the policymakers eyes is "scientifically based".
Eisenhart & Towne's article was a representation of the federal legislation that mimics most education research -- the building the plane as you fly it approach. In many cases, educational research is still using this approach due to the "physics envy". We are the pilots building the plane - and I am actually okay with that. Someone has to do it, so why not the researcher? The article left me feeling indifferent. The problem 11 years ago, is the same problem we experience today in the grant world. As being part of a grant, you get federal dollars to produce a vision of the stakeholder. However, sometimes the stakeholder does not know what that vision is...so they bring in the experts, or pilots (back to the plane approach). This is the first year of any grant, unless you are an educational researcher with extreme superpowers. There are many players involved...and it gets mushy.
ReplyDeleteIn the end, all the qualitative data we collect is just for our nighttime reading material. The stakeholder wants numbers. How many people took your online course? How many people watched the webcast? No one asks for the story behind the numbers, the facilitation of the course that encourages students to finish, and the student's experience in completing a course. Also, no one asks if these participants are actually implementing what they learned. Should we be defining scientific research more broadly to encompass all disciplines, or is this maybe a discipline specific definition? From our quadrant graphic, we can see there is not a clear overlap or distinction between the disciplines, unless you are in the mush zone together.
As a behaviorist by trade, I see the importance of systematic and evidence-based practices. However, the context (in my field, ecological systems theory) cannot be dismissed, that is just not who we are as a field, or we would be physicists not pilots. Is it so wrong to build the plane as we fly it, so that others don't have to? That is our real goal. I feel that we may be focusing on defining something, like we do everything else in education, that will eventually change again, in 5 or 10 years based on where the field and stakeholders focus is at the time.
Taryn
After reading this article, I rather think that the future of scientific research in education is going to be fine. I say that, because if there is anything that Eisenhart and Towne show it is that not even our system of government can come together and give a unified definition of what "scientifically based research" actually is. Some state the research has to be explicitly experimental, others say experimental and/or quasi-experimental, and others still say "qualitative research, that's scientific and okay, too." Now, some might look at the article and say that the limits o what is scientific research defined in NCLB are constricting and terrible (and they are), but then one year later you have the ESRA that has language considerably more broad than the NCLB.
ReplyDeleteI feel that has to be emphasized; the federal government defined scientifically-based research as one thing, and within ONE YEAR, they defined it as something else entirely.
There is little unification in terminology in these Acts, and as researchers and educators, we're all the better for it. Why? Because we can make arguments for our science. If someone says "your study is not using an experimental design, then it isn't science," you can cite a federal document, a more recent document and say "yes it is!" It is kind of bizarre, having to think like a lawyer in a courtroom in defense of your research, but that's who is writing these Acts in the first place; politicians who largely come from backgrounds in law. It isn't about discovering the truth, it is about convincing someone who is more right than the other. That "this definition of scientific research is more correct than this other definition." If I foresee any one, hard obstacle to scientific research in education, it is making sure that the folks who make mandates on how we do research fully understand what research actually is. It isn't as simple to define as a paragraph. It isn't easy to contain in a simple boxed quotation. Science is broad, and its scope ever moving into the beyond. Law is in place to set boundaries and define limits; science is meant to push boundaries and go far beyond the limits of what we know.
After reading Eisenhart and Towne’s article I feel that I understand the complications of trying to clarify scientific educational research a little bit better than before. I am bit disappointed but not surprised that the “idea of scientifically based research in education is still being contested” (page 32). Why does the education research community have problems with the inclusive language? After all education as a field is so diverse that the only way to include all the different facets of education would be to be broad. That way many more research types would have a chance for funding and in the long run that will be beneficial to all.
ReplyDeleteMaybe it would help tremendously if we could all start to look at scientific research as something that is on a continuum and not try to measure all scientific research by the same hard science yard stick heavily based on methodology. Maybe this would only happen if money was not the issue since those research ideas based on loosely defined parameters might not get funded as readily as others that are based in more post-positivist methodology. Just like some of my classmates have said, it all boils down to money and politics and that is unfortunate.
In the end, I think what really matters and I agree with the authors’ on this is that it really doesn’t matter what anybody’s view is on what constitutes scientific educational research so long as the conversation about this issue is still encouraged. People should continue to acknowledge and value the diversity of education research provides. If this happens, then there is a chance that one day the community of educational researchers will come to a consensus as to what the definition of scientific research is in education.
I think of what Eisenhart and Towne wrote and compare it to our conversation last class about what is applied/pure; soft/hard as Labaree proposed. I believe that we all have an obligation to conduct research that is ethical and rigorous. That said, this meets the definition of scientific research. As long as we fulfill the requirements of rigor and ethics, then we should not worry about how "scientific" our research is. Educational research will never be the same as theoretical mathematics research, and quite frankly, if it were, I'd be concerned.
ReplyDeleteEisenhart and Towne based their article on what was reported by the NRC report. While this defines educational research, I am very much troubled that we are using governmental reports as our methods of deciding what is "good enough." I believe that academics have the duty to use the scientific principles and methods that they feel are the best to answer their questions to advance knowledge. While we have to consider policies (such as NCLB), I fear that we are too concerned about promoting (or combating) policy that we lose sight of our objectives.