Tuesday, May 20, 2014

June 3…Scientifically-based Educational Research, pt. 2

By now it should be clear that most (all?) ideas in academia are contested. Today’s readings show some variety in terms of how we might think about educational research but from your knowledge and experience, do any points of view seem missing from the collections of articles?

17 comments:

  1. My first year of teaching I read an anonymous quote that I still have posted on my office wall: “The art of teaching is to awaken the natural curiosity of the learner, only to satisfy it later.” Clearly Kurt employed this art when lining up our class readings.

    I chose to read Gutierrez and Penuel. They address the specifics of how to implement one of the criteria we touched on in class last week: making research applicable to the field. They went beyond applicability and addressed the need for a collaborative process where practice and research can inform one another. Knowing that members of the research community are being strategic in calling for the relevance to practice as an explicit criterion for judging the quality of research proposals makes me hopeful. Gutierrez and Penuel lay out how funders could evaluate this criterion, including “demand[ing] more attention to the process side, asking researchers to address what research would look like and what methods and co-designing process are relevant to the study at hand” (p. 21). Gutierrez and Penuel suggest that generalizability should be contingent on the uptake of research by local actors who must sustain programs. They suggest a “shift in focus away from innovations designed to be implemented with fidelity in a single context and toward cross-setting interventions that leverage diversity (rather than viewing it as a deficit)” (p. 22). Now that’s a definition of what works that I can get behind!

    ReplyDelete
  2. As a student, I was constantly told that only true experiments could determine causality, but lately I've seen more research articles questioning this line of logic and claiming that qualitative research can also be used to argue causality. Since education is filled with both quantitative and qualitative research (with most people favoring quantitative as 'better'), I was surprised none of the articles I read really addressed this point of view or the definition of 'causality.' One of the main arguments for causality comes from the fact that it favors the positivist (and similar theories). Also, some qualitative methodologies, when used correctly, can be more in-depth in detail and findings. Here's an example of an article discussing this point of view (http://fmx.sagepub.com/content/16/3/243.full.pdf+html). Since many of the arguments surrounding research in education seem to focus on qualitative research, as well as contextual research, I was surprised more people weren't discussing the definition of causality.

    ReplyDelete
  3. When suggesting ideas or giving professional development to teachers of mathematics, I always heard the experienced teachers say something to the effect of, “we used to do this years ago” and “the pendulum is swinging back again”. I always wondered why the “pendulum” of mathematics education was constantly moving and if something didn’t work then, why are we going to try it again now? Rudolph (2014) explained that what counts as good teaching and learning today is not likely to be the same in 20 years. He points at the changing politics, global economy and allocation of resources as just some of the factors that change our educational goals over time. After reading this point I wondered if we should be looking at how those factors influence the way we teach and vice versa. Studying the relationship between the two could put us potentially in front of the power curve when it comes to how students learn and what society sees as important aspects of the classroom learning environment. It seems like we are always trying to catch up in the classroom with the latest trend and once teachers (and students) get the hang of it, something changes.
    I agree with Rudolph (2014) that classrooms are very different from controlled laboratory settings, so trying to do research that mimics biology or chemistry is just a waste of time. As we have talked in class about education being a soft science, you can’t apply hard science techniques and expect the same results. So let’s focus on how we are different from the hard sciences and move on so that we can end the “long, unproductive history of educational practice that ebbs and flows without clear direction” (Rudolph, 2014, p. 15).

    ReplyDelete
  4. "But we should rightly be concerned when policymakers (supported by a pubic operating with an incomplete understanding of what science is and how it works) seek to push particular research models and methodological approaches in a misguided attempt to secure knowledge outcomes (reliable, predictable, uniform, etc.) that are unlikely to be obtained given the nature of the activities and enterprise in question."

    I love this quote from pp. 17 of Rudolph. I remember reading this when it came out this past winter-I got just as excited being reminded of the quote as I was when I read it for the first time. He just hits so many things right on point with this quote.

    I must confess that I had read all of these prior to being assigned them for class. Thus, I know that as a group, they cover a whole lot of ground and I, personally, find them very inclusive (again, as a whole). While some of the individual articles may be missing some chunks, I think the group of articles covers just about everything we have all discussed in class (I can't actually think of something at the moment that is missing).

    I agree with Stefani that causality isn't discussed ind detail. However, "predictive power" is discussed a good bit. I think this is what they are sort of using to get at inferring causality. If we can use our results to improve/increase predictive power, then we are saying X causes Y so we can use X to predict Y. I'm not completely sold on those parts of the discussion. I believe it was Weiman who said that should be one of the fundamental facets of high quality research...meh...I don't think that should be as high on the priority list as some of the other things discussed, such as research being contextualized, pragmatic, flexible. Like my man D.C. (aka Dennis) suggests, much research in education would be high quality if predictive precision/utility is a key criteria.

    ReplyDelete
  5. After reading the introduction, I was drawn to the Wieman article. As a physicist, I found his recognition of similarities between his field and education refreshing. I particularly liked the comment, “cutting-edge science is often ‘much messier,
    complicated, and less precise’” (p. 12). I am not advocating for poor research; but rather, hidden gems might be overlooked in research because everything is so contested. As Wieman states, “even ‘unsuccessful’ experiments can have value and predicitive power’” (p. 12).

    Through my experiences in research, I have been conditioned to critique and tear apart articles. If a paper contains too many flaws, it is considered “poor” research. Although a flawed article may not be ideal for exact replication, it may ignite new ideas within researchers. Replication is a necessary component of research that is often overlooked, but educational research cannot be innovative without complicated ideas that might yield less precise results. We do not remember the names of scholars who didn’t challenge the field in some way or another. As a future researcher, this article reminds me not only to search articles for their weaknesses, but also to become proactive and figure out ways to develop good research questions and designs out of poor studies. Every man’s trash is another man’s treasure.

    ReplyDelete
  6. I enjoyed this group of articles, the perspectives represented, and the witty little insights offered by the authors. (Atoms don’t require IRBs. Ha! I didn’t have physics envy before, but now…). The point of view that is missing from several of the articles is one connecting research to practice. What is the point of the “what is scientific research?” argument if insights into improving educational experiences gathered through research are not conveyed to practitioners in a timely manner or if they are improperly implemented after being processed through the machinery of policy and funding? Also, what if research is not even relevant in a given educational context? Guiterrez and Puneul (2014) begin to outline ways in which the field of educational research is already addressing this issue. For example, they mention programs such as the Research-Practitioner Partnership in Educational Research that are working to create a bidirectional flow in research planning and implementation. They also discuss the necessity of situating research in context, the importance of judging the quality of research based on relevance to practice, and the need to embrace the messiness of educational research. This point of view makes a great deal of sense to me.

    In addition to conducting research in partnership with practitioners, to further connect research to practice, the reward structure within academia based on the publication of research also requires revision. In multiple settings, by a range of faculty, we have been informed that publication in respected, academic journals is rewarded and is a key component of promotability within the current tenure system of higher education. There is no incentive for publications written for teachers and educational professionals. This creates distance between research and those most influenced by it. The research-practitioner partnership needs to guide the entire research process including reporting results in an accessible manner and format. We spend too much time arguing about what makes research scientific and applying the principles of Goldilocks to fields of study i.e. “this one is too hard,” “this one is too soft.” In future discussions of educational research, I hope to see more focus on the utility of research, how to advance and maximize research-practitioner partnerships, and how the use of appropriate research methodology to address relevant issues can effectively produce findings to help improve our educational system.

    ReplyDelete
  7. I, too, read several of these articles when they came out a few months ago. I also believe that most of the broad strokes are here. The article I was initially drawn to was the Philips piece, and while most of his article discusses research from a more postpostivist stance, he does give a nod to qualitative work. Perhaps what is missing is a little more discussion of what good qualitative work looks like and how it can be useful to inform quantitative work. This could work both ways as well. Whereas a quantitative study might tell you that two groups are different, qualitative work might shed light on how or why that is the case. Likewise, a qualitative study highlighting a flaw (or at least something perceived to be a flaw) in a particular policy or program might spur a quantitative researcher to control for a factor he or she might not have otherwise considered.
    The question of “what should count as quality education research” begs the reciprocal question “to whom?’ For most here, the answer to the latter is either “the government” or anyone else who is doling out the funding. I would argue that good research should have an opportunity to get funded and definitions sometimes prejudice the types of questions that a researcher might choose to seek answers to in the first place. Again, good research – including that which is “harder” in nature – only benefits from the existence of research that is “softer” in nature. While they don’t always, they can inform each other for mutual benefit. Seeking to prejudice work against the “softer” lines of inquiry only hurts the type of work that these definitions seek to encourage.

    ReplyDelete
  8. I read the Guiterrez article and agree with what Tami said above. Despite not exactly being a "people-person" myself, I find the participatory research model they discuss much more appealing, practical, and interesting than more traditional expert-based models. I'm also glad to see them even extend the participatory model into the definition of "good research" by advocating for utility as a criterion. It just makes good sense to me that practicality should be a consideration in educational research (maybe here's where the engineer in me shows, since designs are always constrained by reality), and it's hard to find a better judge of practicality than the practitioners on the ground.

    But if there's a missing voice in this article, it might be that of the university administrator, who would probably increasingly argue that "good research" is research that brings the largest indirect cost budget and most prestige to the university. Instead, the type of research laid out by Guiterrez is time consuming, and expecting "documentation that the focus [of a proposed project] is perceived by multiple stakeholders to be significant, persistent, and worthy of investigation," in the proposal means that researchers must put in a great deal of legwork before even being considered for grant funding. I would question how willing most universities are to pay for such an extended proposal preparation stage from their own pockets. The emphasis on contextualizing generalizability also threatens the "neatness" of the research, which may have implications for publication opportunities that further impact faculty researchers. So, as much as I like participatory educational research models like this, I would like to hear more about how academia would adjust to accommodate them.

    ReplyDelete
  9. My earliest academic training was in biology and chemistry, and I still carry some of those “hard science” tendencies with me today. I chose to read the Wieman piece because his split perspective made sense to me. I appreciated his definitions of hard and soft science that seemed less political and more structurally based. I think it is useful to see a science’s “hardness” as a progression because it means that every piece has value somewhere (and this is certainly helpful from a motivational perspective!). Complications arise when we try to get other people to see things this way. I am specifically considering policymakers here, but I imagine many people see things differently. The policy world and the academic world are very different places and the pressures on both, at least to me, seem very different. However, there is relatively little communication between academics and policymakers. There is a bias towards large-scale “hardish” research in policy, but with a lack of communication it seems like neither group really understands the way the other values the different kinds of knowledge. By fostering better understandings of science as something that is not hard or soft, but something that develops, we might be able to improve upon that relationship.

    ReplyDelete
  10. Rena
    I read Philips’ article titled, Research in the Hard Sciences, and in Very Hard “Softer” Domains. Phillips sees the similarities in the two types of research since all research is trying to add to the body of knowledge. So really it boils down to proving or disproving a hypothesis and this does not matter across the disciplines, be it “hard’ science or “soft” science. So every type of research is making an argument and finding evidence for the argument.

    Phillips stipulates that research in the hard sciences is somewhat easier than in education since things seem more streamlined. “In the physical sciences, confounding variables can be controlled” (page 10) and you want to (control them) while in educational research confounding factors such as “socioeconomic status, gender, ethnicity and others are of great human and educational significance” (page 10). Research subjects do not live in a vacuum. This makes educational research more complicated than research in the “hard sciences”. I find this to be very ironic that people who do research in the “hard” sciences have greater status, get paid more etc. and yet people conducting educational research are actually working harder since they have to pay attention to all the confounding variables. As if this is not enough, they still have to justify their research as it can be hard to determine causality in educational research as some experiments may be considered “quasi-experiments” since it is often impossible to control for some variables. Dealing with so many ecological factors makes it hard for educational research to have the predictive power that research in the hard sciences possess. Because of all this, I feel Phillip’s conclusion that “while physics is a hard science, education is a very hard- an extremely hard - one” (page 11) is warranted. It is high time that the rest of the research community took a second look at educational research and try to appreciate how hard and complicated it really is.

    Phillips, also briefly touches on the issue of under-funding and politics of funding which affect the type of articles published. However he does not discuss the drive for educational researchers just like other researchers in other disciplines to have their publications in prestigious journals so they can get respect. If most of these highly respected journals are following the “physics” formula controlling for ecological variables, the research would not make sense. Articles that may be more relevant to practice in the field of education might not be published or be relegated into lower impact journals which may have limited circulation.

    ReplyDelete
  11. I just got off the phone with my best friend who is an elementary school teacher for the DC Public School district. She teaches dual language first grade (half of the day in English, half of the day in Spanish). She was telling me how her students are excelling in English, but not in Spanish. She feels that this has a lot to do with because the district has invested a lot of money into English programs that are research-based. However, the Spanish language portion is relied on whatever books that the school happens to muster up. Best practices in education are important, but without the proper resources (including professional development for the practitioners), we will easily lose sight of the purpose of educational research.

    I read the Gutiérrez and Penuel (2014) piece. I certainly agree with their points. Educational spaces are highly contested. We need to make sure that our research is grounded, but we also consider practice in our research. If the protocol or intervention does not have any practicality to it, is the intervention even valid? One thing that I particularly appreciate about the VCU School of Education is the commitment to partnering with school divisions. Without the support of our local schools, any research would not stand a chance to be implemented at the classroom level. (Yes, I agree that I’m thinking more micro than macro, but that’s how I tend to think.)

    I think it was important to include the teachers in this collection of articles. I do wonder though, what we can do to increase more “buy-in” from teachers when conducting research.

    ReplyDelete
  12. After reading the articles assigned this week I felt a sort of rejuvenation. What I mean by that, I felt that I did not have to worry so much about educational research and the problems that we often think are exclusive to us, but recognized there are other fields experiencing the same issues. I particularly enjoyed the Phillips article and the Wieman article. Besides Phillips being quite amusing, I found that he made a lot of great points about which fields should other fields be modeled after. As he would say "this general issue is moot…" (p. 9). As I said in a previous post, we just need to worry about ourselves and the strides we are making in our field.

    And as Molly expressed, Wieman stuck with me because of his experience in a field that we educational researchers are often criticized by. He brought to light that despite what many of the "softer" scientist may think, the hardest sciences are not that cut and dry. He goes on to explain that physicist has been around so long that everything that can be studied has been studied but for education times change so often that research has to follow those steps as well. So yes, we could be just like the physicist if there were not any new technologies invented weekly, if all children learned exactly the same, if the world was not so innovative, and if all the stars aligned. But like Wieman said our (education) research is "more fun" (p. 13). So as I continue to say, be proud of our research because despite what we may think, and what some others may say we are a "hard" science; not many other disciplines have to factor in so many variables into their research than the ones studying humans and their learning.

    WIth that being said, I believe, of course other points could have been argued in this collection but the points that I agreed with were clear and argued well. Great readings!

    ReplyDelete
  13. Reading the Introduction, I thought the goal to “define within our various academic research communities the criteria of rigor so that we can be better positioned to share these criteria with other communities, including policymakers and educators” seemed pretty lofty. I went ahead and read the whole group of articles, because I was interested to see if anything approaching a unified set of criteria for rigorous education research would emerge. Didn’t, but I found, especially with Gutierrez and Penuel, the most explicit call for rigorous qualitative research, which was not really addressed by the other authors. My own personal leaning is that both quantitative and qualitative approaches can work together to meet high standards of quality research. While the other authors definitely emphasized the importance of context in education research, I didn’t get the sense that they were committal about including qualitative methods in the criterion of rigorous research.

    The Gutierrez article also brought up important questions that I don’t think are considered by policymakers when they take what looks like rigorous research and translate it into policy. The questions that struck me most were “What are the unintended consequences of change?” and “Who benefits?” Regardless of the intentions of researchers, policymakers will act on talking points that benefit themselves, so intended consequences and benefits, while they may have been a part of the research design and method, may not translate into policy. Jake pointed out in his comment that there is little communication between academia and policymakers. What I’d really like to read about is how ed researchers can effectively engage policymakers and together develop a set of criteria that is appropriate to research with living human beings who don’t live in a laboratory. It’d be nice if funding sources would recognize the complexities of ed research, rather than rely on the status of certain research methods that don’t always fit.

    ReplyDelete
  14. In some ways, after reading Wieman's article I identified with his overall argument that the hard sciences and education are not as different as critics of either may proclaim. This was an interesting and new stance I had never encountered. I sort of assumed that physicists would look down from their high tower or concrete a reproducible data in judgment of educational researchers. Instead the author argues that the hard sciences are more established, and therefore their research appears more clear and less murky, but in reality all newly generated claims and theories in any discipline are messy and unclear. Education, according to the author, just isn't there yet in terms of maturity.

    I don't know if I completely agree with this idea though. The simple fact remains that educational climate and context is fluid and can never be fully consistent. Wieman himself explains that in physics or any field, results that can't be reproduced are not "wrong" (p. 13). When dealing with individuals as teachers, students, administrators and policy makers, I can't imagine education as a field ever "maturing" to a place where we will get consistent results once we truly understand all variables. Though it's nice to get some respect from someone in "quadrant 1" (to go back to our chart), I'm not sure he completely understands the true nature of educational research.

    ReplyDelete
  15. After reviewing each of the articles, the piece missing seemed to be the "study's relevance to transforming practice' (Gutierrez & Penuel, p. 19, 2014). In reading this article, I felt relief that someone saw the need to know 'what works' inside a program in a direct, systematic, and sustained way. Much of qualitative research on 'what works' does not focus on the broader pictures that policymakers and researchers care about, this article addresses that gap.

    "Consequential research on meaningful and equitable educational change requires a focus on persistent problems of practice, examined in their context of development, with attention to ecological resources and constraints, including why, how, and under what conditions programs and policies work" (Gutierrez & Penuel, p. 19, 2014). I love the phrase 'persistent problems of practice' because this takes away the notion of designing research to fix current issues. Most educational researchers want to address issues with best practices that tend to recur as Heather says, when the pendulum swings back. However, the problem-oriented researcher must remain aware of the rigorous process that is, scientific inquiry. This way educational research will go beyond "what works".

    In addition, I enjoyed thinking about 'empirical tinkering'. This is the essence of education. We are not a hard science. Therefore, fine-tuning and refining interventions and research studies is often necessary. Does this make this research less rigorous? I think not.

    Like Kate suggests, it is necessary for funding sources to take these ideas and make sure it is communicated with the policymakers or stakeholders. Fidelity is not something that can be captured in one context and for one group of students. The recommendations from the Gutierrez and Penuel (2014) article attempt to address the pieces I have always seen as missing in educational research.

    Taryn Goodwin

    ReplyDelete
  16. Full disclosure, I went ahead and read all the articles. It was interesting seeing these authors debate the merits and faults of educational research from multiple perspectives, and I could not get enough. I just imagined this panel discussion, authors going back and forth, agreeing, disagreeing, and perhaps feeling misquoted in certain parts.

    I digress. Perspectives! What I appreciated from these pieces are the fact that you get a fair amount of variation in terms of what angle the authors are coming from, be they devoted educational researchers, converts from "harder" sciences, or analyze from the lens of federal mandate and educational initiative. I liked seeing where there was overlap in certain trains of thought, I enjoyed the divergences, but I do wonder; where is the perspective of the educational researcher who moved into another realm of science? I have learned that, in the professional world, people from pretty much anywhere can and will collaborate in research. While at the VA hospital, I worked with folks on a QEEG brain-mapping project who came from all over the place; chemistry, social work, psychology, nursing, neurology, and myself from a rehabilitation counseling background. Here I am, a person considered to be in "soft sciences" working on "hard scientific" research. So really, that's who I want to hear from. I imagine their thoughts might not be all that different from some of the authors we read, but by just changing the order in which they might approach the topic would seem interesting to me. The perspective of someone who moved away from educational research to go back and reflect on what might make it quality research. THAT would be fascinating.

    ReplyDelete
  17. In order to answer this question, I looked over the introduction as well as the articles by Phillips, Rudolph, and Gutierrez & Penuel. I appreciated Phillips's argument for the "hardness" of the "soft" science of education, Rudolph's historical context, and the idea of practical application as a measure of "rigor" in Gutierrez & Penuel. I still feel that something is missing, though. I think it has to do with audience. These pieces are dedicated to the argument that educational research is "scientific." The idea that exploration in educational issues should be anything other than "scientific" is untouched and unmentioned. For instance, no one is making the argument that teaching is an art, something that can be learned the way that dancing or painting is learned. I think the audience for this publication is the reason why more alternative "points of view seem missing," to use the language of Dr. Stemhagen's prompt.

    ReplyDelete