Posted by: littlez2008 | October 18, 2010

Value Added Testing Pros and Cons (But Mostly Questions)

There has been a major upset among Los Angeles teachers over a recent LA Times article which discussed teachers Value Added Test scores.  The biggest flap has been over the Times publication of an online, searchable database that ranked teachers and schools from most to least effective.  The outrage and fear has been palpable among my colleagues.  Yesterday I gave a short talk at the Division of Adult and Career Education conference and one adult teacher there was asking if there was any way adult students’ test scores could be linked back to teachers.  Would we be next?  Would our effectiveness be published in the paper and our livelihoods called into question?

And I guess at this point I have to reveal my conflict of interest in discussing this topic.  I was giving a talk at that conference about how to increase test performance for adult students, and I was giving this talk because that is what I do for a living.  For eight years, I’ve worked outside the classroom, testing and measuring and sending off data in a gigantic database to the district, where it in turn goes off to the state and federal government to help assess our school’s effectiveness–and also to ensure our funding.

While I work for the Adult Division, and the testing we do is different than the tests in the Times article, I still can bear witness to the idea that teacher effectiveness can be measured (in part) by testing.  The tests the Times article discussed are value-added tests.  This means that if a student starts school in the 60th percentile for math or reading, we would reasonably expect the student to end the school year in the 60th percentile.  If the student lands in the 80th percentile, that jump is, according to the value-added model, attributable to some extent to the student’s teacher–the increase is the value that has been added by the instructor.  Of course, there are tons of other variables that could affect student performance (obviously, including how students feel the day of their test), but if you examine this data over many years and see consistent trends, the idea is that you can make evaluations of at least this one aspect of a teacher’s performance.

Do I think standardized tests can be use to evaluate teacher effectiveness?  The answer, based on my years of experience looking at data like this, is absolutely, yes.  Are they the only measure of teacher effectiveness?  No, of course not.  But I am disheartened that standardized tests are often demonized by educators.  The United Teachers of Los Angeles (UTLA) has excoriated the times.  The latest issue of the UTLA Magazine had photos of a variety of teachers on its cover, all holding an identical sign that read, “I am more than a test score.”

I have to heave a deep sigh when I see a reaction like this, because it confuses rational critique with personal criticism.  The authors of the Times article did not claim that the teachers were good people or bad people.  Their dedication wasn’t called into question, just their effectiveness by one measure–one that is not often revealed to the public.  Isn’t examining teacher effectiveness important?  And isn’t using data one way to approach that problem?  The authors made the point that:

Although many parents fixate on picking the right school for their child, it matters far more which teacher the child gets. Teachers had three times as much influence on students’ academic development as the school they attend. Yet parents have no access to objective information about individual instructors, and they often have little say in which teacher their child gets.

I visited UTLA’s website, expecting to encounter a whiny response to this whole issue.  (I admitted already that I’m biased in favor of testing.)  However, I was impressed by the links they offered with what appear to be legitimate criticisms of the value-added testing model.

This video by Daniel Willingham, Professor of Psychology, makes succinct points about the problems with using test score increases to determine merit pay for teachers.

So I continue to research this issue and I am wondering what others think about it or know about it.  I come from a perspective that has been obviously influenced by what I do every day and the numbers I sort through year after year.  While there are many intervening circumstances that can affect a teacher’s effectiveness, I do believe testing is one measure that should be seriously considered and not dismissed out of hand by unions and teachers.

Advertisements

Responses

  1. While I am whole-hearted NOT for basing hiring, firing, and pay decisions on tests alone, it strikes me as the height of irony that the teachers’ union is completely against testing as one means of determining ability. I mean, c’mon.

    • If it were true that it would be just one of several measures of teacher effectiveness, then I would agree that it is a good piece of the puzzle. However, things like that tend to become the “only” measure – just as it has with our kids. How many bad teachers have your kids had? have you had? really? what made you define them as bad? that is the problem…we have no clear definition of what a good teacher is to compare. there are some things tests don’t measure.

  2. How would it work for certain special ed. teachers?

    • One, I’m not suggesting that student test scores are the issue here. But testing teachers on a regular basis, or say, requiring continued education isn’t necessarily a bad thing. Many other professions monitored by an overaching body require it. Lawyers and doctors come to mind.

      Two, are you saying that special needs children don’t have milestones? That seems like a pretty ineffective process to me.

      • Depends on the child. Hence the “Individual Education Plan.”

        My son did pass a portion of the Compass to start classes at the community college, he would never pass the state tests to graduate from high school. Be aware that the community colleges serve students who have not graduated from high school.

        I just want to know if a special education teacher would be penalized for having students who would never be able to pass a test.

        Have you read Train Go Sorry? The book illustrates that some deaf people do not necessarily process English as well as hearing people, but yet are asked to take the tests in English (when their “native language” is ASL), except that is not a measure of what they know.

        I know that my son’s child neurologist actually apologized for the limitations of the intelligence tests they gave to my son since they were language based. And since my son had a severe speech/language disorder it was not an appropriate measure of his intelligence.

        This was the basis of my question. But your response brings up another argument:

        We used to live near a school school where the special ed. students were very severely disabled, and it was a good year if they did not die (muscular dystrophy was one of those disabilities). (The OT/PT at my son’s school said it was very depressing to work there)

        So would those special ed. teachers be penalized if the milestone of “not dying” was not reached?

      • teachers are required to continue education – at their own expense. Are doctors required to earn credits to keep their certification? Are lawyers? I don’t think there is ANY profession that is REQUIRED to prove that they have earned a certain number of credits each year and the credits have to be in approved subject areas. If there is, please enlighten me.

  3. I know it is an extreme example, but sometimes the outliers that exist need to be pointed out to show the limitations of generalities.

    Why should special teachers be penalized because the students they work with may never pass a test, or even live long enough to graduate? Do they work any less?

    • Chris,

      Your family obviously have special needs, and that is something that needs to be addressed. However, the discussion here is not about the outliers – it’s about the reaction from the UTLA to the idea of whether testing is a means of measuring performance. Again, all I’m saying is that a board of people who monitor a group of professionals that use TESTING as a means measuring student performance, screaming bloody murder about testing as a means of measuring teacher performance, is vaguely Orwellian.

      I get that there are people who don’t fit neatly into categories. And we obviously should have means of accommodating those people. But as a general rule, the UTLA immediately dismissing some sort of rating standard for teachers is ridiculous. It only feeds the people who want to privatize school systems and penalize every teacher that works outside the system.

      Again, I don’t think this should really be about student performance. It should be about measuring teachers’ performance. The UTLA should figure a way to debunk that standardized test scores are a complete answer to judging performance, and I don’t think taking their ball and going home is the way to do it.

      And your remark about dying children, frankly, is so loaded and emotionally charged that it’s offensive. But let’s take it as serious.

      ABSOLUTELY if more kids die under a specific teacher’s or program’s care should we examine that teacher’s methods and competance. Does that teacher or program just take on the hardest cases with the highest level of mortality, or is something else going on?

      • Does that teacher or program just take on the hardest cases with the highest level of mortality, or is something else going on?

        That was one program of severely physically disabled children with several needs (like tubes, etc). Sorry, it was just the extreme example, late at night and due to my experience with the “one size fits all” testing mentality.

        Sorry, again… it is just that sometimes the generalizations tend to get to me.

        Again, there issues with standardized tests, and using them as a metric club to evaluate teacher effectiveness. There is the very real danger of teaching to the test, and then having kids miss out on a whole part of a curriculum because it is not on the test.

        In my hand are the results of my daughter’s High School Proficiency Exam taken in the 10th grade: Reading, Writing and Math. Nothing else.

        Like science. Most standardized tests are on reading and math competence, so science is left out. Especially at the elementary school years.

        Nor is there any history. Passing US History and American Government is required to graduate, so those teachers are required.

        So how do you evaluate teachers whose subjects are not even on the test?

        Then even with the math curriculum, it gets focused on doing the rote calculations, but not understanding the theory behind the methods (my daughter had a conversation with an Italian exchange student who complained the pre-calculus class was more on how to do the problems, but not on the theory!).

        And then there is the problem of figuring out what those “standards” should be. I guess my point is that there are no simple answers.

        (As an aside: my younger children have taken and are taking AP, Advanced Placement, courses. My younger son took AP Calculus AB and BC, AP European History, AP American History and AP American Government, my daughter took AP Human Geography and is now taking AP Japanese —- every single one of those classes were/are taught to the test. To the point that they were given practice tests, it is scary on some of the points, especially in history, that were completely ignored!).

  4. Chris, I’m not sure this line of questioning is relevant to the value-added analysis question. From the LA Times website:

    Is a teacher’s or school’s score affected by low-achieving students, English-language learners or other students with challenges?

    Generally not. By comparing each child’s results with his or her past performance, value-added largely controls for such differences, leveling the playing field among teachers and schools. Research using L.A. Unified data has found that teachers with a high percentage of students who are gifted students or English-language learners have no meaningful advantage or disadvantage under the value-added approach. The same applies to teachers with high numbers of students who are rich or poor.

    Here is the FAQ on the LA Times website about the value-added approach.

    http://projects.latimes.com/value-added/faq/

    In my own experience dealing with data that measures student performance (and therefore by extension teacher effectiveness), the data is used as a starting point to investigate possible issues within the classroom or school.

    And that’s what drives me nuts about the debate surrounding data. Data and data driven decisions are so helpful and powerful, but educators for some reason disdain them. And yes, I agree with the commenter who finds this ironic. I mean, teachers hate tests??? Really? The culture should change, in my opinion, so that we welcome assessment as a way to improve our programs.

    I could go on at length about this, obviously.

    • Sorry. I see you have this passion proper assessment, and you may note that I have an issue with another aspect that is often ignored.

      It is just that it reminded me of the time that the state was starting to require passing of a standardized test to graduate. We were told that they had made no provisions for special ed. students, and that there would be absolutely no accommodations (no readers for the blind, set time limits, etc).

      As it turned out, it was a policy created by a committee that simply did not realize that there was an entire population of students who were more than two standard deviations from the mean, with some federal and state laws that went along with providing them an appropriate education. Plus parents and teachers who were not at all happy with the rigid policy. It got changed (and changed again as standardized testing is under fire by everyone else for their own reasons).

      I am sure that special needs students will not be ignored, nor will their teachers get slighted. No dedicated teacher working with the most difficult students should lose out, and those students should not be left to under qualified staff (which has happened, unfortunately some of my son’s former classmates were victims of a teacher who had a mental breakdown).

  5. Oh…but just one more thing. Here’s another quote from that FAQ page:

    “The Los Angeles Unified School District has had the underlying data in hand for years but has not used them to inform parents — or teachers themselves — about how instructors are doing.”

    As someone who collects data for LAUSD (although it is for the adult division, so a different story than we’re dealing with here), I can say that my own anecdotal experience has shown that we collect much more data than we use. If you’re the person looking at the data every day, you start to feel somewhat Cassandra-ish as you tell administrators, year after year, “Hey, look at our attrition rate. We could really improve our performance by addressing that issue, and then our test scores would perhaps organically improve because our programs would be better.”

    Admin: “Okay, good idea.” But nothing happens.

  6. In my kids’ schools there have been some very good teachers, and some very bad teachers. It used to be very difficult to get rid of some of the bad teachers, and interestingly enough… they were the ones that taught the classes not on the tests (music, social studies, science).

    At a point the school district had a change of focus and actually gave more control over the individual schools to their administration. This meant that the principal, site council (I served on one for a while) and other support personnel had a real voice in who worked in the school. So away went the social studies teacher who “taught” East Asian culture by discussing the recipes, and gone was the biology teacher who mostly played movies. There was much rejoicing.

    Unfortunately the pendulum has swung again to the other side, and the district is taking away local control. So while there are evaluations from each department and administrative staff, again it will be the more effective teachers that will get Reduction in Force notices.

    And RiFs will come from the same office that thought the high school was going to lose students this year, but instead ended up with too many (and they had to create more classes!). Even though the administration of the school warned the district office that their prediction looked suspicious.

    Something must have been lost in all the data they collected. :-/

    • Chris,

      Sorry, for some reason I can’t reply to your replies, so I’m commenting here.

      Thank you for your measured responses. I appreciate your candor (in the rest of the thread as well).

      I think you bring up very valid points about standardized testing. I agree with them. However, my comment was really about how to address these concerns as a unified body. If the teachers’ union just says “go f yourself” anytime anybody questions their methods, they are going to increase the controversy, not alleviate it. The focus will be on the fight between teachers and admin (or whomever is scapegoating teachers at the time).

      I believe that most teachers live for moments when they connect with students, whatever that looks like. I think that they go into the business of teaching for very noble reasons. They are underappreciated and definitely underpaid.

      The goal, obviously, is to teach the best we can, the most we can, and increase the knowledge of young minds. And hopefully increase their desire to think, reason, and learn. What that looks like, I don’t know for sure. However, if teachers and their organizations start an “us against them” game (not that other groups aren’t already doing this), we’re not going to get there. IMO. I do think that defensive posturing on the part of the UTLA probably isn’t helping.

      • True.

        One 7th grade teacher a few years ago actually refused to give the test to his class. I don’t remember what happened to him, but there were parents who supported his actions.

        There has never been any real consensus over the tests, and how they work for teacher effectiveness. In middle to high school it might measure how the school does in general, but not the teachers who teach subjects on the test. This also happens in elementary schools where there are specialists like physical education, art and music (along with the various speech therapists, OT/PTs and reading/math specialists).

        The problem with a “unified” voice may that there really is no such thing. If you satisfy one group, you anger another.

        Take the math curriculum, no one has ever been happy with it. So they mulled it over for a few years and finally settled on one. But one university professor was angered, and he, other parents and some teachers actually sued the school district!

        I personally do not favor one size fits all strategies (I bet you noticed that). I know that standardized tests are a necessary evil, but they can’t be the only rubric.

        I would like to see more local control in the actual school building. Perhaps even letting the teacher choose the book to use in the classroom (from a list! no Of Pandas and People!). But that has its issues, especially around personalities, politics and in-fighting. But they seem more manageable on the school level, versus the district level (and right now the district is not behaving very well, there is a levy coming up and it might fail due to the dissatisfaction with its administration).

        I will also note that whether a school is good or bad does not depend on the “type” of school it is. Public, alternative, parochial, charter, etc are not what is important, it is who is in the building. You hit upon one big thing that makes a good school when you said: “The focus will be on the fight between teachers and admin (or whomever is scapegoating teachers at the time).”

        The better years are when the admin works with the teachers. The others, not so much. There was a mass exodus from a parochial school a few years ago when the new head mistress caused all sorts of conflicts (according to my neighbors that has been rectified).

        (oh, and in the future I will try not to drink and post)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: