An Approach to Specifications Grading: Guest Post by John Ross

I have been involved in a lot of discussions about assessment strategies lately. There is a bit of a swell of young faculty who are rethinking their assessment strategies carefully. For some, this is a first serious step to rethinking their jobs as educators, and for others it is further step into the details of how to be effective.

Today we have a guest post by John Ross of Southwestern University. I met John at the Legacy of R.L. Moore meeting this summer, so I already know he is interested in effective teaching methods. This past weekend he mentioned lightly on twitter that he is using a new assessment setup. I wanted to hear the details, so I invited him to write about it. I am very pleased that he accepted my challenge.


My Version of Specs-Based Grading

by John Ross, Southwestern University
This semester I am running my calculus class using a specifications-based grading system. The decision to do this was made after discovering Robert Talbert’s blog and reading the many informative things he had to say about specs grading. If you’re unfamiliar with this style of grading, I’d recommend starting there (http://rtalbert.org/blog/2015/Specs-grading-report-part-1/).

Continue reading

Advertisements

Integrated Course Design for Euclidean Geometry, III

Today I finished grades for all of my Fall 2014 courses, so it is time to get back to planning next term. As my last “pre-holiday” bit of work, I’ll finish the course design template for Euclidean Geometry following Fink. I previously did the initial design phase which consisted of “building strong primary components” in Steps 1-5.

Intermediate Phase: Assemble the Components into a Coherent Whole

Step 6: Create a Thematic Structure for the Course

Fink encourages us to find 4-7 segments of the course, each focusing on key concepts or topics. I think I have five, or maybe four. It depends on how you want to count them. The first one on this list might be two things.

  1. Polygons, and the axiomatic method, conjectures, and definitions. This is mostly about using triangles to study other polygons. But there is a little mini-unit on arguments with parallel lines mixed in here.
  2. Circles
  3. Straightedge and Compass Constructions (as an efficiency game!)
  4. Area and the regular pentagon

Segment 1 is really long, it can take half a semester. Then segments 2 and 3 are shorter. We almost never finish segment 4 completely, but the fastest class I had finished 4 and started another (bonus) segment!

Step 7: Instructional Strategy

I have no desire to make changes here. For ten iterations of this course I have used a flavor of Inquiry-Based Learning called a Modified Moore Method. Really, I have done something called The Extreme Moore Method (EMM), which I have written about before. Students will spend their time out of class finding and composing arguments for conjectures. We will spend class time presenting and critiquing this work. Then students that will write papers for the class journal. Oh, just go read the other post.

Step 8: The Overall Scheme of Learning Activities

Here Fink wants the instructor to think about the variety of activities the students should do, and when. Also, how does the sequence of tasks for before, during, and after class meetings mesh together? This is all pretty well decided with my EMM.

Final Design Phase

I won’t be able to finish all of this tonight in detail, but I can generally get through the rest of the guide.

Step 9: How are you going to grade?

Well, I have been thinking about this a lot lately. For years. Next term, I will try something called “Specifications Grading” following the work of Linda Nilson. I have been part of conversation about this for a few weeks now. Go find the work Robert Talbert has been doing in this direction for a list of places to start.

I will wrap up work on this and make a new post soon.

Step 10: What Could Go Wrong?

I have run this class often enough that I only have one worry: The new grading system is (a) not a solution to my problem, or (b) actively messes with the other parts of class which were working fine. I guess (a) isn’t too big a deal. I’ll just keep looking in that case. But I worry about (b). So far, the assigning of grades has been kinda vague, and this pushes students to keep working. (It is just like the real mathematics community.) And I want to make things more reliable and, I hope, more “fair.” But I don’t want students to start micro-managing publication counts instead of trying to solve more problems. For now, I need an experiment.

Step 11: Make a Plan for Communicating with Students

Fink writes this as if the big deal is to just write a syllabus. Well, yes, and no. I see these tasks ahead of me.

  1. Rewrite my syllabus
  2. Rework my first day handout (which is like a mini textbook)
  3. Update the course web page (which is like a digital syllabus and record of class).
  4. Flesh out the detailed specifications for the grading system. In particular, this means I will have to update

* The class journal style guide
* the instructions for referees
* my grading policy document
* specifications for the “non-mathematical” writing assignments (reflections and essays)

Step 12: Make a Plan to Assess your Teaching

This is a challenge. How many of us do this with such foresight? Well, my plan is this: I will use a simplified SALG instrument at final exam time to assess student satisfaction and understanding of the new grading system.

Linear Algebra Technology Implementation

One of the components of my linear algebra course that has felt like a real struggle is finding a meaningful way to integrate use of technology into the course. By meaningful, I intend something that requires the students to engage with modern computational technology. I’ll have more to say about that below, but a big part of the reason for writing this post is to hash out ideas about what I want to do and how I will do it.

Context: Departmental Student Learning Outcomes

Two or three years ago, the UNI Math Department did a bit of a curriculum review. As part of this, we adopted an official Student Learning Objectives Document (you know, assessment and accountability are everywhere these days) and we discussed tweaks to a few courses to make everything fit.

One of the formal Learning Objectives became this:

Technology specification:

Students will demonstrate basic proficiency with mathematical software. Students will be able to make informed choices about when the use of technology is viable and useful.

And the place we chose to address this learning outcome formally is…Linear Algebra. The main reason for this is that linear algebra is part of our core curriculum: it is part of all three of our major programs. Another reason is the timing: linear algebra comes early enough in each of those programs that we can hope to make use of the technology skills built in later courses, but late enough that we are not impacting many of our client departments heavily with this adjustment.

My Previous Attempts

I have taught linear algebra three times at UNI, once in each of the last three years. Each time I had the revision to our Student Learning Objectives in mind, and I tried to do something to address it. This fit nicely with another project I got involved in: UTMOST is a project funded by the NSF and run through the American Institute of Mathematics focused on adoption issues for open-source textbooks and software in the mathematics undergraduate curriculum. This came about right as I was starting to teach linear algebra and I got involved as a “test site.” Fitting in with the project’s aims, I have been learning how to use Sage: and I have tried to incorporate it into my classroom.

At first, this meant giving some large Sage-based homework assignments. These were not quite projects, but they were stand-alone assignments. This was a bad idea. The size and complexity of the assignments meant that students did not really learn how to deal with using Sage because they procrastinated, got frustrated on deadline, and gave up. I learned the hard way that most of my students have very little sense of how to deal with a computer. Even simple tasks like navigating to a web site and making an account were cause for grief and apprehension. It didn’t help that very few of them attended the introductory workshops I held on how to use the software.

Then incorporating software meant giving out weekly homework assignments as Sage worksheets, with embedded instructions. I worked harder at breaking things down into manageable bits to be learned each week. To get the homework and do it, students would have to open Sage and work with a worksheet right away. I made sure to assign problems that were challenging, but workable if you explored using the computer. As I learned the day before the first midterm, most of the students got as far as logging in, and then printed the worksheets and attempted to work out all of the tasks with a pencil and paper.

Last term, I again required students to get their assignments through use of Sage. This time, we used the new cloud service, and I made dedicated tutorial worksheets to go with each reading. I started assigning tasks that explicitly required using the software. (Use Sage to...) This worked better. I gave a take-home midterm that required using the computer, and a few did quite well. But I still found many students avoiding the computer like the plague. I had one admit to me eight weeks into the term that she never bothered to figure out how to log in, and a friend in class sent her a pdf copy of the assignment for each class meeting.

Clearly, we are failing to meet the spirit of the learning goal above.

Going Forward

So it is time for a new plan. I had two disastrous failures, and one mixed experience. But this coming fall I will have two sections of linear algebra, and the curriculum changes that we have proposed officially take effect. It is time for a new, better-informed plan.

Sharpening the Student Learning Objectives

I like the Student Learning Objective statement above. (I helped write it.) But I have come to realize it is inadequate. I don’t have the power to rewrite it unilaterally. But as most of my department seems to be of the opinion that I should just figure this out and do it, I have taken it upon myself to add depth and structure for future use.

First, I added some specific, measurable goals.

Student Learning Goals associated with the Technology Specification

Goal 1: Students can name multiple examples of computer algebra systems for doing work in mathematics.

Goal 2: Students can use one system at the level of beginner, by starting the system, opening a worksheet or development environment, performing basic computations, and making plots.

Goal 3: Students can find information about the capabilities of their chosen system to determine if the system has a particular feature or functionality built-in.
Students can access documentation on how to use unfamiliar features or functionality, and then use that information to make use of that feature.

Goal 4: Students can describe circumstances where use of a computer is a reasonable or appropriate choice to further work in mathematical investigation, and identify features of the circumstance which call for the computer-based work.

I hope these will suit my colleagues. I have asked a few of them for comment, but not heard back much. I choose to believe that this is because it is officially summer.

The Plan for Assessment

The goals don’t mean much if I don’t assess them. So, I plan the following set-up. At the start of the term, I will give the students detailed information about what is expected of them and resources to learn about how to meet those expectations (a simple page on the course web-site with links, a collection of short video tutorials, and other things). Of course, I will also keep using the software in class myself, and I will still give the students the short tutorials that go with the daily assignments.

We will begin the term by using embedded Sage cells in course web pages, but transition to forcing students to log in to the SageMathCloud to get their work.

A few weeks into the term, students will be directed to schedule a short appointment (10-15 minutes) with me, or perhaps the grader, to do a “gateway assessment.” The gateway exam will be an all-or-nothing event. Either the student demonstrates competence on all of the goals, or she does not. I expect that an interview should end as soon as a student fails to demonstrate competence at any stage–there should be no hemming and hawing over these tasks. We will conduct the assessments while sitting at a computer station. I think that the labs in my building are more than sufficient for this. During the interviews, we will ask questions aimed directly at the goals outlined above.

I have not, yet, decided how much data to keep from these assessments. At a bare minimum, I need to keep a record of which students pass the assessment. But I think I might keep a spreadsheet which records each attempt, the date of those attempts, and how far into the assessment a student gets.

The Assessment Script

The real details hide in the questions I ask to check on my goals. To keep things running smoothly, I have written an “assessment script.” Each question in the script is explicitly tied to one of the four goals. It looks like this:

Technology Specification SOA Script

The following are questions to be asked in determining if a student has met the goals of the Technology Specification.

general questions

[G1.] Can you name some computer algebra systems? How many of those do you know how to use?

[G2.] Choose one of these that you know how to use. Open the program/sign in to the service and then open a new worksheet/start up the computational environment.

[G2.] Use the software to find the first 12 decimal digits of the number 2pi/3 -sqrt(e).

linear algebra specific questions
(replace with something appropriate if used in a different course)

[G2.] Define two 3-vectors a and b and add them.

[G2.] Define a 3×3 matrix A. Use the system to find the determinant and rank of this matrix.

[G2.] Use the computer algebra system to solve the system of linear equations represented by Ax = b.

[G2.] Use the computer algebra system to plot one of the equations from the system Ax = b.

more general questions

[G2.] Save this worksheet/session so that you can access it later.

[G2.] Find a way to share this work with me. You can download and print, email, or use any other way that this system allows you to share your work. How many ways can you share this work?

[G4.] Give an example of a time when you might want to use this computer algebra system instead of just a pen and paper. Explain why this is a time that choice should be made.

[G3.] There is a mathematical construction called <insert new term here>. Show me how you would find out if your chosen computer algebra system has any functionality related to <new term>. Now that you see there is some functionality, show me how you can access the help or documentation of this system to learn how this bit of the software works. Now that you have the documentation, show me how to use this functionality.

For linear algebra, a possible list of ideas for the <new term> includes: minimal polynomial, eigenvector, Cholesky decomposition, polar decomposition, cross product, Jordan form, positive definite. This is just a sampler. The important thing is to choose something new to the student.

Resources I Should Provide

I have started compiling a list of resources I should make available to the students.

Some Discussion on a web page

I will make a page on my course web site that discusses possible computer algebra systems, including Maple, Mathematica, Matlab, graphing calculators, etc.

I will lay out my reasons for choosing Sage, and provide links to resources for using it:

  • the official Sage web site,
  • online documentation,
  • the sage cell server,
  • the cloud service,
  • a few tutorials (from lengthy to short: official one, the SDSU tutorial, my beginner’s tutorial)
  • my youtube channel with short tutorial videos

Video Tutorials to Make

I have been impressed with the short video tutorials that Vincent Knight has made for his students. And recently William Stein made a few that were similar in their tight focus and short length. This seems a good approach: Here is something you want to know how to do, described clearly with an example in two minutes or less.

I want to make some of these, or steal link to some of these, all of which are Sage-specific:

  • How to make an account on SageMathCloud
  • How to use git to pull down all of the course materials
  • How to make a new worksheet and evaluate some cells (basic arithmetic)
  • How to do some basic plotting 2d
  • basic plotting 3d
  • How to make and manipulate vectors and matrices
  • How to share work: printing a pdf, sharing a project with another user, downloading a worksheet
  • How to get help: tab completion, the ? and ?? methods.
  • searching Sage documentation and source code

Well, two thousand words seems like enough. Thanks for those of you who stuck it out so far into this. I welcome all constructive comments and any questions.

A reflection on “Assessment Interviews, Phase 2”

I have spent a large portion of today in one-on-one conversation with the students in my Euclidean Geometry course. To prepare the students for these meetings, I asked them to complete a one page reflection paper, with this prompt. If you don’t want to click through, I basically ask the students to read through the “standards for assessment,” which is just a fancy name for my student learning goals, and do a self-assessment. Then I want them to make a plan of action for improvement during the next three weeks.

The striking part is the strength of the negative correlation between student self-assessment and my assessments.

Students who I recognize as having developed strong skills come it with focused critiques and tight plans for how to improve.

Students who I recognize as having not yet demonstrated many of our foundational skills show up with some confidence that they are doing everything just fine, and weak plans for self-improvement.

(This relationship is not perfect. Some students were spot on, of course.)

I have enough experience that I expected this, but to watch in unfold all day was really something.

An Assessment Idea: Git

I just had a short conversation with Dave Grant. Dave and I started at UNI at the same time, we have kids of similar ages, so I saw him tonight when he came to pick up my son for a sleepover. While the kids were playing we chatted about “stuff,” and the topic of assessment came up.

At this point, I am talking about assessment of a program, or its pieces, rather than assessment of students. Dave mentioned that he had described a process for several of the Languages and Literature courses involving keeping drafts and comments on those drafts as evidence of student work and improvement.

It immediately made me think of Git. Basically, the problem of documenting all of these things is one about version control. Computer scientists have solved this problem (and more).

So, is it crazy to imagine a world in which college students are asked to use a version control system to document their work?

This doesn’t solve the problem of how to take that data and turn it into some useful measurement of “success.” Still, I think it might be useful.

Tell me why I am wrong.

Meditations on Feedback

I am taking part in a “Talking Teaching and Learning” group, and my homework this week was to think about the last few details of my new assessment structure for Euclidean Geometry. In particular, how will I handle the “regular, daily feedback” part of the process?


So, if I am to provide regular feedback to my students at “assessment opportunities” they take, how shall I do it? I want this to be meaningful and effective. And it would be nice if it didn’t consume my working time.

I think I will try a format beloved by politicians: I will ask and answer my own questions. If any of you wishes to play investigative journalist and ask other questions that I should be forced to answer, go hit the comment box. I would like to play.

What counts as an assessment opportunity?

Any student presentation, meaningful engagement in class discussion, a discussion with me outside of class where I learn something. Those things count as opportunities for me to assess student performance that don’t necessarily have written feedback attached to them. In each case there is plenty of verbal feedback from classmates–but I don’t always participate. In fact, I prefer to leave it to the students.

Why are written papers not on this list?

Students will get feedback in the form of a referee report on each paper. I am not as concerned about providing more structured feedback here because I feel it is adequately covered.

Why do you prefer to leave the process of verbal feedback to the students?

One of the skills I am trying to encourage is the ability to evaluate arguments critically and thoughtfully.

If there is a reason to leave the verbal feedback to the students, might written feedback from the instructor corrupt this process?

Oh, yes. That is my main worry.

How can you avoid this trouble?

um. uh. [blink. blink.]

I hope this writing will spur me to some ideas about that…

What are the goals for this written feedback?

I want to focus student attention on some aspects of what they did. Ideally, quality feedback should help speed up a student’s process of improvement by directing his or her attention to something concrete.

What kind of constraints are you going to impose upon yourself?

I am a constructivist at heart. The student must come to grips with the material and how to do it. Each one should do this on his or her own terms. One idea would be to give feedback by asking questions.

I am just not sure what kinds of questions I would ask that are detached from the process of running a class meeting. We handle lots of things in class, and I almost always do it by asking questions. Maybe I will just reiterate some of the unanswered questions. That doesn’t feel like a very good answer.

Another idea is to use a sandwich approach: mention something positive, make a suggestion for improvement, reiterate the positive outcomes. And be relentlessly optimistic.

Now I’ve run out of questions. So.

I think whatever I do will have to play to my strengths. I am at my best when I split my time as a cheerleader, mentor, and coach. Students are capable of amazing things, and sometimes they just need for me to believe in them and expect it out of them. Sometimes they just need a little bit of commiserating about how frustrating it is to do mathematics. Sometimes they need a concrete suggestion of what to do when they are stuck and at their wit’s end.

That was unsatisfying.

Here we are. 500+ words in, and no answers I feel wonderful about.

Never mind, time for some unbridled confidence.

When I got into IBL teaching, I recognized that a major asset I had was hubris. I just believed that I could do this. Usually it works.

What? So I will have to help each student in as individual a way as possible, thinking on
my feet and being careful about everyone’s feelings? Why should I worry? I can do that.
I’ll think about this some more, and just try to roll with it.

I don’t feel like I finished my homework.

Standards Based Assessment for a Moore Method Course

Motivation

I have been working on developing a reasonable assessment model for my IBL Euclidean Geometry Course for a while now. I have several reasons for this:
1. It would be more fair, and better for my students, if I found a way to communicate with them about their progress. At the very least, I need to open the line of communication, so students feel they can have a conversation with me about how things are going.
2. So far, I have been going with a “you will have to trust me” approach. I have gotten away with it. But someone who wants to raise hell will make my undocumented life difficult.
3. This class is conducted as a lightly modified Moore Method course. Standard assessment with homework, quizzes and exams just doesn’t feel right.
4. The accountability movement is coming. Sooner or later, I will have to deal with a top-down mandate to deal with how I assess my students, and how I assess my teaching. I choose to start, on my own, with the parts I can control before that pressure gets here. First up: how I assess students.

The Main Idea

I will try to use a Standards Based Assessment scheme. I will attempt to focus on this mainly as a feedback mechanism. Grades will only happen to the minimal extent that is required.

What didn’t work well enough, and why.

I tried to implement a simple SBG/SBAR scheme in each of the last two semesters. Neither worked because I had not found a method of dealing with the administrative details. At first, I asked too much of myself. Then, I asked even more of myself, but on deadlines. Ugh.

What is working

I am happy with my set of standards (read that as learning goals). I am very proud that they are weighted toward process goals: what one does and how one behaves as a mathematician. This is intentional—I want students to become acculturated to doing mathematics, and to acquire some of a mathematicians habits of mind.

A New Attempt

For next semester, I have devised a two-prong approach to administering a standards based assessment mechanism.

The First Prong: Face to face meetings

In order to make for better communication about expectations, I will meet with each student individually every three weeks. This will involve splitting the class. I will meet with half one week, half the next, and then take a week break.

Before each meeting, I expect the student to write a one page reflection about their progress in the course. To tighten this process up, I have written specific prompts to which the students must respond. This must be done before the meeting. It can either be sent to me electronically, or it can be brought to the meeting on paper, but it has to be done before the conversation. Really, the paper is not important. But the time for reflection is crucial. The meeting could too easily be wasted without it.

Second Prong: Professional Feedback at each Assessment Opportunity

Each time a student participates in some sort of assessment opportunity (a presentation at the board, turning in a written paper), I will provide feedback. I have a little electronic system built (with the help of my friend Stephen Hughes) using a Google Docs form/spreadsheet/script combo. I have a web form into which I will type comments. When I click the “enter” button, my comments are saved in a spreadsheet, emailed to me, and emailed to the students.

It is too much to manage class and write out feedback at the same time, so I will be doing this during the hour after my class meeting. I normally take time to convert my notes into a blog post for the students anyway. Now I will just add a little bit to the “post meeting decompression” that I do.

What is left to do?

I need to think some more about how I will provide feedback. I want this to be a narrative process, but what are my aims? What constraints should I observe?

That should be my next post. 🙂

Where is all of my stuff?

Well, I keep a blog for the students, and it has a page all about assessment. Go have a look. Not all of the links are live, yet, but they will be at the appropriate time of the semester.

In the end, what about grades?

Here, I have no substantive changes, but Ed Parker has pushed me a bit…