An Approach to Specifications Grading: Guest Post by John Ross

I have been involved in a lot of discussions about assessment strategies lately. There is a bit of a swell of young faculty who are rethinking their assessment strategies carefully. For some, this is a first serious step to rethinking their jobs as educators, and for others it is further step into the details of how to be effective.

Today we have a guest post by John Ross of Southwestern University. I met John at the Legacy of R.L. Moore meeting this summer, so I already know he is interested in effective teaching methods. This past weekend he mentioned lightly on twitter that he is using a new assessment setup. I wanted to hear the details, so I invited him to write about it. I am very pleased that he accepted my challenge.


My Version of Specs-Based Grading

by John Ross, Southwestern University
This semester I am running my calculus class using a specifications-based grading system. The decision to do this was made after discovering Robert Talbert’s blog and reading the many informative things he had to say about specs grading. If you’re unfamiliar with this style of grading, I’d recommend starting there (http://rtalbert.org/blog/2015/Specs-grading-report-part-1/).

Continue reading

Linear Algebra Technology Implementation

One of the components of my linear algebra course that has felt like a real struggle is finding a meaningful way to integrate use of technology into the course. By meaningful, I intend something that requires the students to engage with modern computational technology. I’ll have more to say about that below, but a big part of the reason for writing this post is to hash out ideas about what I want to do and how I will do it.

Context: Departmental Student Learning Outcomes

Two or three years ago, the UNI Math Department did a bit of a curriculum review. As part of this, we adopted an official Student Learning Objectives Document (you know, assessment and accountability are everywhere these days) and we discussed tweaks to a few courses to make everything fit.

One of the formal Learning Objectives became this:

Technology specification:

Students will demonstrate basic proficiency with mathematical software. Students will be able to make informed choices about when the use of technology is viable and useful.

And the place we chose to address this learning outcome formally is…Linear Algebra. The main reason for this is that linear algebra is part of our core curriculum: it is part of all three of our major programs. Another reason is the timing: linear algebra comes early enough in each of those programs that we can hope to make use of the technology skills built in later courses, but late enough that we are not impacting many of our client departments heavily with this adjustment.

My Previous Attempts

I have taught linear algebra three times at UNI, once in each of the last three years. Each time I had the revision to our Student Learning Objectives in mind, and I tried to do something to address it. This fit nicely with another project I got involved in: UTMOST is a project funded by the NSF and run through the American Institute of Mathematics focused on adoption issues for open-source textbooks and software in the mathematics undergraduate curriculum. This came about right as I was starting to teach linear algebra and I got involved as a “test site.” Fitting in with the project’s aims, I have been learning how to use Sage: and I have tried to incorporate it into my classroom.

At first, this meant giving some large Sage-based homework assignments. These were not quite projects, but they were stand-alone assignments. This was a bad idea. The size and complexity of the assignments meant that students did not really learn how to deal with using Sage because they procrastinated, got frustrated on deadline, and gave up. I learned the hard way that most of my students have very little sense of how to deal with a computer. Even simple tasks like navigating to a web site and making an account were cause for grief and apprehension. It didn’t help that very few of them attended the introductory workshops I held on how to use the software.

Then incorporating software meant giving out weekly homework assignments as Sage worksheets, with embedded instructions. I worked harder at breaking things down into manageable bits to be learned each week. To get the homework and do it, students would have to open Sage and work with a worksheet right away. I made sure to assign problems that were challenging, but workable if you explored using the computer. As I learned the day before the first midterm, most of the students got as far as logging in, and then printed the worksheets and attempted to work out all of the tasks with a pencil and paper.

Last term, I again required students to get their assignments through use of Sage. This time, we used the new cloud service, and I made dedicated tutorial worksheets to go with each reading. I started assigning tasks that explicitly required using the software. (Use Sage to...) This worked better. I gave a take-home midterm that required using the computer, and a few did quite well. But I still found many students avoiding the computer like the plague. I had one admit to me eight weeks into the term that she never bothered to figure out how to log in, and a friend in class sent her a pdf copy of the assignment for each class meeting.

Clearly, we are failing to meet the spirit of the learning goal above.

Going Forward

So it is time for a new plan. I had two disastrous failures, and one mixed experience. But this coming fall I will have two sections of linear algebra, and the curriculum changes that we have proposed officially take effect. It is time for a new, better-informed plan.

Sharpening the Student Learning Objectives

I like the Student Learning Objective statement above. (I helped write it.) But I have come to realize it is inadequate. I don’t have the power to rewrite it unilaterally. But as most of my department seems to be of the opinion that I should just figure this out and do it, I have taken it upon myself to add depth and structure for future use.

First, I added some specific, measurable goals.

Student Learning Goals associated with the Technology Specification

Goal 1: Students can name multiple examples of computer algebra systems for doing work in mathematics.

Goal 2: Students can use one system at the level of beginner, by starting the system, opening a worksheet or development environment, performing basic computations, and making plots.

Goal 3: Students can find information about the capabilities of their chosen system to determine if the system has a particular feature or functionality built-in.
Students can access documentation on how to use unfamiliar features or functionality, and then use that information to make use of that feature.

Goal 4: Students can describe circumstances where use of a computer is a reasonable or appropriate choice to further work in mathematical investigation, and identify features of the circumstance which call for the computer-based work.

I hope these will suit my colleagues. I have asked a few of them for comment, but not heard back much. I choose to believe that this is because it is officially summer.

The Plan for Assessment

The goals don’t mean much if I don’t assess them. So, I plan the following set-up. At the start of the term, I will give the students detailed information about what is expected of them and resources to learn about how to meet those expectations (a simple page on the course web-site with links, a collection of short video tutorials, and other things). Of course, I will also keep using the software in class myself, and I will still give the students the short tutorials that go with the daily assignments.

We will begin the term by using embedded Sage cells in course web pages, but transition to forcing students to log in to the SageMathCloud to get their work.

A few weeks into the term, students will be directed to schedule a short appointment (10-15 minutes) with me, or perhaps the grader, to do a “gateway assessment.” The gateway exam will be an all-or-nothing event. Either the student demonstrates competence on all of the goals, or she does not. I expect that an interview should end as soon as a student fails to demonstrate competence at any stage–there should be no hemming and hawing over these tasks. We will conduct the assessments while sitting at a computer station. I think that the labs in my building are more than sufficient for this. During the interviews, we will ask questions aimed directly at the goals outlined above.

I have not, yet, decided how much data to keep from these assessments. At a bare minimum, I need to keep a record of which students pass the assessment. But I think I might keep a spreadsheet which records each attempt, the date of those attempts, and how far into the assessment a student gets.

The Assessment Script

The real details hide in the questions I ask to check on my goals. To keep things running smoothly, I have written an “assessment script.” Each question in the script is explicitly tied to one of the four goals. It looks like this:

Technology Specification SOA Script

The following are questions to be asked in determining if a student has met the goals of the Technology Specification.

general questions

[G1.] Can you name some computer algebra systems? How many of those do you know how to use?

[G2.] Choose one of these that you know how to use. Open the program/sign in to the service and then open a new worksheet/start up the computational environment.

[G2.] Use the software to find the first 12 decimal digits of the number 2pi/3 -sqrt(e).

linear algebra specific questions
(replace with something appropriate if used in a different course)

[G2.] Define two 3-vectors a and b and add them.

[G2.] Define a 3×3 matrix A. Use the system to find the determinant and rank of this matrix.

[G2.] Use the computer algebra system to solve the system of linear equations represented by Ax = b.

[G2.] Use the computer algebra system to plot one of the equations from the system Ax = b.

more general questions

[G2.] Save this worksheet/session so that you can access it later.

[G2.] Find a way to share this work with me. You can download and print, email, or use any other way that this system allows you to share your work. How many ways can you share this work?

[G4.] Give an example of a time when you might want to use this computer algebra system instead of just a pen and paper. Explain why this is a time that choice should be made.

[G3.] There is a mathematical construction called <insert new term here>. Show me how you would find out if your chosen computer algebra system has any functionality related to <new term>. Now that you see there is some functionality, show me how you can access the help or documentation of this system to learn how this bit of the software works. Now that you have the documentation, show me how to use this functionality.

For linear algebra, a possible list of ideas for the <new term> includes: minimal polynomial, eigenvector, Cholesky decomposition, polar decomposition, cross product, Jordan form, positive definite. This is just a sampler. The important thing is to choose something new to the student.

Resources I Should Provide

I have started compiling a list of resources I should make available to the students.

Some Discussion on a web page

I will make a page on my course web site that discusses possible computer algebra systems, including Maple, Mathematica, Matlab, graphing calculators, etc.

I will lay out my reasons for choosing Sage, and provide links to resources for using it:

  • the official Sage web site,
  • online documentation,
  • the sage cell server,
  • the cloud service,
  • a few tutorials (from lengthy to short: official one, the SDSU tutorial, my beginner’s tutorial)
  • my youtube channel with short tutorial videos

Video Tutorials to Make

I have been impressed with the short video tutorials that Vincent Knight has made for his students. And recently William Stein made a few that were similar in their tight focus and short length. This seems a good approach: Here is something you want to know how to do, described clearly with an example in two minutes or less.

I want to make some of these, or steal link to some of these, all of which are Sage-specific:

  • How to make an account on SageMathCloud
  • How to use git to pull down all of the course materials
  • How to make a new worksheet and evaluate some cells (basic arithmetic)
  • How to do some basic plotting 2d
  • basic plotting 3d
  • How to make and manipulate vectors and matrices
  • How to share work: printing a pdf, sharing a project with another user, downloading a worksheet
  • How to get help: tab completion, the ? and ?? methods.
  • searching Sage documentation and source code

Well, two thousand words seems like enough. Thanks for those of you who stuck it out so far into this. I welcome all constructive comments and any questions.

A reflection on “Assessment Interviews, Phase 2”

I have spent a large portion of today in one-on-one conversation with the students in my Euclidean Geometry course. To prepare the students for these meetings, I asked them to complete a one page reflection paper, with this prompt. If you don’t want to click through, I basically ask the students to read through the “standards for assessment,” which is just a fancy name for my student learning goals, and do a self-assessment. Then I want them to make a plan of action for improvement during the next three weeks.

The striking part is the strength of the negative correlation between student self-assessment and my assessments.

Students who I recognize as having developed strong skills come it with focused critiques and tight plans for how to improve.

Students who I recognize as having not yet demonstrated many of our foundational skills show up with some confidence that they are doing everything just fine, and weak plans for self-improvement.

(This relationship is not perfect. Some students were spot on, of course.)

I have enough experience that I expected this, but to watch in unfold all day was really something.

Opening Week for a Moore Method Course: Getting Comfortable

I am teaching iteration number ten of my Modified Moore Method Euclidean Geometry course. This semester I am making an effort to refocus on the basics: managing and mentoring the students as much as I can.

At this point, my theorem sequence is very stable. (I am no longer surprised much by what happens in this course.) This allows me to work on the other aspects of the course. I feel like I have started to let some important things go in the last year or so, and now I want to sharpen up. What has been lacking? I don’t think I have kept on top of the students to keep them engaged as well as I might. And I don’t think I have done a good job selling the method of instruction, either.

So, I was much more deliberate about introducing myself to each of my sixteen students on the first day. I have been very explicit about my expectations and my willingness to help them meet those expectations (which are rather high). And I will be making a conscious effort to check in with as many students as possible each day.

The first week was a rousing success, I think. Each day we got at least one theorem. We have already set the expectation for what counts as an argument. (Well, surely, there is still some work to be done.) The class has made two conjectures. We took some time to discuss some basic points of what acceptable writing will look like. I even successfully navigated our first potential difficult situation and found something positive in it. All in all, I am feeling pretty good about this.

I think our next test comes when we have to finish conjecture 1.1. They haven’t addressed the second statement in that, yet.

And sometime next week I will have to steal ten minutes to talk with them about my Standards Based Assessment experiment for the term.

Standards Based Assessment for a Moore Method Course

Motivation

I have been working on developing a reasonable assessment model for my IBL Euclidean Geometry Course for a while now. I have several reasons for this:
1. It would be more fair, and better for my students, if I found a way to communicate with them about their progress. At the very least, I need to open the line of communication, so students feel they can have a conversation with me about how things are going.
2. So far, I have been going with a “you will have to trust me” approach. I have gotten away with it. But someone who wants to raise hell will make my undocumented life difficult.
3. This class is conducted as a lightly modified Moore Method course. Standard assessment with homework, quizzes and exams just doesn’t feel right.
4. The accountability movement is coming. Sooner or later, I will have to deal with a top-down mandate to deal with how I assess my students, and how I assess my teaching. I choose to start, on my own, with the parts I can control before that pressure gets here. First up: how I assess students.

The Main Idea

I will try to use a Standards Based Assessment scheme. I will attempt to focus on this mainly as a feedback mechanism. Grades will only happen to the minimal extent that is required.

What didn’t work well enough, and why.

I tried to implement a simple SBG/SBAR scheme in each of the last two semesters. Neither worked because I had not found a method of dealing with the administrative details. At first, I asked too much of myself. Then, I asked even more of myself, but on deadlines. Ugh.

What is working

I am happy with my set of standards (read that as learning goals). I am very proud that they are weighted toward process goals: what one does and how one behaves as a mathematician. This is intentional—I want students to become acculturated to doing mathematics, and to acquire some of a mathematicians habits of mind.

A New Attempt

For next semester, I have devised a two-prong approach to administering a standards based assessment mechanism.

The First Prong: Face to face meetings

In order to make for better communication about expectations, I will meet with each student individually every three weeks. This will involve splitting the class. I will meet with half one week, half the next, and then take a week break.

Before each meeting, I expect the student to write a one page reflection about their progress in the course. To tighten this process up, I have written specific prompts to which the students must respond. This must be done before the meeting. It can either be sent to me electronically, or it can be brought to the meeting on paper, but it has to be done before the conversation. Really, the paper is not important. But the time for reflection is crucial. The meeting could too easily be wasted without it.

Second Prong: Professional Feedback at each Assessment Opportunity

Each time a student participates in some sort of assessment opportunity (a presentation at the board, turning in a written paper), I will provide feedback. I have a little electronic system built (with the help of my friend Stephen Hughes) using a Google Docs form/spreadsheet/script combo. I have a web form into which I will type comments. When I click the “enter” button, my comments are saved in a spreadsheet, emailed to me, and emailed to the students.

It is too much to manage class and write out feedback at the same time, so I will be doing this during the hour after my class meeting. I normally take time to convert my notes into a blog post for the students anyway. Now I will just add a little bit to the “post meeting decompression” that I do.

What is left to do?

I need to think some more about how I will provide feedback. I want this to be a narrative process, but what are my aims? What constraints should I observe?

That should be my next post. 🙂

Where is all of my stuff?

Well, I keep a blog for the students, and it has a page all about assessment. Go have a look. Not all of the links are live, yet, but they will be at the appropriate time of the semester.

In the end, what about grades?

Here, I have no substantive changes, but Ed Parker has pushed me a bit…

My Problem with Assessment: A Rant

On a plane flight earlier this summer I was trying to get this out of my system. I am absolutely bedeviled by assessment.


I don’t like grading. Well, no one enjoys marking papers. It is tedious and deflating work—endless stacks of repetition makes the tedium; focusing on errors, misconceptions, and failures leaves a sour taste in your mouth.

But I don’t really want to complain about marking papers, per se. Much good can come from interacting thoughtfully with written student work. A well-crafted assignment and thoughtful comments from an instructor provide students with the feedback required to generate reflection and intellectual growth.

The part I take issue with is the grading by assigning some sort of score. One might use points, one might use letters: I want neither.

I suppose that what would make me happiest is some sort of narrative dialogue of evaluation. That way, the students and I can keep our eyes on what is important—striving for personal improvement.

What is it about grades that I dislike? They poison the enterprise. Focus shifts from learning for its own sake to “making a grade.” Students perceive school as a game and then they “play it” instead. This leads to point grubbing. I am too lazy to look for the links: but I have heard of studies demonstrating (1) that students will ignore comments if a paper has any kind of summative mark on it, and (2) that enjoyment in an activity dissipates after being paid to do that activity.

The problem remains that everyone expects grades. Whole subsystems of modern tertiary education depend upon the A-F system we use: scholarships, graduation requirements, grad school apps, the dean’s list, etc…

So, what can be done? How do I turn the system on itself to my advantage?

Some Ideas

  1. Provide only narrative evaluations unless pressed.
  2. Avoid legalistic “contract-type” language on a syllabus.
  3. Give meaningful feedback often and in person.
  4. Share with students my goals for their personal growth.
  5. Design a system where students can see this in a positive light. Sell it.
  6. Ensure my “standards” cannot be gamed. That is, if a student really sees the external motivation of a certain grade as primary, then an attempt to structure work to improve to that grade should lead to behaviors for personal growth and authentic learning. (Whatever that means.)

So, I’ll have to put together something coherent in this direction. Soon, I’ll share what I have been doing.


Added After Publication: Well, I didn’t expect this to be the most popular item today, but twitter seems to like this one. I did get this

I admit, there is a valid point here. I don’t actually have a problem with the fact that we do assessment. But I still have a problem with how I do assessment, so I am going to leave the title alone.

standards based confusion

Today I set up a digital grade book for Euclidean Geometry. This was different from the usual set up (get student names and ID numbers, make columns for assessments) because I am trying Standards Based Assessment.

I feel like I got a handle on the technology. UNI has a Google Apps set up, so I opened up Google Drive and made a spreadsheet/form combination. The form is a simple webpage with spots for me to enter assessment data, and when I hit submit, it records the stuff as a row in my spreadsheet and sends me an email copy. Google has a convenient collection of basic question types that makes it relatively easy to get rolling.

In this arrangement, a row is an assessment, not a student. And a column is a standard, not an assessment. So, where do he students go? Well, that is one of the data points associated to an assessment, so student name is a column, too.

I fiddled around with the Google Scripts capabilities until I made it so that when a form is submitted, the student gets an email, too. For this, I spent time on three things: man pages for Drive, Web searches for things other people have done with forms and scripts, and refreshing my knowledge of javascript syntax. (It has been a long time since I last wrote javascript code.) So much for my afternoon.

Anyway, that all works. I have a simple form, and submitting it does three things: archives to my spreadsheet, emails me a confirmation, and sends the student my evaluation. I am generally happy with that arrangement.

Now, what is wrong? (There is always something wrong.) I tried running through a batch of summative assessment from the first four weeks. I am 17/25 of the way done, and it doesn’t feel right. Somehow the work I just did feels unmoored from reality. It lacks the specificity of being tied explicitly to a particular piece of work performed by the student. And I feel like I am writing the same thing for each person.

Tomorrow I will take a few minutes and discuss with my students and see if we can come up with a better way. We need something that feels authentic and valuable to everyone.

claimtoken-51214ad515963