Tag Archives: statistics

Student Affairs and Assessment – The Tale Continues

Here’s part 3 in my ongoing tale of bringing the kind of student affairs assessment that I want to our student activities office.

If you need to catch up, here’s Part 1 and Part 2.

When we last left the tale, I had been given notice by our IT guy that not only were we capable of storing and saving the data that I wanted, we already had the software that we needed present on our computers and, not only that, we were already using it, just not correctly.  I was more amused than anything, but not overly surprised.  Given an environment of high turnover, a system that is fairly complex, and an overall student affairs fear of data, I found it not altogether shocking that this information would have disappeared somewhere along with the names of the guys who shot Tupac and Biggie and the Iraqi WMD’s.

So, Chapter 3 starts where the IT guy warns me that he was nearing the end of his effectiveness, as, most likely, we didn’t have access to the student database that we needed to cross reference the IDs that we swiped so that we could actually make something reasonable out of the data other than just a collection of scrabble like number combinations.  Quite literally, the data we were collecting would be useless without access to this database.

Let’s pause here for a little personal interlude.  After we’d figured out that we could in fact collect the IDs with our swipes and save the information, I was feeling particularly empowered.  I went across the hall to @curtistmed’s office and immediately told him that, not only were we good to start collecting data, but that we would eventually be able to use it.  Frankly, I had nothing to back this up at the time.  However, we all know that the university is largely about politics.  Going into an assistant vice president’s office and saying “we need to collect some data and use it” is a far different than “we’ve been collecting this data all semester but IT won’t let us use it”.  Huge difference, in fact.  Question 1 is complex and multi layered, Question 2 is not complex.  My political clout, if you will, to get what I need, would ultimately be in the data that we were going to collect and I knew we would be able to collect it.

Personal interlude 2 – I’m relentless.  Keep turning me down and I will keep knocking at your door.  Acting like that is a stupid philosophy when you’re being selfish.  Acting like that when you’re trying to do the best job possible at work, and the thing that you’re trying to get will make you better at your job is smart.  Play your political cards correctly, push when pushing is wise, and step your game up when pushing isn’t wise.  I should explain the step your game up piece – going and whining to your AVC when you haven’t done your research and due diligence is foolish.  If you’re going to appeal to a higher power, then at least do your due diligence.  I’m reminded of the story about the man drowning on the roof of the house and praying for God to save him and he sees a man with a raft, a man with a plane and a man with a hot air balloon (or was it a jet ski or something?  I swear I’ve never seen a hot air balloon just randomly flying over a flood.  Maybe it was a dinghy), but he turns down all three rides .  When God comes back, the man says “you didn’t save me” and God says “who do you think sent the boat, the plane and the balloon?”  Don’t be that dude.  Regardless of your religious affiliation, I hope you understand that the idea is about using all the resources at your disposal before you start wasting your AVC’s time.  The AVC is the nuclear option.  If you play your cards right and utilize all your resources, you shouldn’t need to use them,  unless you get to the point you really can’t do anything else.  Then it’s time to push the red button.

Anyway, back to the story.

So, the IT guy sets up another appointment to come by and check out whether we had access to this student database on Banner.  I’m going to spare the argument that I think could (and maybe should) be made about whether all of us should have access to these files.  He was surprised to find that our office did, in fact, have access to all of the student records through Banner, and within about five minutes of clicking around in Microsoft Access, we had a query set up to pull student data like Year/Classification/Age/Housing Status/Meal Plan/Major from the student files.  The one piece we don’t have access to was, this is slightly hilarious, the file that actually contains the data we’re collecting.  In IT guy’s opinion, this was not especially secure information and he saw no reason that we shouldn’t be able to get access.  I concur.  It would be even more fun to go into our AVC’s office and say “i’ve been collecting data all semester, can i please access it?”  The access has been requested, though, so we’ll see how it goes.

At this point, I felt comfortable to approach our interim director, the Assessment Director for our Student Affairs Department about the process I’d been going through and how far I’d made it, and I think it’s fair to call her reaction complete and total shock.  I won’t go into the details of the conversation because that’s for her to share, but suffice it to say, she was shocked that I’d been able to get this access to data and the ability to swipe cards to tally who was attending events.  My reaction “i really wish I could tell you this was complicated and I’d done something magical, but I genuinely think people had had difficulty getting access to the student data that they need, so they just started telling people they couldn’t event track attendees at all and shouldn’t waste their time”.  Obviously, that statement was complete conjecture on my part, but it’s my best guess to why I got to where I did and others haven’t.  I should add that the overwhelming assistance from the man in IT was a huge help.  IT man, if you read this, I owe you a beer (or six).

Asking the Right Questions Makes All the Difference with Technology….And an Update on Our Assessment Efforts

I posted back in April about what I thought student affairs assessment should look at.  The inspiration for that post was my own work, where our assessment program has been essentially non-existent and as I was trolling through the data we do have, I was realizing that it was woefully inadequate to provide the sort of information that I was looking for.  I’m no fan of surveying, but I think with some data collection, we might be able to better figure out more about what exactly is going on within our department.

So…here’s my update….

I honestly anticipated this process of creating what I want to be like pulling teeth.  The general opinion from everyone I’ve talked to was that, not only was the data unavailable, that IT wouldn’t allow us to get the data and use it, they’d tried and no one would let them at it, and they’d appreciate if I’d quit asking, thank you very much!  Okay, that’s a little dramatic.  But the general idea of all conversations was something like “we’ve tried to get that, it doesn’t work and IT won’t help you”.

So, I set up a meeting with our swipe card office, expecting a war, instead I got no resistance whatsoever and was told that, not only can we help you, but we think what you’re looking for already exists!  WHAT????  Needless to say, I was pleased and began to contact the necessary parties to make this happen.

After a quick call into the department that had the system I needed to copy, I contacted the IT department person by email who was in charge of their project.  After not hearing back for a month, I anticipated I’d hit a snag.  I decided to call and the IT person couldn’t have been more helpful.

And here’s the kicker….we already had what we needed.

Not only that…we also had instructions in a cabinet to do what we needed.

So…what in the hell is going on here?

Obviously, there had been some information lost in the turnover between staff; I think you have to actively catalogue said information in the process (particularly with technology that’s so different from workplace to workplace) or it disappears.  In the transition, the move had been made to use a “good enough” solution for the program in question due to not understanding the full functionality that was at their fingertips.  I think all of that’s going to happen with technology implimentation processes.

But I’m more curious about the no’s.

I go into situations assuming that something can be done when it comes to technology.  I know what I need, I know how I think it can be done, but instead of saying “i need ____ technology to do ________”, I say “I need to get ________, how do I do that?”  I’ve found this to be particularly effective for acquiring any information or service that you need, but particularly positive in this situation or with technology in general.  It’s the difference between saying “i need to be on twitter, how do i do that” and saying “i need to communicate with students and I think I need to be online, what do you recommend?”  The core goal isn’t to be on Twitter, the core goal is community building and communication; in this situation, the core goal isn’t Banner utilization, the core goal is data gathering, I’m agnostic as to the method.

So, I plan on keeping you updated as this process proceeds forward.  It’s crazy to me that we’ve made it this far with essentially no new information.  I, possibly wrongly, attribute it to persistence and asking people who know the technology the right questions and trusting their expertise could help.  I’d like to hear your stories of tech implimentation!

What I Think #studentaffairs Assessment Should Look Like

  1. We need to track individual student’s attendance at events over the course of their time at the school.  I’m not particularly interested in doing this for any purpose other than to see that student affairs programming is actually reaching all of the students.  I suspect we’ll find that we’re hitting a small portion of the population.  To my knowledge, the infrastructure does not exist to do this, but what I’m seeing in my head is an enormous spreadsheet with individual student ID #s and tracking of what events they attended.  Frankly, this could be an entire student affairs (as well as athletics) effort to figure out what experiences our students are actually attending.
  2. Learning outcomes listed and justified for EVERY event.  I posted about the CAS standards here previously, and i’m thinking that you could list out individual learning outcomes for each event and then track these over the course of the year.  In theory, each office (even better if this was a collaborative SA effort) should be hitting all of the learning outcomes repeatedly.  Being able to cross reference these to determine what portion of our students are getting hit (not just shots in the dark, but actual individual students) by each learning outcome will give you some idea of what’s actually happening.
  3. Cost per student for each event.  If you’re tracking who attends, you should be able to get actual attendance numbers instead of estimates as well.  Calculating cost per student will help to determine whether students are actually getting the value that they should be getting out of their student fees.  A healthy look is to determine how much a similar experience might cost elsewhere; if cost per student is lower, you’ve done your job.

A few thoughts:

  • This plan places the responsibility on staff for accountability instead of surveying students to determine whether they’re engaged.  We’re responsible for creating an environment for student learning and this plan tracks whether we’re actually creating that environment.
  • For the most part, student affairs learning/community building is tracked over a longer time frame than classroom learning.  You can definitely learn chemical structures (okay…maybe you can’t…but i did) over the course of a day of studying.  You can’t learn how to have meaningful relationships over the course of a day.  Short time frame assessment, in light of this thought, is rather pointless and this system would provide the infrastructure to do a more meaningful long term study.
  • Self reported assessments of students are of marginal value anyway.  Incentive exists for students to either not take these seriously, say what they think the surveyor wants them to say, or outright lie.  In light of the incentives, the data received from these assessments (unless you’ve managed to limit these incentives somehow) is questionable.
  • Tracking financial expenditures with more accountability for said expenditures is imperative.  I’ve heard SA folks refer to activities fees as “play money”.  Please.
  • Tremendous research opportunities would be made available by tracking all of this data.  I think we all know that’s needed.

I’m genuinely looking forward to reading the comments.

A Defense of Cost Per Student As An Assessment Metric

I feel like I must defend cost per student as a method of assessing the quality of programs. I feel that I must defend it because I was told it was worthless and “not worth digging into”. Frankly, I disagree with this view and I think this view (at least partly) arises out of desire to run from the accountability and fiscal responsibility that said metric demands.

During my grad school internship in assessment, I stated that the currency of student affairs was learning outcomes. After thinking about this comment for almost 2 years, I’ve realized that I was wrong. Our currency does certainly lie in outcomes and the assessment of those outcomes, but more than that, the currency lies in a need to provide the best possible efficiency, and it follows quality, of programs with the dollars we’ve been provided.

At it’s root, looking at cost per student is at the root of what we should be understanding, no? The current calculation is to use the overall cost of a program and divide that by student attendance and that gives you cost per student. My analytical mind tells me that this is grossly oversimplified though and does not measure all of the metrics needed to determine success in a program. Shouldn’t learning outcomes also be included in the equation? I say yes.

Another question…can we figure out a way to estimate attendees for a particular program, as well as cost and thus in theory determine whether we SHOULD do an event? If we can apply learning outcomes, get an estimated attendance figure and an estimated cost per student, then shouldn’t this allow us to ask the “should we do this?” question a little more clearly? I think so.

Either way, if we’re looking into the question of whether an event is worth it (either pre or post event), cost per student is a key efficiency metric that we need to make better decisions about the true cost of our events and programs.

Book Review: “Fooled By Randomness” – Nassim Nicholas Taleb

I knew what I was getting into here. I had read The Black Swan and it was one of my favorite books. Taleb looks at the world and see it for what it is; whether there’s some power that controls everything or not, we have little control over many events and have incredible difficulty projecting or even guessing the future.

While this fault can provide incredible variety and intrigue to daily life, it can cause havoc in our personal lives and our financial lives if we are not always cognizant of our incredible inability to guess what may come next. My only problem with this book is that Taleb says in 500 pages what probably could have been said in 100. By the time you get to the end of the book, you feel as if you’ve read the same thesis 10 times (yet inexplicably Taleb says that the book continues to grow). While the Black Swan felt like a tight dissertation on our opaque and often pathetic view of the future, fooled by randomness feels like a mess and Taleb’s personal testing ground for his ideas.

Nonetheless, the book is a good read. It helped me to reflect on how much I anticipate that I can figure out how the future will play out and reminded me that you can always use circumstances for your advantage. While we certainly do not want to always plan for the worst, we can act in such a way that things do not have to be perfect for us to profit.

Share