What does ratemyprofessors.com actually rate?

"For almost a century, attempts have been made to evaluate the teaching effectiveness of instructors in higher education. Much of this effort has centred on an attempt to create instruments that would allow students to make this assessment. These instruments have varied over both place and time, and have created one of the longest lasting debates in higher education. ... One source stated that there were close to 3000 articles published on student evaluation of teaching (SET) from 1990 to 2005 alone. Published findings on the topic are so voluminous that many researchers have been using the method of meta–analysis, in which the case is not a subject but an entire published article. Nevertheless, little agreement has been reached on key points. ... One of the difficulties of studying SET has been finding adequate and appropriate large samples, because of the confidential and anonymous nature of the inventories for both students and instructors. An alternative has become available online. Increasingly popular databases, such as ratemyprofessors.com, uloop.com, koofers.com, myEdu.com and studentsreview.com, offer large and easily accessible sources of data. Although these sources are tempting, a question remains about their validity. ... ‘Does ratemyprofessor.com really rate my professor?' "

What does ratemyprofessors.com actually rate? (2013) Assessment & Evaluation in Higher Education doi: 10.1080/02602938.2013.861384
Abstract: This research looks closely at claims that ratemyprofessors.com creates a valid measure of teaching effectiveness because student responses are consistent with a learning model. While some evidence for this contention was found in three datasets taken from the site, the majority of the evidence indicates that the instrument is biassed by a halo effect, and creates what most accurately could be called a ‘likeability’ scale.


More on this exciting field of research

A.J. Cann

Read More

The murky world of academic anayltics

I've been involved in a number of discussions this week about the use and abuse of metadata. The reactions have been mixed. Many people seem unaware of the potential dangers in this area, some of them who clearly should be better informed. Others are sanguine about the prospects. The final group have their tinfoil hats clamped securely over their ears - obvious nutjobs.

Or are they?

Before you decide, read this recent article in The Atlantic: Your Job, Their Data: The Most Important Untold Story About the Future.

One of the things I have been thinking about this week is how much metadata is captured by academic systems such as Blackboard, Echo360 and Panopto. Most people seem to be blissfully unaware of this, or simply don't care. Stephen Walker's post Lecture Capture and Data Capture did a good job of raising the key issues. We know what use commercial services such as Amazon, Facebook and Google make of this data - but what about academia? Is "Do No Evil" the watchword or do uses such as performance management - of students or staff - take precedence?

There is an easy way to deal with this problem - clear and transparent institutional policies on these issues. But more often than not, policies are lacking, either through omission or commission, resulting in loss of trust. The University of Manchester Policy on the Recording of Lectures and other Teaching and Learning is exemplary in this regard and should be followed by all institutions:

3.2 Prior to the beginning of each Semester, teaching staff will be informed that recordings will be made if their teaching and learning activities take place in a location containing automated lecture capture technology. Staff not wishing to have their sessions recorded should respond to this stating 'Opt out'.

3.6 Recordings will not be used for staff performance management purposes.

Such clear policies allow both staff and students alike to concentrate on learning rather than spending time wondering what Big Brother will do with their personal data.


Update: Helpful link via Martin Hawksey:
Prinsloo, P., & Slade, S. (2013, April). An evaluation of policy frameworks for addressing ethical considerations in learning analytics. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 240-244). ACM.

A.J. Cann

Read More

#ITFocusWeek - What's wrong with "lecture capture"? This:

Today I'm taking part in a Question Time panel about lecture capture technology. I'm cast comfortably in the role of curmudgeonly old git who doesn't like it. The reality is slightly more complicated.

Everyone likes online videos. I do, that's why my YouTube channel is closing in on 1.5 million views. At least, I like good online video. But what does that mean? It certainly doesn't mean the lowest common denominator of talking head video - plonking yourself in front of a webcam and droning on because that's the easiest thing to do. And it certainly doesn't mean any video longer than five minutes. All of which means that the idea of "lecture capture" is wrong headed.

Alexa.com reports that people average 17 minutes per day on YouTube and watch 12 videos, that's 85 seconds per viewing. Here's a graph which shows YouTube's own data for "audience retention", i.e. the length of time people pay attention to online videos before stopping or skipping ahead. What do you think this data says about the concept of "lecture capture"?

Ah, you say, but people sit on their sofas watching David Attenborough with rapt attention for an hour. Well first of all, you're not David Attenborough. Second, you don't have 1% of 1% of the budget or the time that the BBC or Sky spent on making that program. But most important of all, you don't understand the difference between lean forward and sit back media. The interactivity of an integrated control bar in online videos banishes the passivity of watching linear video on a big screen such as a television and encourages use of fast forward skipping. Online, all attention spans are short.

The Atlantic recently ran a piece titled Lectures Didn't Work in 1350—and They Still Don't Work Today. It's hard to know where to start listing what's wrong with this article. First, it's not about higher education or about lectures at all, it's about primary and secondary education where lectures (rightly) don't feature as a mode of instruction as far as I'm aware. But even if you apply it to higher education, the whole piece conveniently ignores the rather obvious fact that lectures have indeed worked since 1350 - that's why students pass rather than fail. It is simple minded twaddle to simply ignore this. And in an age where technology increasingly threatens to distance us from each other, the physicality of the lecture room increases its impact and its value. Sitting in front of a recording erodes that valuable link between staff and students.

So what's my problem with lecture capture technology? The very name. The idea that a lecture can be captured and bottled is laughable and shows a fundamental misunderstanding. This technodeterministic language cements the idea of technology replacing rather than augmenting pedagogy, the wrong-headed notion that disruptive technologies kill older technologies. This is demonstrably false - TV didn't kill radio, and radio didn't kill newspapers, etc. Instead, in the real world we have the Lindy Effect - the longer a technology has been around, the longer we can expect it to survive. Lectures are here to stay, so the question is, how do we use new technologies to augment and improve lectures, not to replace them?

And that's where the use of technologies such as Echo360 and Panopto come in useful. We use them to produce short (less than 5 minute - look at the graph) high impact videos covering the threshold concepts that students struggle with. And we use those recordings to augment and improve the vital lecture experience. Give 'em the old TED razzle dazzle. More importantly, we make these technologies available to students and we challenge them to make their own videos explaining threshold concepts. Instead of sitting passively in front of recorded lectures students become active partners in transmitting knowledge. And we stop pretending that technology solves problems. Technology doesn't solve problems, people do. But technology can cause problems, such as increasingly passive learning. Let's not let that happen. Let's encourage the use of online video. Let's stop talking about lecture capture.

But don't take my word for it:

• Jones, S.E. (2007) Reflections on the lecture: outmoded medium or instrument of inspiration?. Journal of Further and Higher Education, 31(4), 397-406
• Mulryan-Kyne, C. (2010) Teaching large classes at college and university level: challenges and opportunities. Teaching in Higher Education, 15(2), 175-185
• Mark Smithers: Is lecture capture the worst educational technology?

A.J. Cann

Read More

What happens when an economist measures teaching quality?

... probably not what you're expecting.

"Although assessment exercises are nowadays a widespread practice in educational systems, we lack an effective understanding of what is the actual feedback effect on the overall efficiency (macro level) of the system and on the individual level (micro level). Given the lack of any sort of incentive to perform better in teaching activity, I formulated the hypothesis that this effect is null (at least for the Italian academic system), implying that the overall quality of teaching which is supplied does not change over time. Even worse, I claimed that the same is true at the individual level. ... This study ... shows that the evaluation of teaching quality might not have been conducive in improving it. On the one hand, it is possible that feedback from the questionnaire was not used effectively, and on the other (much more plausible) that the incentive system is not working properly. This last argument might entail dramatic consequences for the Italian academic system (and in general for European education systems), which tends to be increasingly research-oriented and neglects teaching duties. At this point one crucial question remains open: does it make any sense to continue carrying out teaching quality assessment when it does not enhance teaching performance? The tentative answer is ‘no’, at least until some incentives are introduced for providing better teaching or a more efficient teaching performance management is established."



Bianchini, Stefano. Feedback effects of teaching quality assessment: macro and micro evidence. Assessment & Evaluation in Higher Education (2013): 1-15. doi: 10.1080/02602938.2013.842957
Abstract: This study investigates the feedback effects of teaching quality assessment. Previous literature looked separately at the evolution of individual and aggregate scores to understand whether instructors and university performance depends on its past evaluation. I propose a new quantitative-based methodology, combining statistical distributions and transition probabilities matrices, to take into account the dynamics of teaching quality over time both at the macro and the micro level. Using a three-year longitudinal panel from an Italian university, it is shown that evaluation exercises do not impact future teaching performance at either the university level or the individual level.

A.J. Cann

Read More

Blogging 2013

The LSE Impact of Social Sciences blog asked me to write a guest post about the blogging session at #solo13. So I did, and they have just published this:

---

As academic blogging becomes mainstream, science communication must facilitate depth and breadth in online discourse.

The SpotOn London conference, formerly called Science Online London, has just run for the sixth time. Held at the British Library and sponsored by Digital Science, The Wellcome Trust and a number of other high tech, high profile sponsors, SpotOn London (or #solo13 as it is more widely known) is both the flagship event for current science communication and collaboration and represents the cutting edge of developments in the way science is being conducted. The three official strands at the meeting, science communication and outreach, online tools and digital publishing, and science policy, reflect the interest and enthusiasms of those involved.

I have been fortunate enough to be involved with all six Science Online London meetings. Although I was not physically present at the first Science Blogging conference at the Royal Institution in 2008 I spent a day following the event online - my first exposure to how new media were changing academic life. The feeling of excitement around the emerging tools - Friendfeed (precursor of the present day Facebook interface), Twitter, informal video livestreams (from people's phones and of varying degrees of reliability), and aggregation of discussions from blogs, Flickr, etc - had sufficient impact to shape my thinking about and approach to online communication, as well as my own academic practices.

As a grizzled veteran, #solo13 inevitably didn't have the same impact for me that some of the earlier meetings did. This is just part of the natural cycle of exposure - adoption - commonplace with emerging technologies. What was noticeable at #solo13 was the emergence of a younger demographic of PhD students and science communicators experiencing it all for the first time. I don't intend to try to cover all the content and discussion from this packed two day event in a single post, but I would like to describe some of my reactions to the "Using blogs and other online forums for communication between scientists" session held on the second day. You can watch a video recording of the whole session online:

As I commented during the session, I'm a serial blogger who couldn't stop if I wanted to. In fact, the only major change I could envisage in my blogging practice might be to obscure my identity when writing about certain topics - more of this later. I currently maintain five active blogs. My personal blog, Science of the Invisible http://scienceoftheinvisible.blogspot.com , is where I write about my educational research and professional life, but also share more personal (but not intimate) content such as recipes and commentary. MicrobiologyBytes is my oldest continuous blog, and writing about microbiology online helps me stay up to date for my teaching and my personal interest in science communication now that I am officially no longer "research active" in this field (which is a reflection of grant income rather than the end of my scholarship). I am the Internet Consulting editor for Annals of Botany, responsible for social media policy, and our blog is very much a team effort aimed at promoting the content in the journal, but also awareness of plant science to a more general audience. @leBioscience is an online shop window, the 21st century equivalent of the staff newsletter we used to distribute in print 20 years ago. My student support blog is both a content management system and an authoring tool for advice and guidance to the students I teach, and this growing personal repository of information saves me endless amounts of time by allowing me to refer frequently asked questions to this resource rather than to have to answer each one anew. In addition to this core, project blogs and grant blogs come and go with fixed lifespans as dissemination tools and archives for particular projects. You get the idea....

With this history, it was hard for me to resist a gentle smile at the panel member who introduced themselves with the line "I've been blogging for two months", but at the same time, it was encouraging that there is more of an acceptance of blogging as an important academic tool. Unlike the Science Blogging 2008 meeting, there is now a feeling of establishment rather than revolution about blogging - part of the process of scholarship, but particularly with regard to individual development rather than as an institutionally-owned process. Few academic institutions actively encourage their staff to blog and there remains a justified wariness of the possible consequences of being judged to have frittered away valuable time which could have been "better" spent on traditionally mainstream academic activities. Across a wider sector, Marketing still retain control of blogging in many organizations and we have still to fully realize the benefits that can arise from reflective cross-institutional communication and knowledge exchange which can arise from a blogging culture. At a simple level, it would be easy for institutions to encourage more staff to blog by simply providing the tools - for example, a site-wide installation of Wordpress available to all. Moving beyond that to a culture where blogging is the norm rather than is much more complex and not as easy to achieve. In a time of financial stringency, senior academic management will need to provide the lead in such matters in order to enable this sort of change.

There is also something of a pervasive feeling of weariness around blogging. There appear to be some bloggers who don't enjoy the process but do it out of a sense of obligation or not wanting to miss out. Blogging is still a peripheral rather than a core (paid) activity for most. For me, blogging is fun (although not always enjoyable, and not as much fun as teaching, with which it overlaps) - blogging is not my day job, so it's something I'm often doing in the early hours of the morning or in snatched minutes between meetings if I can get my brain in gear. If the reward is not financial nor much institutional credit for most, what keeps academic bloggers going? One common experience at the meeting was of colleagues and students who read your blog secretly, then come up to you at some point and say "I read your blog". It doesn't count as REF impact, but this peer acknowledgement is a significant personal reward. If your institution does not provide the platform, other sites such as the Huffington Post, The Guardian and New Statesman all accept guest posts, and if you have the time and talent to get your writing accepted by such sites it will certainly attract attention within your institution and from future employers.

The issue of commenting on blogs was also discussed in the session. For most blogs, comments are dead and any conversation is now based around is now retweets and reposts to social media sites rather than a threaded stream on the original site. The same is also true of online commenting on scientific papers - on the PLOS and Nature websites for example. There is plenty of discussion going on about the content, but it is distributed across many networks rather than being focussed into an easily accessible thread. In spite of a relaxed attitude to this by some, in my opinion this is a problem as it encourages superficial commenting at the expense of more in depth conversation. Sharing on Facebook is a good thing but it is not a substitue for a good discussion thread on a thorny issue. In science in particular, putting your head above the parapet and being seen to criticise the work of senior scientists is still a risky business, even when such comments are made in neutral tones and intended as a positive contribution to ongoing work.

Overall, I came away from this session feeling buoyed up by the discussion. A blog is not a book or a newspaper, but it is more than a Twitter account or a Facebook page. Blogs are still the centre of serious online academic communication but there is still a long way to go until the Republic of Blogs is established and academic blogging moves from being a spare time activity to a mainstream output. I'm delighted to have been asked to contribute this guest post to the LSE Impact blog and I intend to expand my guest blogging activity as time permits. Sleep is over rated anyway.

Read More

Why Are We Bowling Alone?

The most influential book I have read in the past 10 years is Robert D. Putnam's Bowling Alone (Simon and Schuster, 2001. ISBN 0743219031). I'm aware of some of the criticism which has been leveled at Putnam's work, but for me, as for many others, Bowling Alone provides a description of the society I see around me. The problem I've always had, and which Putnam does not fully address, is why has there been the erosion of social capital that he describes?

Last weekend I read Thinking, Fast and Slow by Daniel Kahneman (Penguin UK, 2012. ISBN 0141033576). Kahneman's book does not have the impact of Putnam's and while not undermining the overall corpus of his work, comes across as rather superficial in many ways. But one aspect that Kahneman discusses definitely struck a chord, the issue of money priming:

“Reminders of money produce some troubling effects. Participants in one experiment were shown a list of five words from which they were required to construct a four-word phrase that had a money theme (“high a salary desk paying” became “a high-paying salary”). Other primes were much more subtle, including the presence of an irrelevant money-related object in the background, such as a stack of Monopoly money on a table, or a computer with a screen saver of dollar bills floating in water. Money-primed people become more independent than they would be without the associative trigger. They persevered almost twice as long in trying to solve a very difficult problem before they asked the experimenter for help, a crisp demonstration of increased self-reliance. Money-primed people are also more selfish: they were much less willing to spend time helping another student who pretended to be confused about an experimental task. When an experimenter clumsily dropped a bunch of pencils on the floor, the participants with money (unconsciously) on their mind picked up fewer pencils. In another experiment in the series, participants were told that they would shortly have a get-acquainted conversation with another person and were asked to set up two chairs while the experimenter left to retrieve that person. Participants primed by money chose in the exto stay much farther apart than their non-primed peers (118 vs. 80 centimeters). Money-primed undergraduates also showed a greater preference for being alone.”

Reading this instantly bought Bowling Alone to my mind. I spent a happy/unhappy 24 hours reflecting on this new knowledge (depending on how you want to look at it). Sadly, the next day I read Michael Kraus's post, A Social Priming Finding with Direct Replications, which is another take on the recently discussed "Replication Problem" in psychology.

"Social priming researchers are faced with the public perception that our field lacks methodological rigor. In the past, the dominant reaction has been to simply downplay these concerns and assure people that our findings replicate."

Together with the rather glib tone of Thinking, Fast and Slow, this has somewhat undermined my confidence that I have finally found a reason we are bowling alone. But I can't shake the gut feeling that affluence must be at least part of the explanation for the accelerating enclosure of public goods and the continued decline of social capital.

• Is it time to up the statistical standard for scientific results?
• Social capital isn't an unadulterated good

 

A.J. Cann

Read More

Time is of the essence

I don't think there was a single session at #solo13 where time was not mentioned as an obstacle to getting more of what we want.


ScienceGist (tagline: "Simplifying science") aims to demystify scientific papers by publishing easily understandable summaries. The project is the outcome of a hackday earlier this year. It's very early days for ScienceGist, and there's a limited range of content there at present, for example, Plant Biology such as this.

I like the idea of ScienceGist a lot. But there's a problem. The site is in competition with Wikipedia, reddit, Buzzfeed, etc, and compared to those sites it is a minnow. Brain Kelly quoted this yesterday:

"as Cameron Neylon had said in the “Wikimedia UK Annual Review 2012-13″:
“If you’re serious about ensuring public engagement in your research then you need to make damn sure your work can be incorporated into Wikipedia. Wikipedia is the most important engagement channel for your research."
(The Wikipedia Editing Workshop Session at the SpotOn 2013 Conference)

So far I have only reposted content I have written for other sites. Since both are published under a CC license this isn't a problem, but if I wasn't already writing this content for another site, would I have time for ScienceGist, good as it is?

One thing I've learned about curation is to find a model that works for me and not to try to do everything. In cutting back to a single channel, popularity has to be a key feature. I really like ScienceGist, but they've chosen a hard furrow to plough.

A.J. Cann

Read More