e-Benchmarking closing event – 26th June 2008

Event details on the HEA site – for background on the e-Benchmarking project see the e-benchmarking HEA blog .

A few things by way of preamble. Looking back, the HEA and JISC did very well at creating an e-benchmarking exercise which was seen to respect confidentiality, contexts, and the inappropriateness of imposing ‘best practice’. I thought that the consultants, EdSUT and the HEA/JISC people were understanding, discreet and, well, kind. It was clear that they had the sector’s improvement at heart rather than constructing some kind of institutional hierarchy of performance.

They also did very well in their work to engage senior management – this is the right approach. Grass-roots change butts up against intransigent institutional structures all the time. Responsibility for pushing change has rested far from the actual decision makers for too long, unfacilitated by uninvolved and unaware senior management. Change shouldn’t depend on personalities – e-learning people are as susceptable to demoralisation as anybody else. So it was good to see so many senior people at the event.

Benchmarking has had an institutional focus, rather than a technological one or a pedagogical one – and that has given a fresh and helpful perspective.

(Come off it, we aren’t all excellent)

How has it become so controversial to say this?

It’s just wishful thinking to say we are. I never fell out of love with the concept of ‘excellence’ – maybe because I’m relatively junior in my organisation and it’s no loss of face to accept that people are more capable and knowledgeable than I am; maybe because I perceive the term as mostly being used aspirationally rather than descriptively.

Does it help to state the obvious – that not being excellent doesn’t mean we’re worthless? Only, I suppose, if the funders think so too.

So let me put the case. Maybe this will come across as perverse (I sometimes get the impression the Derek &tc have concluded that The Sector as extremely reactionary, perhaps made so by the pressures of competitiveness, the RAE, the NSS etc) but I (we here, actually) think that ‘benchmarking’ is an okay term to use. You compare with yourself last year and you have a way to compare yourself with other institutions insofaras a given area of work intersects with yours. Similarly ‘Pathfinder’ is an acceptable enough term to us (unless I’ve snoozed through something critical – maybe I have, I wasn’t at the seminal Town Hall meeting back in 2005 with the understandable fears about league tables and apportioning of funding).

I’ve no objection to these terms. After all, everybody seems to agree that money is a hugely important catalyst for innovation – why should we balk at referring to the funded teams as Pathfinders when we have implicitly acknowledged that money makes all the difference and are begging for more? I don’t feel the Pathfinders were ever presented as higher beings – to me they’ve been merely advantaged, engaged beings and I understand the reasoning behind funding them in this way. I also think it’s alright to talk about ‘good’, or sometimes even ‘best’, practice in some areas and it’s also alright to fund successful teams to push forward and research a given e-learning aspect. If something is working well or (and this is important because HEA/JISC should as a matter of… call it social mobility, fund potential as well as ongoing success) has potential in an institution, why shouldn’t that be promoted and advanced? Sure, Pathfinders don’t necessarily discover the path – it’s a path. That’s good, isn’t it? Provoking or inspiring, certainly enlightening – “Why might that path work here? Why might it not?”. Interestingly among the sorrow about the choice of the term ‘pathfinders’ I don’t remember any alternative being proposed.

Fair enough, we shouldn’t attempt to essentialise ‘best practice’ or ‘excellence’, but I was quite surprised to hear Derek say that we are “all excellent in our own way” and David Cairns talking about binning the idea of “best practice”. OK, if the terms are now sullied, it’s probably not a battle worth fighting to retrieve them – who am I to judge. But I think we should understand that there is enormous variation in practice, some of us fall short of what is hoped of us, and it’s not terminal because we don’t work in eg investment banks. It is possible for e-learning support to languish – it may or may not be down to something malaligned within the team (call it a team) or even the institution but still, we should certainly not think of the languishing or flourishing as a permanent condition – good e-learning team or bad e-learning team, for ever – teams’ fortunes can turn. Their work can improve or it can slump – you have to look at the team and the context of the team to find out why. Like that of any team, all that is good about the work of an e-learning team requires conservation. That’s what reflection is for.

All I’m saying is that, at the current time and subject to shifts, some e-learning teams are more ufunctional, more agile, more energetic, more help in their institutions than others – we have to admit it (while simultaneously resisting those who seek to tie this to funding). So I would like to shore up Derek Morrison &tc against those who say that a) we’re all excellent in our own way and b) there’s no such thing as good, better or best practice so we should stop looking for it. Is that a straw man I’ve just set up? Well, I’m happy to be disillusioned, but it’s based on the impression I got from Derek’s, Jane’s and Terry’s presentations.

Instead of fighting linguistic battles and imposing political correctness I hanker for a mutually supportive sector which not only looks for and commends good practice but also supports languishing practice with understanding and without prejudice. We have this between some London institutions – our natural affinity and friendliness overcomes any high-level notion that we’re competing – we understand that competition which only looks to improving things for our own students is bad for the sector.

Session on institutional perspectives

As somebody who has had half an eye on the benchmarking exercise for some time, I found that these went unecessarily over old ground. I’d have preferred sharper, more focussed presentations about the distinctive stuff, the institutional quirks, surprises, messages, key outcomes – i.e. the stuff we couldn’t have assumed already. There definitely was this, but there was too high a proportion of scene-setting, in my opinion.

In that spirit, I noted the following helpful and practical things:

• Chester runs a Podcasting SIG which is fast gaining a good reputation – Podcasting for Pedagogic Purposes – has 150+ members and its own wiki.
• Glamorgan now has a statement on what students can expect from e-learning
• QUB is finding that the boundaries between blended and distance learning are blurring
• Bristol, with its independent autonomous faculties, restructured its e-learning support into a hub and spokes model with a strong central provision but satellite areas of expertise
  • it concentrates strongly on communication between these
  • it has special interest groups across the institution which also build capacity
  • It has a network of e-learning support officers and academic e-learning advisors who help to identify good practice and alert the centre to it
• The U of L external system chose eMM precisely because distance learning requires great attention to systems and processes

Session – John Selby from HEFCE

• A man with an expanding portfolio and too much work to do
• Like all of the organisers, he has become very self-conscious and inhibited about language
• He is fighting back though – most quotable, to general hilarity: “the sector should not read too much into what we say”. Meaning don’t look for nuance and insinuation.
• But that said, “The words we use are important”.
• “We can no longer do what is required of us without e-learning”. Good point.
• As a result of this e-benchmarking exercise we know that “There are different approaches for different institutions in different contexts for a diverse student body”.

Really interesting and helpful Pathfinder break-out sessions followed. I realise I’ve omitted presenters’ names – ask me and I’ll dig them out.

Break out session 1 – Professional Development

Aberystwyth

• 30 M-level credit award in e-learning

Sunderland

• Professional standards framework – http://soolin.sunderland.ac.uk/pathfinder/ for e-learning professionals, based on the PSF previously developed by the HE Academy for teaching and supporting learning in general
• It was £ that got departments involved
• the team approach to planning e-learning interventions worked better than the individual approach.

UEL

• Conceptualising skills, they have what they call their ‘monster document’ – 450 individual skills divided into 13 categories. They aim to produce an online self-assessment to help people become aware of their own skills
• Offer an MA in Online Blended Learning
• To promote engagement with staff development, they
  • work with the competitiveness of the Deans
  • offer crash courses in technology (esp. fast changing ones) which often leads to an invitation to return for a longer session
  • begin with needs assessment – ask the group to identify in advance if possible 5 priorities
  • individualise all presentations – find school-specific examples
  • have an e-learning lab, including a sound-proofed area functioning as recording studio
  • build comfort with technologies in one-to-one sessions tailored to individually-identified needs

Breakout session 2 – enhancement technologies

Kingston on mobile technologies

• Kingston have been experimenting with PRS systems, incoming text messaging, tablet PCs with wireless data projection, and wireless mics for podcasting
• PRS was the most used (10 tutors) and while this does take a bit of time to set up, students found it positive

Gloucestershire Uni – digital story telling

• This was part of a horizontal PDP theme to develop reflection
• difficulties marking – narrative flair does not equate to good tech skills nor to reflection. Some tutors asked for reflective work on the development of the reflective work… sounds tricky
• copyright issues – directing students to stock photo / copyleft sites
• Freely available tools (mentioned Moviemaker or Photostory 3 – also PowerPoint

De Montfort on Web 2.0

• podcasts to enhance face-to-face
  • tutors suggested ways of using the podcasts
  • but students soon adapted them
• specific activities for sychronous classrooms

Panel

This was mostly concerned with the need to involve senior policy makers in institutions

• Sarah Porter (JISC Innovation)
  • the difficulties of effecting top-down change.
  • learning technologists must marry ideas with strategic plans
• John McLaughlin (DIUS)
  • “student as consumer”
  • Pro VCs interested in saving/gaining money, retention, progression and achievement, and any initiative must be tied to these.
• David Cairns (QAA)
  • the idea of best practice has been consigned to the dustbin.
  • students are a driver but our key audiences for e-learning development are staff members – including senior staff
  • he would seek support for a QAA SIG on e-learning, organise a session on e-learning within QAA, and disseminate findings from e-benchmarking to QAA audit and review teams
  • What happens when you consumerise HE? Most institutions resist instrumentalisation and are doing a good ethical job. Not all though – hence the need for a QAA
  • QAA is owned by the sector and exists because students should receive something worth having
• Derek Morrison (HEA)
  • “you have to be perceived to be relevant” rather than merely being relevant.
  • no more funding so we have to build intrinsic sustainability within institutions before thinking about extrinsic sector sustainability
  • Working from the top-down is necessary – through the HEA Pro VC network and also the Leadership Foundation.

And that is that. Thanks Cliff Wragg for energising us. Thanks Derek, Jane, Veronica, Terry. It’s been good.

 

Report template

The template for the Benchmarking Phase 2 summary report. (N.B. This is different from the IRD document we submit as part of the ACU/OBHE methodology).

The deadline for the summary report is January 15th 2008.

 

C6 – student interview transcript

A student interviewee has made the transcript of his interview public – and he has confidence in our e-benchmarking approach too. (No money changed hands.)

Thanks for letting us know, Padraig.

 

C6.8 – meeting set up with AH from Widening Participation

As suggested by Sonja. Wed 15 Aug, 11:30 here in CELT. Not much more to say for now – will add minutes afterwards.

 

Student questionnaire update – C6

Thanks to Oghale for all the hard back-end work to get this up, running and secure with a data access interface and a printable set of responses for each participant. There were so many fields (we’re using a matrix of learning activities and different technologies) that the results go into 3 different spreadsheets. It is public access (because we’re keeping the anonymity option) but hidden from search engines.

Over to the Student Union shortly. Sonja (a research student here) completed it in 10 minutes and suggested changes which we worked in.

Since this questionnaire is something that IT Services, the Union and the Learning and Teaching Office think would be worthwhile to run annually, it’s worth spending the time to get it right.

Important – when analysing data

• There are no column headings in any of the data files (need to be carefully inserted)
• crucial to maintain the relative order and position of each record across the 3 files – i.e. adding column headings as a row will inevitably shift each record down a row – crucial to do this for each of the 3 files in order to keep the relative postition and match up each record across files

 

C4 (resources and value for money)

After speaking with GC (Finance) MV emailed GC and UQ with the relevant questions from C4 (omitting 4.7 and 4.8), deadline (Sep 21st but request for speedy response), reassurances about exemption from FoI and anonymisation, and background with link to this blog.

 

Meeting about staff questionnaire (Section C)

Andrew, John and Mira met on 12 Jun 07 (Sonja being out of the office).

John and Andrew had been working on Section C from a central support service perspective and had had to negotiate some dilemmas, namely:

• the questions suggested by the IRD do presuppose certain values, standards and ways of doing things with respect to e-learning. While we are free to depart from these, the point of being part of a cohort is to be able to make some comparisons between institution, so there is a balance to be struck.
• When the IRD asks what processes are in place, and how they are planned and implemented, there are two ways to respond. We can, for a significant proportion, respond with a snapshot of the here-and-now: “No processes are in place”. But we can also use the IRD as a stimulus to set up those processes. The former – snapshot – is not very helpful on its own. The latter is more in keeping with the reasons we signed up to benchmarking in the first place. At the same time, we must be open about how new these processes are.

We agreed that we would inevitably have an extremely low response rate unless the questions were closed, and that this indicated the need to also interview staff.

We decided to approach identifying the questionnaire questions by going through section C of the IRD sub-section by sub-section.

The following questions are tentative. Keeping them in section order is not necessarily good for the flow of the questionnaire – maybe better to rearrange. The 4 point scales may be better of with more points – to decide. Each section will need a free text ‘Your comments’ field.

C1 – Strategy Development

• Have you seen (or ‘are you familiar with’?) the parts of the institution’s strategies which relate to e-learning? (Y/N/DK)
• Were you involved in the consultation process through which these were developed? (Y/N/DK)
  • If so, how (checklist)

C2 – Collaboration and partnership

• Are you using online information and communications technologies to support any collaborative partnerships outside the institution for
  • Learning and teaching (Y/N)
  • Research (Y/N)

C3 – Management and leadership

• Have the institution or your department formulated specific plans or objectives regarding online learning, towards which you are requested to work? (examples, maybe) (Y/N)
• To whom, if anyone, are you formally requested to report your progress on developing e-learning? (checklist)

C4 – Resources and value for money

• How do you obtain resources (e.g. time, money, equipment, incentives) to develop e-learning? (checklist)
• Are you asked to report the full cost of your e-learning activities (staff time, equipment, any proprietary software licenses?) (Y/N)
• (this might be better asked of senior management…) Are you asked to assess and report the risks associated with any e-learning activities? (Y/N)

C5 – E-learning delivery

• How would you describe your awareness of the following with respect to online learning:
  • Disability and accessibility (for people with special educational needs) (scale 1-4)
  • Data protection and privacy (scale 1-4)
  • Freedom of information (scale 1-4)
  • Copyright (scale 1-4)
• Where do you look for existing online learning materials in your subject? (checklist including well-known repositories, HEA subject centres, Google, “Other – please describe” and an “I don’t”)
• How do you keep aware of other e-learning initiatives in your discipline? (checklist including HEA subject centres and others).

C6 – E-learning and students

• How do you know the needs and expectations of your students with respect to e-learning (checklist including observation, questionnaire, informal discussion, feedback – something that will help us understand whether this is formalised or not, without implying that it should be)
• How do you know which of your students are participating in the e-learning you have set up? (checklist including don’t know and “I haven’t set up any e-learning”)
• How are your e-learning activities evaluated for educational impact? (checklist including integrated with programme monitoring, informal discussion, separate questionnaire, I don’t and I haven’t set up any e-learning etc)
• How do you think Goldsmiths can help ascertain these needs and expectations? (we could list questions here, or modes of inquiry)

C7 E-learning and staff

• How well-informed do you feel about Goldsmiths expectations and plans with regards to e-learning? (4 point scale)
• How well do you feel you understand the potential of e-learning? (4 point scale)
• To what extent do you feel that the work you do on e-learning is recognised in your department? (4 point scale)
• Where do you go for staff/professional development in e-learning? (checklist)

C8 Communications, evaluation and review

• How do you know how other teaching staff across Goldsmiths are approaching e-learning? (checklist including I don’t)
• How well-informed do you feel about the impact of e-learning interventions across Goldsmiths on students and learning? (checklist including I don’t)

 

meeting amongst CELT

We met today to discuss a few issues that have arisen and need to be dealt with before the meeting on 17th July.

1. On the basis of Mira’s notes (recorded in a post here), will draft the staff questionnaire, which will be sent out before the next meeting.

2. Oghale has been working on the student questionnaire, SG (who is also student) has filled it out, it all works and can be sent to

5. questionnaire 4 students: out yet? Oghale working on last bits (Sonja piloted, needs to be printable) – will THEN go out to the students union

6. We need to collate some material for the meeting on the 17^th July.

Actions: SG to finish off her input on the John/ Andrew IRD.

JP to look at IRD on Monday. SG to get in touch with Steve Collins to find out what he has brought together; MV to get in touch with Gisele re finance section; JP to deal with B5 today; JP to book a room for thet 17^th. All to get in touch again for next meeting on

 

short meeting 27th Sep 2007 – private post

Divvied up the sections to be written

Sonja: C3, c5, c7

Mira: c4, c6

John: c1, c2, c8, D, b5 (in sense that some more data is in the staff questionnaire.

Meet again tomorrow.

 

Student questionnaire and interview update – C6

The following have looked at the questionnaire draft and made suggestions and changes:

• Student union campaigns and communications officer
• Student union president 2007/8
• CELT
• Head of IT
• Head of Web team
• Systems administrator

The Student Union is very keen to run the questionnaire yearly.

Now the questions are finalised, needs to be properly worked up and piloted (probably within the Union).