Read Dimensions_of_Quality.pdf text version

Dimensions of quality

byGrahamGibbs

ThehiGhereducaTionacademy

The views expressed in this publication are those of the authors and not necessarily those of the Higher Education Academy. ISBN 978-1-907207-24-2 © The Higher Education Academy September 2010 The Higher Education Academy Innovation Way York Science Park Heslington York YO10 5BR www.heacademy.ac.uk Tel: +44 (0)1904 717500 Fax: +44 (0)1904 717505 All rights reserved. Apart from any fair dealing for the purposes of research or private study, criticism or review, no part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any other form or by any other means, graphic, electronic, mechanical, photocopying, recording, taping or otherwise, without the prior permission in writing of the publishers. To request copies of this report in large print or in a different format, please contact the Academy. Designed by Daniel Gray Printed by The Charlesworth Group

2

dimensionsofqualiTy byGrahamGibbs

2 4 8 11 14 19 38 43 50 51 52

forewordbycraigmahoney 1. executivesummary 2. introduction 3 . Thenatureofdimensionsofquality 4. Presagedimensionsofquality 5. Processdimensions 6. Productdimensionsofeducationalquality 7. summaryandconclusions 8. abouttheauthor 9. acknowledgements 10.references

ThehiGhereducaTionacademy

foreword

Theperennialdebateaboutwhatconstitutesqualityinundergraduateeducationhas beenreignitedrecently,notleastbyarangeofpublishedresearch,selectcommittee activity,tighteningofresource,andthelarge-scalereviewbylordbrowne. astheorganisationdedicatedtoenhancingthequalityofstudents'learning experiences,thehighereducationacademyispleased,throughthispieceofwork,to contributefurthertothisimportantdebate. ourstarting-pointistwofold:first,thathighereducationshouldbea transformativeprocessthatsupportsthedevelopmentofgraduateswhocanmake ameaningfulcontributiontowidersociety,localcommunitiesandtotheeconomy. second,thatanydiscussionaroundqualityneedstobeevidence-informed.asa result,weidentifiedaneedtosynthesiseandmakesenseofthescatteredresearch inthefieldofhighereducationquality.wewantedtofindoutwhattheresearch evidencetellsusandwhatfurtherworkwecandotoapplytherelevantfindingsin ourquesttoimprovethequalityofstudentlearninginuKhighereducation. GrahamGibbsstatesthatthemostimportantconclusionofthisreportisthat whatbestpredictseducationalgainismeasuresofeducationalprocess:inother words,whatinstitutionsdowiththeirresourcestomakethemostofthestudents theyhave.examiningtheevidence,hedrawsconclusionsaboutsomekeytopicsthat havebeenthesubjectofmuchdebatearoundquality.forexample,heconcludes thatthenumberofclasscontacthourshasverylittletodowitheducationalquality, independentlyofwhathappensinthosehours,whatthepedagogicalmodelis,and whattheconsequencesareforthequantityandqualityofindependentstudyhours. healsoreiteratesresearch(nasret al.,1996)thatshowsthatteacherswhohave teachingqualifications(normallyaPostgraduatecertificateinhighereducation,or somethingsimilar)havebeenfoundtoberatedmorehighlybytheirstudentsthan teacherswhohavenosuchqualification.ithinkthisisacrucialpoint.attheacademy webelievethathighqualityteachingshouldbedeliveredbyacademicstaffwhoare appropriatelyqualifiedandcommittedtotheircontinuingprofessionaldevelopment. Tothisendwewillcontinuetoprovideanddevelopanadaptableframeworkfor accreditedteachingqualificationsinhe,incorporatingtheuKProfessionalstandards frameworkandotherrelevantteachingqualifications.wewillalsocontinuetowork withheistodevelopandmanagecPdframeworksforlearningandteaching. Thereportalsoconcludesthatsomedimensionsofqualityaredifficultto quantify,anditisthereforedifficulttoseewhateffecttheymighthave.aspectsof

2

dimensionsofqualiT y

departmentalcultureareonesucharea:whetherteachingisvaluedandrewarded, whetherinnovationinteachingissystematicallysupportedandfunded,etc.The academyhasalreadyconductedresearchintotherewardandrecognitionofteaching whichshowedthatover90%ofacademicstaffthoughtthatteachingshouldbe importantinpromotions.wewillcontinuetofocusonthiswork. someofthefindingsofthisreportmayconfirmaspectsofinstitutionalpolicy onenhancingquality,someofthemwillpromptnewanddifferentapproachesto focusedinvestmentoffundingandexpertiseinordertomaximiseeducationalgain, particularlyatatimeofdiminishingresource.someofthemwillcallintoquestion theefficacyandappropriatenessofpracticesandpolicies,andcauseustolooknotat howmuchisspentpercapita,butonhowitisspent;lessonhowmanycontacthours areprovidedbutwithwhomandwithwhatconsequencesforindependentlearning; ontheextenttowhichwetrulysupportandadoptthekindsofpedagogicpractices thatengenderstudents'intrinsicengagementintheirlearning. Grahamarguesforabetterfocusonevidenceinordertounderstandquality properly,toensurethatourqualityprocessareinformedtoagreaterextentbywhat weknowaboutwhatconstituteseffectivepracticeandabouttheextenttowhich thesepracticesareemployed,tomakebetterandmorecoordinateduseofthefull rangeofavailabledata,andtounderstandtherelationshipbetweenthem. Thispaperisprimarilyforanaudienceofseniormanagersofheis­the colleagueswhodevelopandimplementthekindsofinstitutionalpoliciesthathave thepropensitytoimprovestudentlearningandwhoconceptualisetheframeworks tosupportthatvitalprocess.wehopethatthisreportwillmeaningfullyinformboth policyandpracticeandlookforwardtofollowingupthisworkinthecomingmonths byengagingwithyouindebatesanddiscussionsaboutthedimensionsofquality. Professorcraigmahoney chiefexecutive Thehighereducationacademy.

3

ThehiGhereducaTionacademy

1.

execuTivesummary

"A ... serious problem with national magazine rankings is that from a research point of view, they are largely invalid. That is, they are based on institutional resources and reputational dimensions which have only minimal relevance to what we know about the impact of college on students ... Within college experiences tend to count substantially more than between college characteristics." --Pascarella,2001 1.1 Thefocusofthereport

Thisreporthasbeenwrittentocontributetothecurrentdebatesabouteducational qualityinundergraduateeducationintheuK,andabouttheneedtojustifyincreases inresourcesonthebasisofindicatorsofeducationalquality.Thisreportwillidentifya rangeofdimensionsofqualityandexaminetheextenttowhicheachcouldbeconsidered avalidindicator,withreferencetotheavailableresearchevidence.itattemptstoidentify whichkindsofdataweshouldtakeseriouslyandwhichweshouldbecautiousofplacing weighton.someofthesedimensionswemightbewisetopayattentiontocurrentlylack asolidevidencebase,especiallyinrelationtoresearchcarriedoutintheuKcontext,and sothereportalsoidentifiesprioritiesforresearchandfordatacollectionandanalysis. 1.2 T heapproachtakentoconsideringdimensionsofquality

Thereportidentifieswhichdimensionsofeducationalqualitycanreasonablybe usedtocompareeducationalsettings.itadaptsbiggs's`3P'model(biggs,1993)of `presage',`process'and`product'tocategorisethevariablesunderconsideration (seesection3.2).Thereportexaminesawiderangeofpotentialindicators.Presage variablesdefinethecontextbeforestudentsstartlearning,processvariablesdescribe whatgoesonasstudentslearn,andproductvariablesrelatetotheoutcomesof thatlearning.forpresageandprocessvariablestheavailableevidenceisexamined concerningthevalidityofthevariable:theextenttowhichitpredictsstudent learningoutcomesandeducationalgains.eachproductvariableisexaminedforits abilitytoindicatecomparativequality.

4

dimensionsofqualiT y

1.3

T helimitationsofpresageandproductvariables

Presagevariablessuchasfunding,researchperformanceand,thereputationthat enablesaninstitutiontohavehighlyselectivestudententry,donotexplainmuch ofthevariationbetweeninstitutionsinrelationtoeducationalgains.measuresof educationalproductsuchasgradesdoreflectthesepresagevariables,butlargely becausethebeststudentscompetetoenterthebest-fundedandmostprestigious institutionsandthequalityofstudentsisagoodpredictorofproducts.measures ofproductsuchasretentionandemployabilityarestronglyinfluencedbyaraftof presagevariablesthatgowellbeyondthoseusedbyhefceinsettingperformance benchmarks.Thelackofcomparabilityofdegreestandardsprovesanobstacleto interpretationofstudentperformancedataintheuK.Thismakesinterpretingand comparinginstitutionalperformanceextremelydifficult. 1.4 Theimportanceofprocessvariables

whatbestpredictseducationalgainismeasuresofeducationalprocess:what institutionsdowiththeirresourcestomakethemostofwhateverstudentsthey have.Theprocessvariablesthatbestpredictgainsarenottodowiththefacilities themselves,ortodowithstudentsatisfactionwiththesefacilities,butconcerna smallrangeoffairlywell-understoodpedagogicalpracticesthatengenderstudent engagement.intheuKwehavefewdataabouttheprevalenceoftheseeducational practicesbecausetheyarenotsystematicallydocumentedthroughqualityassurance systems,norarethey(inthemain)thefocusofthenationalstudentsurvey. classsize,thelevelofstudenteffortandengagement,whoundertakesthe teaching,andthequantityandqualityoffeedbacktostudentsontheirworkareall validprocessindicators.Thereissufficientevidencetobeconcernedaboutallfourof theseindicatorsintheuK. 1.5 Theimportanceofmultivariateanalysis

fewrelationshipsbetweenasingledimensionofqualityandasinglemeasureofeither educationalperformanceoreducationalgaincanbeinterpretedwithconfidence becausedimensionsinteractincomplexwayswitheachother.Tounderstandwhat isgoingonanddrawvalidconclusionsitisnecessarytohavemeasuresofarangeof dimensionsofqualityatthesametimeandtoundertakemultivariateanalysis.largescalemultivariateanalyseshavebeenrepeatedlyundertakenintheus,andhave successfullyidentifiedthoseeducationalprocessesthataffecteducationalgains,and thosethatdonotorthatareconfoundedbyothervariables.incontrasttherehas beenlittleequivalentanalysisintheuK.ThisispartlybecausedataintheuKthat

5

ThehiGhereducaTionacademy

couldformthebasisofmultivariateanalysisforthatpurposearecurrentlycollected bydifferentagenciesandhaveneverbeenfullycollated. 1.6 Theimportanceofeducationalgain

becauseeducationalperformanceispredictedbytheentrystandardsofstudents, tocompareinstitutionalperformanceinavalidwayitisnecessarytomeasure educationalgain:thedifferencebetweenperformanceonaparticularmeasurebefore andafterthestudent'sexperienceofhighereducation.whilethemostinfluential usstudiesmeasureeducationalgaininavarietyofways,thereisverylittleevidence availableintheuKabouteducationalgain. 1.7 d imensionsofqualityindifferentkindsofinstitutions

institutionshavedifferentmissions,andcomparingthemusingproductdimensionsof qualitythatarethegoalsofonlyasubsetoftheinstitutionsleadstoconclusionsof doubtfulvalue.Processdimensionsgiveafairercomparativepictureofqualitythan dopresageorproductdimensions.however,differentpedagogicphenomena,and hencedifferentprocessvariables,arelikelytobesalientindifferentinstitutions.for example,onlysomeoftheverydifferentwaysinwhichTheopenuniversityorthe universityofoxfordachievesuchhighnationalstudentsurveyratingsarerelevant tootherkindsofuniversity. 1.8 d imensionsofqualityindifferentdepartments

indicatorsofdimensionsofqualityoftenvarywidelybetweendepartmentswithin thesameinstitution,foravarietyofreasons.Prospectivestudentsneedquality informationaboutthespecificdegreeprogrammetheywishtostudyataninstitution ratherthanaboutinstitutionalaveragesoraboutclustersofdegreeprogrammes aggregatedinto`subjects'asatpresent.Providingsuchinformationatasufficientlevel ofgranularitymaybeimpractical. 1.9 d imensionsofqualitythataredifficulttoquantify

studiesofthecharacteristicsofbothinstitutionsanddepartmentsthathavebeenfound tobeoutstandingintermsofvaliddimensionsofeducationalqualityhaveidentified processvariablesthatwouldbeextremelydifficulttoquantifyormeasureinasafeway, suchastheextenttowhichteachingisvalued,talkedaboutanddeveloped.

6

dimensionsofqualiT y

1.10

evidenceoftheproductsoflearning

oneofthemosttellingindicatorsofthequalityofeducationaloutcomesisthework studentssubmitforassessment,suchastheirfinal-yearprojectordissertation.These samplesofstudentworkareoftenarchived,butrarelystudied.Thereisconsiderable potentialforusingsuchproductsasmoredirectindicatorsofeducationalqualitythan proxiessuchasnssscores. 1.11 T hepotentialforimprovedquality,andtheevaluationof improvementsinquality

Thereisclearevidencethateducationalperformanceandeducationalgainscanbe enhancedbyadoptingcertaineducationalpractices.intheusthenationalsurvey ofstudentengagement(nsse)hasbeenusedsuccessfullybymanyinstitutions toidentifywherethereareweaknessesincurrenteducationalprocessesandto demonstratethepositiveimpactoftheintroductionofcertaineducationalpractices. Poolingdataacrosssuchinnovationsthenprovidesavalidbasistoguideother institutionsintheadoptionofpracticesthatarelikelytobeeffective.Thenss cannotbeusedintheuKinthesameway,despiteitsreliability.Thereisavaluable roletobefulfilledbynationalagenciesinsupportingtheuseofvalidmeasuresofthe impactofchangededucationalpractices,andinpoolingevidenceacrossinstitutions. 1.12 T hepotentialforinformingpotentialstudentsaboutquality

itseemsunlikelythatcomparativeindicatorsofqualitycurrentlyavailableintheuK couldprovideprospectivestudentswithavalidbasistodistinguishbetweenindividual courseswithregardtotheireducationalquality.Thecollationofcurrentlyavailable dataintoleaguetablesisinvalidandmisleading.evenintheuswherearangeof morevalidindicatorsaremorewidelyavailable,thoseresponsibleforcollectingand interpretingthedatacounselstronglyagainsttheircollationintoasingleleaguetable.

7

ThehiGhereducaTionacademy

2.

inTroducTion

Theextenttowhichindicatorsofqualityhaveshapedboththepoliticsofhigher educationandinstitutionalprioritiesisnotanewphenomenon(Patrickandstanley, 1998).however,thereiscurrentlyincreasedemphasisontheoverallqualityof undergraduateeducationintheuK.datafromanumberofrecentsurveysand studieshaveraisedchallengingissuesabout: -- differencesinqualitybetweeninstitutionswithintheuKthatinthepast have,rightlyorwrongly,beenassumedtobebroadlycomparable; -- differencesinqualitybetweennationalhighereducationsystems,towhominthe pasttheuKhasbeenassumed,rightlyorwrongly,tobesuperior,inthecontext ofanincreasinglycompetitiveinternationalhighereducationmarketplace; -- theadequacyofnationalqualityregimesthathaveemphasisedscrutinyof aninstitution'squalityassurancetoagreaterextentthanofitseducational processesoroutcomesofthekindemphasisedinsomeoftherecenthigh profilesurveysandstudies. aParliamentaryselectcommittee(houseofcommons,2009)hastaken evidencefromawiderangeofsourcesandreachedchallengingconclusionsbothabout thequalityofuKhighereducationandhowthatqualitycanbeassuredinthefuture. amongallthedebatetherehassometimesbeenuncriticalacceptanceof somesourcesofevidencethatcannotbeartheweightofinterpretation,andalso rejectionofevidencethatdeservestobetakenmoreseriously.eveninpublic reportsargumenthassometimesmadenouseofavailableevidence.Togiveone examplethequalityassuranceagency(2009)hasrespondedtodatathatsuggest boththatuKstudentsmightstudysignificantlylesshardthantheireuropean counterparts,andthattherearewidedifferencesbetweeninstitutionsandsubjects withintheuKinrelationtohowmanyhoursarestudied(hePi,2006,2007; brennanet al.,2009).fromtheperspectiveofthecurrentreportthekeyquestions inthiscaseare: -- doesitmatterthatsomestudentsreceivelessclasscontactthanothers? areclasscontacthoursanindicatorofquality? -- doesitmatterthatsomestudentsputinlesstotaleffortthanothers?are totalstudentlearninghoursanindicatorofquality?

8

dimensionsofqualiT y

insection5.2below,evidenceisreviewedthatmightinformtheqaa'scurrent positiononthisissue. similarlythefindingsofastudyofstudentexperiencebythenationalunionof students(nus,2008)mightbeinterpreteddifferentlyiftheywereinformedbythe availableempiricalevidenceontheissuesitaddresses,suchastheeffectsofpaid workonstudents'studyhours. Theliteratureonthevalidityofindicatorsofqualityisvast,widelydispersed andmostlyamerican.ittendstobefocusedonspecificpurposes,suchascritiquing aparticularuniversityleaguetable,critiquingaparticulargovernment-defined performanceindicator,establishingthecharacteristicsofaparticularstudentfeedback questionnaire,orexaminingthecharacteristicsofaparticularindicator(suchas researchperformance).muchofthisliteratureistechnicalinnatureandwrittenfor aspecialistaudienceofeducationalresearchers.Thecurrentreportattemptsto bringmuchofthisdiverseliteraturetogetherencompassingmany(thoughnotall) dimensionsofquality.itisnotintendedtobeanexhaustiveaccount,whichwouldbea veryconsiderableundertaking,anditiswrittenforageneralaudience.itwillnotdelve intostatisticalandmethodologicalminutiae,althoughsometimesanappreciationof statisticalissuesisimportanttounderstandingthesignificanceoffindings. Thisreportisintendedtoinformdebatebypolicyformersoffourmainkinds: thoseconcernedabouttheoverallqualityofuKhighereducation;thoseconcerned withinstitutionalandsubjectcomparisons;thoseconcernedwithfundingonthebasis ofeducationalperformanceandthosewithininstitutionsconcernedtointerpret theirownperformancedataappropriately.itmayalsobeusefultothosedirecting resourcesatattemptstoimprovequalityasitidentifiessomeoftheeducational practicesthatareknowntohavethegreatestimpactoneducationalgains. itisimportantheretobeclearwhatthisreportwillnotdo.itwillnotreview alternativequalityassuranceregimesormakeacaseforanyparticularregime. inidentifyingdimensionsofqualitythatarevaliditwill,byimplication,suggest elementsthatshouldbeincludedinanyqualityassuranceregime,andthosethat shouldnotbeincluded. ThereportwillnotbemakingoverallcomparisonsbetweentheuKandother hesystems,betweeninstitutionswithintheuK,betweensubjectsnationallyor betweensubjectsordepartmentswithininstitutions.ratherthepurposeisto identifythevariablesthatcouldvalidlybeusedinmakingsuchcomparisons. Thereportisnotmakingacaseforperformance-basedfunding.reviewsof theissuesfacingsuchfundingmechanismscanbefoundelsewhere(Jongbloedand vossensteyn,2001).however,validindicatorsofqualitywillbeidentifiedthatany performance-basedfundingsystemmightwishtoinclude,andinvalidindicatorswill beidentifiedthatanyperformance-basedsystemshouldeschew. finally,thereportisnotmakingacasefortheuseof`leaguetables'basedon combinationsofqualityindicators,nordoesitconsidertheissuesinvolvedinthe compilationanduseofexistingorfutureleaguetables.Trenchantandwell-founded

9

ThehiGhereducaTionacademy

critiquesofcurrentleaguetables,andoftheiruseingeneral,alreadyexist(bowden, 2000;brown,2006;clarke,2002;eccles,2002;GrahamandThompson,2001;Kehm andstensaker,2009;Thompson,2000;yorke,1997).someofthesecritiquescover similargroundtopartsofthisreportinthattheyidentifymeasurescommonlyused withinleaguetablesthatarenotvalidindicatorsofeducationalquality. Throughoutthereportthereisadeliberateavoidanceofusingindividual institutionsintheuKasexemplarsofeducationalpractices,effectiveorineffective, withtheexceptionofanumberofillustrationsbasedonTheopenuniversityand theuniversityofoxford.despitebeingfarapartinrelationtofunding,theyare closetogetheratthetopofrankingsbasedonthenss.Theyhaveachievedthis usingcompletelydifferenteducationalpractices,butthesepracticesembodysome importanteducationalprinciples.Theyaresodifferentfromotherinstitutionsthat therecanbelittlesenseinwhichtheycanbecompared,orcopied,exceptatthe levelofprinciples.itistheseprinciplesthatthereportseekstohighlight,because theyilluminateimportantdimensionsofquality.

10

dimensionsofqualiT y

3.

ThenaTureofdimensionsofqualiTy

3.1

conceptionsofquality

`quality'issuchawidelyusedtermthatitwillbehelpfulfirsttoclarifythefocus ofthisreport.Therehavebeenanumberofattemptstodefinequalityinhigher education,orevenmultiplemodelsofquality(e.g.chengandTam,1997).Themost commonlyciteddiscussionofthenatureofqualityinhighereducationintheuKis thatbyharveyandGreen(1993),andtheirhelpfulnomenclaturewillbeemployed here.first,qualityisseenhereasarelativeconcept­whatmattersiswhetherone educationalcontexthasmoreorlessqualitythananother,notwhetheritmeets anabsolutethresholdstandardsothatitcanbeseentobeofadequatequality, norwhetheritisreachesahighthresholdandcanbeviewedasoutstandingand ofexceptionalquality,norwhetheracontextisperfect,withnodefects.whatis discussedhereisthedimensionsthatarehelpfulindistinguishingcontextsfromeach otherintermsofeducationalquality. qualitymayalsobeseentoberelativetopurposes,whethertothepurposes andviewsofcustomersorrelativetoinstitutionalmissions.Thisreportdoes nottakecustomer-definedorinstitutionallydefinedconceptionsofqualityasits startingpoint.ratheraneffortwillbemadetofocusonwhatisknownaboutwhat dimensionsofqualityhavebeenfoundtobeassociatedwitheducationaleffectiveness ingeneral,independentlyofpossiblevariationsineithermissionsorcustomers' perspectives.Thereportwillthenreturntotheissueofinstitutionaldifferencesand willcommentinpassingondifferencesbetweenstudentsinthemeaningthatcanbe attachedtoqualityindicatorssuchas`drop-out'. afurtherconceptionofqualitymadebyharveyandGreenisthatofquality astransformation,involvingenhancingthestudentinsomeway.Thisconception comesintoplaywhenexaminingevidenceoftheeducationalgainsofstudents(in contrasttotheireducationalperformance).Thistransformationconceptionof qualityisalsorelevantwhenexaminingthevalidityofstudentjudgementsofthe qualityofteaching,wherewhattheymaywantteacherstodomaybeknownfrom researchevidencetobeunlikelytoresultineducationalgains.whatisfocusedon hereisnotnecessarilywhatstudentslikeorwant,butwhatisknowntoworkin termsofeducationaleffectiveness. itisusualtodistinguishbetweenqualityandstandards.Thisdistinctionismost relevantinsection6.1onstudentperformance,wheretheproportionof`good

11

ThehiGhereducaTionacademy

degrees'canbeseentobeinpartaconsequenceofthequalitiesofwhatstudents havelearntandinpartaconsequenceofthestandardsappliedinmarkingthe productsofstudentlearning.Thisreportwillnotfocusonstandardsthathavebeen thesubjectofmuchrecentdebate;forexample,concerningtheoperationofthe externalexaminersystem. 3.2 c ategorisingdimensionsofquality:presage,processandproduct

educationisacomplexbusinesswithmanyinteractingdimensionsofqualityinmany variedcontexts.Tounderstandwhatisgoingonitisnecessarytohaveawayof conceivingofthevariablesinvolvedandoforganisingandinterpretingstudiesofthe relationshipsbetweenthesevariables.Thisreportwilladoptthecommonlyused`3P' model(biggs,1993),whichapproacheseducationasacomplexsystemwith`Presage', `Process'and`Product'variablesinteractingwitheachother.The`3P'modelis essentiallythesameasthatusedbylarge-scalestudiesintheus(e.g.astin,1977, 1993):the`input-environment-output'model.Presagevariablesarethosethatexist withinauniversitycontextbeforeastudentstartslearningandbeingtaught,and includeresources,thedegreeofstudentselectivity,thequalityofthestudents,the qualityoftheacademicstaffandthenatureoftheresearchenterprise.noneofthese presagevariablesdeterminedirectlyhowtheeducationalprocessmaybeconducted, althoughtheyoftenframe,enableorconstraintheformeducationtakes. Processvariablesarethosethatcharacterisewhatisgoingoninteaching andlearningandincludeclasssize,theamountofclasscontactandtheextentof feedbacktostudents.Processvariablesalsoincludetheconsequencesofvariables suchasclasssizeforthewaystudentsgoabouttheirlearning,e.g.howthose variablesimpactonthequantityandqualityoftheirstudyeffortandtheiroverall levelofengagement. Productvariablesconcerntheoutcomesoftheeducationalprocessesand includestudentperformance,retentionandemployability.Productscanalsoinclude psychometricmeasuresofgenericoutcomesofhighereducation,suchasstudents' abilitytosolveproblems.insomestudiesthekeyproductmeasureisnotstudent performance,buteducationalgain:thedifferencebetweenperformanceona particularmeasurebeforeandafterthestudent'sexperienceofhighereducation.The differencebetweenperformanceandgainwillbecrucialinunderstandingdimensions ofquality,asweshallsee. Thecategorisationofvariablesaspresage,processorproductisnotalways straightforward.forexample,someprocessvariablessuchasthelevelofstudent engagementmayberelatedtootherprocessvariables,suchclasssize,whichmay inturnberelatedtofundinglevels.whicharethepresagevariablesandwhich theproducts?classsizeisnotseenasapresagevariableinthe3Pmodelasitis inpartaconsequenceofpolicydecisionsabouthowtouseresourcesandinpart

12

dimensionsofqualiT y

aconsequenceofeducationaldecisionsaboutteachingmethods.Thepresage variableofresourcesdoesnotnecessarilypredicteither.norisstudentengagement conceivedofinthe3Pmodelasaproduct.bothclasssizeandstudentengagement areconceivedofaspartoftheprocessesthatmayinfluenceeducationoutcomes, whicharecategorisedasproducts. inexaminingtheusefulnessofpotentialperformanceindicatorsinvolved in`leaguetables',presage,processandproductvariableshavesometimesbeen subdividedintomorecategorieswithinamorecomplexmodel(finnieandusher, 2005;usherandsavino,2006),butforthepurposesofthisreportthesimple3P modelwillsuffice. Thisreportexaminesawiderangeofpresage,processandproductvariablesin turn,andindoingsoidentifiesrelationshipsthatareknowntoexistbetweenthem.

13

ThehiGhereducaTionacademy

4.

PresaGedimensionsofqualiTy This section considers four presage dimensions of quality: funding, staff:student ratios, the quality of teaching staff and the quality of students

4.1 funding

institutionalfundingpredictsstudentperformancetosomeextent.itpredicts cohortsize(boundandTurner,2005),andclasssizepredictsstudentperformance (seesection5.1).fundingalsoaffectsthekindofteacherstheinstitutioncanafford toundertaketheteachingandthisaffectsstudentperformance(seesection4.3). howmuchfundingperstudentisallocatedtotheprovisionoflearningresources alsopredictsstudentstudyeffort,whichinturnpredictsstudentperformance (seesection5.2).however,fundingpredictsperformancelargelybecausethebest studentsgotothebest-resourcedinstitutionsandthequalityofthestudentspredicts theirperformance(seesection4.4).aseriesoflarge-scaleusstudieshavefound littleornorelationshipbetweeninstitutionalfundingandmeasuresofeducational gain(PascarellaandTerenzini,2005). eventheabilityofinstitutionallevelsoffundingtopredictstudentperformance issomewhatlimited.astudyintheushascomparedgroupsofcollegeswithnear identicalfundingperstudentandfoundgraduatecompletionratesvaryingbetween 35%and70%(ewell,2008),sothedifferencesinwhatcollegesdowiththeirfunding mustbeverywide.inaddition,institutionswithsimilarlevelsofperformancedisplay widelyvaryinglevelsoffundingwithsomereceivingonly60%oftherevenues perstudentthatothersreceive,butachievingnearidenticalperformanceona wholerangeofoutcomemeasures.Twentyinstitutionsthathadbeenidentified asunusuallyeducationallyeffective,inrelationtostudentretentionandlevelsof studentengagement,havebeenmatched,intermsofawholeraftofotherpotentially influentialpresagevariablessuchasstudentselectivity,withagroupofmarkedly lesseffectiveinstitutions.Therewasfoundtobenodifferenceintheleveloffunding allocatedtoteachingbetweentheeffectiveinstitutionsandthecarefullymatchedless effectivegroup(nchems,2003). whatdistinguishedtheseeffectiveinstitutionswasthatthefundingwasused differently;forexample,onfacultydevelopment(seesection5.3.1),teachingand learningcentresandacademicsupportstaffsuchastutorsandcounsellors(see section5.7.3).Theseusesoffundingwereexactlywhatpreviousandsubsequent studieshavefoundtobecharacteristicof"acampusethosdevotedtostudent success"(Gansemer-Topfet al.,2004).Thisissueofethosandvaluesisreturnedtoin theconclusion.

14

dimensionsofqualiT y

4.2

student:staffratios

whileattheleveloftheinstitutionstudent:staffratios(ssrs)mayseemtobe aninevitableconsequenceoffundinglevels,institutionsinpracticespendfunds onbuildings,onadministration,on`centralservices',onmarketing,onteachers undertakingresearch,andsoon,toveryvaryingextents,ratherthanspendingitall onteachingtime.Thedoublingoftuitionfeesintheusinrecentdecadeshasnot beenaccompaniedbyanyoverallimprovementinssrs,buthaslargelybeenused foradministrationandmeetingaccreditationrequirements.institutionsspendvery differentproportionsoftheiravailablefundingonteachers.sossrsmightbeseento beamoredirectindicatorofeducationalqualitythanfunding. lowssrsofferthepotentialtoarrangeeducationalpracticesthatareknownto improveeducationaloutcomes.first,closecontactwithteachersisagoodpredictor ofeducationaloutcomes(PascarellaandTerenzini,2005)andclosecontactismore easilypossiblewhentherearenottoomanystudentsforeachteachertomakeclose contactwith.lowssrsdonotguaranteeclosecontact,asharvard'srecentselfcriticismhasdemonstrated,buttheydomakeitpossible. second,thevolume,qualityandtimelinessofteachers'feedbackonstudents' assignmentsarealsogoodpredictorsofeducationaloutcomes(seesection5.6),and againthisrequiresthatteachersdonothavesomanyassignmentstomarkthatthey cannotprovideenough,high-qualityfeedback,promptly.again,lowssrsdonot guaranteegoodfeedbackorfeedbackfromexperiencedteachers.intheuKturnroundtimesforfeedbackmaybeamatteroflocalpolicyratherthandrivenbyssrs andturnaroundtimesvaryenormouslybetweeninstitutions(GibbsanddunbarGoddet,2009). Third,whilelowssrsdonotguaranteesmallclasses,theycertainlymakethem possible,andclasssizepredictsstudentperformance(seesection5.1below). however,oncestudententrycharacteristicsaretakenintoaccount,educational gainshavebeenfoundtobelargelyunrelatedtossrs(TerenziniandPascarella, 1994).Thissuggestseitherthatinstitutionswithlowssrsarenotexploitingtheir potentialadvantagesthroughtheuseofeffectiveeducationalpracticesorthatssr figureshideothervariations,orboth. ssrsreportedatinstitutionalleveldonotnecessarilygiveagoodindication ofthessrsstudentsactuallyexperience.Patternsofworkvary;forexample, academicsdoagreaterproportionofadministration,withfewersupportstaff,in someinstitutions,effectivelyreducingtheiravailabilitytostudents.Theyundertake moreresearchinsomeinstitutionswhiletheproportionoftheirresearchtime fundedbyresearchincomevaries.Thedifferencebetweenstudents'yearsofstudy canbemarked,withmuchgreaterfundingperstudentcharacteristicallybeing allocatedtothird-yearcoursesthantofirst-yearcourses,leadingtobetterssrsand smallerclassesinthethirdyear(andthenationalstudentsurveyisadministered inthethirdyear).furthermoreinstitutionsdonotallocatefundingtodepartments

15

ThehiGhereducaTionacademy

instrictproportiontostudentnumbers,buttakeadvantageofwell-recruitingand income-generatingcourses,andsoincreasetheirssrs.Theoverallconsequenceis thateffectivessrs,asstudentsexperiencethem,withinayearoftheirstudy,withina department,maybearlittleresemblancetossrsreportedatinstitutionallevel. Therearealsocommonlyexpressedconcernsabouttheaccuracyandmeaningof ssrdataasreportedtoinstitutions,andasmadepublicbyhesa,duetovariations inhowtheyarecalculated,complicatedbyvariationsinthenatureofemploymentof teachersandhowtheyaredeployed(seesection4.3below). 4.3 qualityofteachingstaff

baldssrdataareunhelpfulinthattheydisguisetherealitiesofwhothestaffare withwhomstudentshavecontact.forexample,undergraduatesatyaleoftendo notreceivefeedbackfromtenuredfacultyuntiltheirthirdyear.inusresearch universitiestheteachingundertakenbygraduateteachingassistantsisaconstant qualityconcernandisregularlycitedinstudentexitsurveysastheirnumberone complaintaboutthequalityofteaching. anhourofagraduateteachingassistantmaycostafractionofanhourofa tenuredacademic,andmostinstitutionsarequicktoexploitthis.recentsurveys (hePi,2006,2007)revealwidevariationsbetweeninstitutionsintheproportionof teachingthatstudentsexperiencethathasbeenundertakenbyresearchstudents asopposedtotenuredacademics.Themajorityofsmallgroupteachingwasfound tobeundertakenbyteachersotherthanacademicsatrussellGroupandpre1992universities.attheuniversityofoxfordtheextenttowhichstudentstakea `surfaceapproach'totheirstudy,emphasisingonlymemorisation(seesection5.5.2 below),islinkedtotheproportionoftheirtutorialstakenbyteachersotherthan collegefellows(Trigwellandashwin,2004).amuchlowerproportionofteachingis undertakenbyresearchstudentsatoxfordthanatotherrussellGroupuniversities. I am currently an undergraduate at a Russell Group University... I have three hour-long lectures (taught by world-class professors) and three hour-long group seminars (taught by unpaid postgraduate students) a week. That is it. If we are going to compromise social mobility and academic freedom in order to fund our universities better, we damn well ought to receive a standard of education that's worth the sacrifice. --lettertoEducation Guardian,25may,2010 inteaching-orientedinstitutionswithfewresearchstudentswhomightdothe teaching,asignificantproportionofteachingmaybeundertakenbywhattheus terms`adjunctfaculty'whomayhaveportfolioteachingcareersspanninganumber ofinstitutions,withanofficeinnoneofthem.invocationalandcreativeartscourses

16

dimensionsofqualiT y

studentsmayexperienceasignificantproportionoftheirteachingundertakenby professionalswhomayspendonlyafewhoursaweekoncampus.adjunctfaculty andprofessionalsmaybringspecialcharacteristicsandtalentstotheirwork,butthey mayneverhavetheopportunitytodevelopacourseoverseveralyearssothatit providesarichandeffectivelearningenvironmentforstudents.Theymaynotbepaid tomeetstudentsoutofclassortoprovidedetailedcommentsontheirassignments. Theymayneverattenddepartmentalmeetingsorfullyunderstandhowthedegree programmeworkstowhichtheirteachingissupposedtocontribute.studentsmay notknowwhothefull-timetenuredacademicsareandwhoarenot­theywillsimply experienceteachingofvaryingqualityandcommitment.whilethehePidatauncover variationintheproportionofteachingundertakenbyresearchstudents­who studentscanseearemainlyyoung­itisdifficultforstudentsurveystoidentifythe proportionofteachingundertakenbyadjunctfacultyandhePidatadonotilluminate thisissue. intheustheproportionofpart-timeandnon-tenure-track`adjunctfaculty'has increasedenormouslyinpublicinstitutions.institutionswithhigherproportionsof adjunctfacultyhavelowergraduationrateswhenotherfactorsareheldconstant.an increaseof10%inpart-timefacultyisassociatedwithareductionof3%ingraduation rates.withininstitutions,first-yearstudentstaughttoagreaterextentbyadjunct facultyarelesslikelytopersistintothesecondyear(ehrenberg,2006). 4.4 qualityofstudents

intheus,byfarthebestpredictorofstudents'educationaloutcomeswhetherthe measureisgrades,apsychometrictestofprincipledreasoning,orcareersuccess,is theirschoolsaTscorewhentheyentercollege,withcorrelationsintherange0.85 to0.95.inotherwordsupto90%ofallvariationinstudentperformanceatuniversity cansometimesbeexplainedbyhowtheyperformedbeforetheyentereduniversity. intheuKthelinkislessstrong,buttherehasfordecadesbeenclearevidenceof theextensiveimpactofschoolingonstudentperformanceinhighereducation,both intermsofschoolleavinggradesandtypeofschool(smithandnaylor,2005).inthe uKstudentsfromindependentschoolsperformlesswellthandostudentsfromstate schoolswithequivalententrygrades(hoskinset al.,1997;smithandnaylor,2005). Thequestionthatthenarisesiswhetheranyofthisenhancedperformanceisdue toqualitiesoftheinstitutionotherthantheirabilitytobehighlyselectiveonentry. large-scalelongitudinalstudiesofadiverserangeofinstitutionshavetestedstudents abouttheiracademicbehaviourandexperience(includingtheirengagement,see section5.5.3below)fromatotalofnearly300collegesandinvolvingdatafromnearly 80,000students(forasummaryofthisworkseeKuhandPascarella,2004).These studieshavefoundverylittlerelationshipbetweeneducationalselectivity(i.e.quality ofstudentintake)andtheprevalenceofwhatareknowntobeeducationallyeffective

17

ThehiGhereducaTionacademy

practices.selectivitywasfoundtobenegativelyassociatedwithsomepractices,such astheamountofteacherfeedbacktostudents,andevenwheretherewerefoundto besmallpositiverelationships(forexamplewithhigherexpectationsonstudents), selectivityonlyaccountedfor2%ofthevarianceineducationalpractices. itmightbearguedthatselectiveinstitutionsdonotneedspecialeducational practicesbecausetheirstudentsareableenoughtoengagethemselves.however,the degreeofselectivitydoesnotpredictthedegreeofstudentengagement­students arejustasengaged(orunengaged)innon-selectiveinstitutions(Pascarellaet al., 2006).sowhileleaguetablesintheuKinvariablyincludea-levelpointscoresasan indicatorofeducationalquality,iftheusevidenceisanythingtogobytheytellus almostnothingaboutthequalityoftheeducationalprocesswithininstitutionsorthe degreeofstudentengagementwiththeirstudies. itmightbearguedthatthereareeducationalbenefitstoastudentofbeing surroundedbyotherablestudents.Thiscouldraisestudents'expectationsof themselves(onetheof`sevenPrinciples')1,anditisknownthatingroupworkit isthepreviouseducationalattainmentofthebeststudentinthegroupthatbest predictsthegroupgrade,nottheaveragelevelofpriorattainmentorthelevelof theweakeststudent(Gibbs,2010).wewouldthenneedtolookattheextentto whichtheeducationalprocessmaximiseshowstudentscouldgainfromeachother, forexamplethroughcollaborativelearning.Theextentofcollaborativelearningisa goodpredictorofeducationalgains(the`sevenPrinciples'again).however,itwillnot helpastudentmuchiftheotherstudentsarehighlyableiftheythenengagelargely insolitarycompetitivelearning.Theusdatacitedabovemakeitclearthatstudents arenotmorelikelytobeinvolvedincollaborativelearning,ortobeengagedbyit,in institutionswithmoreselectiveentry,inwhichthestudentsaremoreable. studentsbringmoretohighereducationthantheira-levelscores.itislikelythat theirculturalcapital,theiraspirations,self-confidenceandmotivationsallinfluence theirperformanceandinteractwithteachingandcoursedesignvariables.

1

The`sevenPrinciplesofGoodPracticeinundergraduateeducation'(chickeringandGamson 1987a,1987b,1991)arebasedonaverywidereviewofempiricalevidence,andhavebeen usedwidelyintheusandelsewhereasguidestotheimprovementofuniversityteaching.The principlesarethatgoodpractice:encouragesstudent-facultycontact,encouragescooperation amongstudents;encouragesactivelearning;providespromptfeedback;emphasizestimeontask; communicateshighexpectations;andrespectsdiversetalentsandwaysoflearning.

18

dimensionsofqualiT y

5. Processdimensions This section considers the effects on educational effectiveness of class size, class contact hours, independent study hours and total hours, the quality of teaching, the effects of the research environment, the level of intellectual challenge and student engagement, formative assessment and feedback, reputation, peer quality ratings and quality enhancement processes

5.1

classsize

meta-analysisoflargenumbersofstudiesofclass-sizeeffectshasshownthatthe morestudentsthereareinaclass,thelowerthelevelofstudentachievement(Glass andsmith,1978,1979).otherimportantvariablesarealsonegativelyaffectedbyclass size,suchasthequalityoftheeducationalprocessinclass(whatteachersdo),the qualityofthephysicallearningenvironment,theextenttowhichstudentattitudes arepositiveandtheextentofthemexhibitingbehaviourconducivetolearning(smith andGlass,1979).Thesenegativeclass-sizeeffectsaregreatestforyoungerstudents andsmallestforstudents18orover(ibid.),buttheeffectsarestillquitesubstantialin highereducation.lindsayandPaton-saltzberg(1987)foundinanenglishpolytechnic that"theprobabilityofgainingan`a'gradeislessthanhalfinamoduleenrolling 50-60thanitisinamoduleenrollinglessthan20"(p218).allsubsequentuKstudies havereportedsizablenegativecorrelationsbetweenclasssize(asmeasuredby thenumberofstudentsregisteredonacourse)andaveragestudentperformance, inmostbutnotallsubjects,andinmostbutnotallcontexts(Gibbset al.,1996; fearnley,1995).largeclasseshavenegativeeffectsnotonlyonperformancebutalso onthequalityofstudentengagement:studentsaremorelikelytoadoptasurface approachinalargeclass(lucaset al.,1996)andsotoonlytrytomemoriserather thanattempttounderstand(seesection5.5.2ondepthofapproachtolearning). atamicro-levelthereisevidencethattheeducationalprocessiscompromisedas classsizeincreases.inhighereducationdiscussiongroups,forexample,awholerange ofthingsgowrongasclasssizeincreases.Thereisamuchlowerlevelofparticipation byallbutaminorityofstudentsandthecontributionsthatstudentsdomaketendto concernclarificationoffactsratherthanexplorationofideas(baleset al.,1951).

19

ThehiGhereducaTionacademy

usresearchshowsthathighereducationstudentsgiveloweroverallratings toteachersoflargeclasses(woodet al.,1974;feldman,1984).however,thereare reasonstoquestionthemeaningofthisfinding.Thesameteachersaregivenhigher ratingswhentheyteachsmallerclasses.assuchratingsofteachersarerelatively reliableandstablethissuggeststhatstudents'ratingsofteachersinlargeclasses arereflectingsomethingotherthantheteachersthemselves.aqualitativestudyof students'experienceoflargeclasses(GibbsandJenkins,1992)hasthrownlighton variablesotherthantheteaching.Theremaybeintensestudentcompetitionfor limitedlibraryandotherresourcesinlargeclasses,andteachersmayhavetorely onafewtextbooksifstudentsaretoreadanything.Theamountandpromptnessof feedbackonassignmentsislikelytodecline,asteachertimeissqueezed.Thenature ofassessmentsmaychangefromengagingopen-endedprojectstoquicktests,as markingotherwisetakestoolong.closecontactwithteachersoutsideofclassand accesstoremedialtutoringandadvicemaybemorelimited.largeclassesmaybe associatedwithweaksocialcohesion,alienationandabreakdowninsocialbehaviour, leadingtocheating,hidinglibrarybooks,andsoon.allthisismoretodowith whathappensoutsideofclassoncourseswithlargeenrolments,ratherthanwhat happensinclass,butitisclassroomactivitythatisthefocusofmostschool-based researchandushighereducationresearch.whereout-of-classstudyingisthemajor componentofstudentlearningthecrucialvariablemaybecourseenrolmentrather thanclasssize.usdatashowthatcohortsizeisstronglynegativelycorrelatedwith studentperformance(boundandTurner,2005). anotherdifferencebetweenschoolandhighereducationinrelationtoclasssizeeffectsisthatinhighereducationtherangeofclasssizesbeingstudiedisvery muchwider:perhaps20to1,000insteadof10to40inschools.differentvariables inevitablybecomeprominentinsuchverylargeclasses.inschool,studentsmay experiencealltheirclassesasmuchthesamesize.inhighereducationwhatmay mattermostisnotthesizeofthelargestlecturethatisattendedonanyparticular coursebutthesizeofthesmallestseminargrouporproblemclassthattheyattend withinthesamecourse.openuniversitystudentsmayattendacoursewithan enrolmentofover10,000,buttheyusuallyonlyexperienceatutorgroupof24, andeachtutorusuallyhasonlyonetutorgroupsotheycangettoknowstudents individually.attheopenuniversityitwouldprobablymakeadifferenceifthistutor groupwas12or48butnotiftotalenrolmentwas500or20,000. classroomsusedforspecialistpurposes,suchaslaboratoriesandstudios,usually limitthenumberofstudentsitispossibletoteachatonce,regardlessofhowmany studentshaveenrolled,andalthoughlaboratorieshavebecomemuchlarger,there arelimitstoclass-sizeeffectswhilewithinthelab.however,increasedenrolments withfixedspecialistspaceshavetheinevitableconsequenceofreducingtheamount oftimestudentshaveaccesstothesespecialistfacilities.Thishastransformedartand designeducation.insteadofstudents`owning'apermanentspacetheycantaketime tobecomecreativeandcompetentin,whenenrolmentincreasestheyvisitashared

20

dimensionsofqualiT y

spaceoccasionally.Thenumberofstudentsinthestudioatanyonetimemaynothave changedmuchbutartstudents'experiencehasbeenchangedoutofallrecognition. Gibbset al.(1996)foundthatinart,designandthePerformingarts,eachadditional12 studentsenrolledonacoursegaverisetoadeclineof1%inaveragemarks. negativeclass-sizeeffectsarenotinevitableandacertainamountisknownabout howtosupportgoodqualitylearningdespitelargeclasses(GibbsandJenkins,1992). TheTeachingmorestudentsinitiativeintheearly1990strained9,500polytechnicand collegelecturersontheassumptionthatsuchimprovementswerepossibledespite largerclasses(Gibbs,1995).ThenationalcentreforacademicTransformationinthe ushashelpedscoresofinstitutionstoredesignlarge-enrolment,first-yearcourses. Theyhaveshownthatitispossibletoimprovestudentoutcomeswhilereducing teachingcontacttimeandreducingfunding.Theopenuniversityhasretainedthe numberandnatureofassignmentspercourse,theamount,qualityandturnaround timeoffeedbackfromtutors,andthesmallsizeoftutorgroups,throughstrictcourse approvalrules,withcourseenrolmentsthatareseldombelow500. Theconundrum,ofcourse,isthatintheuKoverallstudentperformance hasincreasedatthesametimethatoverallclasssizehasincreased.Thisissueis addressedinsection6.1. 5.2 c lasscontacthours,independentstudyhoursandtotalhours

Thenumberofclasscontacthourshasverylittletodowitheducationalquality, independentlyofwhathappensinthosehours,whatthepedagogicalmodelis,and whattheconsequencesareforthequantityandqualityofindependentstudyhours. independentstudyhours,toalargeextent,reflectclasscontacthours:ifthereis lessteachingthenstudentsstudymoreandifthereismoreteachingstudentsstudy less,makinguptotalhourstosimilartotalsregardlessoftheratioofteachingto studyhours(vos,1991).however,somepedagogicsystemsuseclasscontactinways thatareverymuchmoreeffectivethanothersatgeneratingeffectiveindependent studyhours.areviewofdatafromanumberofstudiesbyGardiner(1997)found anaverageofonly0.7hoursofout-of-classstudyingforeachhourinclass,inus colleges.incontrasteachhouroftheuniversityofoxford'stutorialsgenerateon average11hoursofindependentstudy(Trigwellandashwin,2004)andoxford's studentshavebeenfoundtoputinthegreatestoverallweeklyeffortintheuK despitehavingcomparativelyfewerclasscontacthours(hePi,2006,2007).what seemstomatteristhenatureoftheclasscontact.`closecontact'thatinvolves atleastsomeinteractionbetweenteachersandstudentsonapersonalbasisis associatedwithgreatereducationalgains(Pascarella,1980)independentlyofthetotal numberofclasscontacthours(PascarellaandTerenzini,2005);theprovisionofclose contactisoneofthe`sevenprinciplesofgoodpracticeinundergraduateeducation' (chickeringandGamson,1987a,1987b,1991).

21

ThehiGhereducaTionacademy

Seven principles of good practice in undergraduate education Good Practice Encourages Student-Faculty Contact Good Practice Encourages Cooperation among Students Good Practice Encourages Active Learning Good Practice Gives Prompt Feedback Good Practice Emphasizes Time on Task Good Practice Communicates High Expectations Good Practice Respects Diverse Talents and Ways of Learning --chickeringandGamson,1987a

asanillustrationofthelackofrelationshipbetweenclasscontacthoursand outcomes,medicaleducationworldwidehasmigratedfromtraditionaldidactic pedagogies,characterisedbylargenumbersoflargeclasslectures,towards problem-basedpedagogies,characterisedbyamuchsmallernumberofsmall interactiveclasses,usuallywithinthesameresourceenvelope.Thischangehasbeen accompaniedbyasubstantialincreaseinindependentlearninghoursandevidenceof greaterpedagogicaleffectivenessmeasuredinavarietyofways(dochyet al.,2003). similarlytheopenuniversityhasmanagedtotopthenationalstudentsurveyleague tableswhilehavingverymuchthelowestclasscontacthoursintheuK. Thisisnotthesameasarguingthatyoucancutclasscontacthoursfroman existingunchangedpedagogywithoutmakinganydifferencetostudentlearning,or thatincreasinghourswillmakenodifference.ifstudentsreadprimarilyinorderto discusswhattheyhavereadinaseminar,andtheseminaristakenaway,thenthey willprobablyreadagooddeallessandlearnlessasaconsequence.verylittleclass contactmayresultinalackofclarityaboutwhatstudentsshouldbestudying,alack ofaconceptualframeworkwithinwhichsubsequentstudycanbeframed,alackof engagementwiththesubject,alackoforalfeedbackontheirunderstanding,and soon.itdependswhatroletheclasscontactisperforming.whatmattersisthe quantityandqualityofengagementgeneratedbytheparticularusestowhichclass contactisput. whatseemstobemoreimportantthanclasscontacthoursisthetotalnumber ofhoursthatstudentsputin,bothinandoutofclass(thequalityofthateffortis consideredinsection5.5below).Thenumberofhoursthatstudentsstudyoutofclass isusuallymeasuredinoneoftwoways:byaskingstudentstoestimate,retrospectively, howmuchtheythinktheyhavebeenstudying,often`onaverage',orbyaskingstudents tokeepalog,astheystudy,ofwhattheydowiththeirtime,overarelativelyshort periodoftime(suchasaweek).whilelogsarelikelytobemoreaccuratethan retrospectiveestimates,theyalsoruntheriskofchangingstudentstudybehaviour asstudentsbecomemoreawareoftheirpatternandlevelofeffort.indeedkeeping

22

dimensionsofqualiT y

alogisacommonlearningactivityonstudyskillscourses.whenaskingstudents toestimatetheirstudyhoursretrospectively,theformofthequestionusedvaries betweendifferentsurveysandthetimingofthesurveysvariesinrelationtohowlong agostudentsareattemptingtorememberorhowwideaspreadofcoursestheyare beingaskedtomakeaverageestimatesacross.studentswhoattendlessandstudyless maybemissedbysurveyswhileconscientiousstudentswhoattendmoreandstudy moremaybemorelikelytoreturnsurveys.Theimpactofsuchpotentialbiasesisnot wellresearchedandthereliabilityofstudy-hoursdataisnotknown. Thequestion:`arehigherstudyhoursassociatedwithbetterstudentlearning andperformance?',canbeposedintworatherdifferentways.first:`arethestudents whostudylongerhourstheonesthatperformbest?'Theanswertothisquestion isnotstraightforward(stinebricknerandstinebrickner,2008),becauseveryable studentsmaybeabletomeetassessmentrequirementswithouthavingtostudy veryhard,whilelessablestudentsmayputinmanyhoursunproductively(ashbyet al.,2005).Thereisalsoevidencethatstudentswho,inappropriately,takea`surface' approachtotheirstudies(seesection5.5.2below)findthissounproductivethatthey graduallyreducetheireffortafterinitiallyworkinghardandendupstudyingfewer hoursthanstudentswhotakea`deep'approach(svensson,1977). if,however,thequestionisframeddifferentlyas:`ifastudentweretostudy morehours,wouldtheyperformbetter?'oreven`ifaveragestudyhoursonadegree programmewerehigher,wouldaverageperformancebehigher?',theanswerismuch moreclearly`yes'.`Timeontask'isoneoftheevidence-based`sevenPrinciplesof GoodPracticeinundergraduateeducation(chickeringandGamson,1987).The reasonableassumptionhereisthatifyoudon'tspendenoughtimeonsomethingthen youwon'tlearnit,andthatincreasingthenumberofhoursstudentsspendstudying isoneofthemosteffectivewaysofimprovingtheirperformance.northamerican researchanddevelopmentworkon`studentengagement'(seesection5.5.3below) usesstudenteffortasanimportantindicatorofengagement. Thebolognaprocesshasusedtotalstudenteffort(classcontacthoursplus independentstudyhours)asitsmetricfordefiningthedemandsofabachelors degreeprogramme,setat1,500to1,800hoursayear:4,500to5,200hoursover threeyears.aseriesofstudieshavefoundthatuKstudents'totalweeklyeffortin hoursislowerthanincomparisonwitheithertheparticulareuropeancountries studiedorincomparisonwithoveralleuropeannorms(brennanet al.,2009; hochschul-informations-system,2005;sastryandbekhradnia,2007;schomburg andTeichler,2006).Thesefindingsdeservetobetakenseriouslybecausetheyare relativelyconsistentacrossdifferentstudiesandmethodologies,carriedoutin differentcountries,oracrossrepetitionsofthesamestudyindifferentyears. itshouldbepossibletoironoutgrossdifferencesbetweeninstitutionsand subjectareas,asthenumberofstudyhourspercredit,andhencethenumber ofhoursrequiredforabachelorsprogramme,areclearlydefinedincourse documentation.however,therangeinweeklystudyeffortbetweenenglish

23

ThehiGhereducaTionacademy

institutions,withinsubjects,foundinthehePistudiesiswide,forexamplefrom14 hoursaweektonearly40hoursperweekwithinPhilosophy(sastryandbekhradnia, 2007).differencesbetweensubjectsarealsowide.broaddifferencesintotalstudy hoursbetweenscienceandtechnologyprogrammes(whichtendtohavebothhigh classcontacthoursandweeklydemandsforworksuchasproblemsheetsand laboratoryreports)andthehumanities(whichtendtohavebothlowerclasscontact hoursandlessregularassignmentssuchasessays)arewellknownandhavebeen reportedfrequentlyovertheyears(e.g.vos,1991).however,thedifferencesbetween subjectsidentifiedbythehePisurveysaresubstantial,withsomesubjectshaving nationalaverageweeklystudyeffortsofonlyaround20hoursperweek.Twenty hoursperweekwithinthecomparativelyshortuKsemestersequatestoaround 500hoursayear:onethirdoftheminimumspecifiedunderthebolognaagreement. Toachievethebolognaspecificationofaminimumof4,500hoursforabachelors programme,studentsinthesesubjectsintheuKwouldhavetostudyfornineyears. differencesonthisscalecannoteasilybearguedawaybyclaimingthatuKstudents aresomehowinherentlysuperiororthatuKeducationalpracticesaresomehow inherentlymoreefficient,intheabsenceofanyevidencetobackupsuchclaims. asurveyofinternationalstudentswhohaveexperiencedbothauKhigher educationinstitutionandanothereuhighereducationinstitution(brennanet al., 2009)foundthatsuchstudentsaremorelikelytorateuKbachelorsprogrammesas `lessdemanding'andlesslikelytoratethemas`moredemanding',afindingthatdoes notjustifythelowernumberofhoursinvolved.uKstudentshavebeenreported tohavedonemoreworkthanwasrequiredofthemtoagreaterextentthaninany othereuropeancountry(ibid.).yetthetotalnumberofhoursstudiedintheuKisstill beloweuropeannorms,whichsuggeststhattheuKrequirementsmustbelower. ifitwerethecasethatlessablestudentsneededtostudymore,thenone wouldfindthelargerstudyhoursfiguresininstitutionsthathavestudentswith weakereducationalbackgrounds.insteadthereverseisthecase,withhigherweekly studyhoursreportedininstitutionswithstudentswiththestrongesteducational backgrounds(sastryandbekhradnia,2007).Themostlikelyexplanationthereforeis thatthedemandsmadeonstudentsaredifferentindifferentinstitutions,andthateven weakerstudentsareabletomeetthesedemandswhilestudying,insomeinstitutions, andinsomesubjects,athirdofthehoursthebolognaagreementspecifies. Thereareanumberofpossibleexplanationsofwhysuchstudentsmightstudy fewhours: -- somedegreeprogrammesrelyheavilyoncourseworkassessment,with over90%ofmarksonsomedegreeprogrammescomingfromcoursework ratherthanonexaminations.whilethismayhavecomeaboutforsound educationalreasons,itmayallowstudentstostudyonlyoneortwotopics indepthpercourse,becausethatisalltherearetheteachingresources tomark,reducingstudenteffortinweekswhentherearenoformal

24

dimensionsofqualiT y

assessmentdemands.studentshavebeenfoundtoworkmoreregularlyand coverthesyllabustoagreaterextentwhenthereisahigherproportionof marksfromexaminations(Gibbsandlucas,1997). -- highlevelsofdetailincoursespecifications,oflearningoutcomesand assessmentcriteria,inresponseinparttoqaacodesofpractice,allow studentstoidentifywhattheyoughttopayattentionto,butalsowhatthey cansafelyignore.arecentstudyhasfoundthatinsuchcoursesstudentsmay narrowtheirfocustoattentiontothespecifiedassessedcomponentsatthe expenseofeverythingelse(Gibbsanddunbar-Goddet,2007).studentshave becomehighlystrategicintheiruseoftimeandadiarystudyhasfoundstudents toprogressivelyabandonstudyinganythingthatisnotassessedastheywork theirwaythroughthreeyearsoftheirdegree(innisandshaw,1997). -- studiesoftheeffectsofpaidworkundertakenbyfull-timestudentshave reportedasubstantialreductioninstudyhoursinrelationtotheextentof theirpaidwork,althougheffectsvarysomewhatbetweencontextsanddegree programmes(carneyet al.,2005;curtisandwilliams,2002;fordet al.,1995; huntet al.,2004;Paton-salzbergandlindsay,1993).atypicalstudyisthatof svanumandbigatti(2006),whofoundthatpaidworkreducedcourseeffort andreducedgrades.intheus,studentsrespondtotheneedto`worktheir waythroughcollege'bytakingfewercreditsatatimeandgainingtheirdegree overmoreyears(andacrossseveralinstitutions)asaconsequence.intheuK full-timestudentsseemtoassumethattheycanundertakepaidworkwhile meetingthedemandsofafull-timecourseandstillgraduateinthreeyears. -- studentswholiveathome,ratherthanonaresidentialcampus,arelikelyto experiencecompetingdemandsontheirtime,andlesssocialandacademic integration(Tinto,1975).TheinstitutionsintheuKwiththelowestaverage studyhoursincludeuniversitiesinurbanconurbationswithasubstantial proportionofstudentslivingathome. -- Theuniversitieswithlowaveragestudyhoursareoftenalsoinstitutionswith lowannualinvestmentperstudentinlibrariesandotherlearningresources.This wouldmakeitmoredifficultforstudentstogainaccesstotheresourcesthey needfortheirstudy:thebookwillbeoutandthestudyspacewithacomputer willbeoccupied.datafromhesa,hePiandthenationalstudentsurveyhave beenanalysedforthepurposeofthecurrentreportonthisissue.Theyshowed thatinstitutionalfundsallocatedtolearningresources,perstudent,predicttotal studentlearninghours(withcorrelationsof+0.45forthesocialsciencesand humanitiessubjectsanalysed).fundingforlearningresourcesalsopredictsaverage students'responsestothenssquestiononthequalityoflearningresources,

25

ThehiGhereducaTionacademy

althoughlesswell.Theinstitutionwiththehighestweeklyaveragestudyhoursalso hasthegreatestannualinvestmentinlearningresourcesandthehighestnational studentsurveyratingsfor`learningresources':theuniversityofoxford. whilethefocusofthisreportisonundergraduateprogrammes,therehasrecently beenagooddealofattentionpaidtotherelativequalityofuKmasterslevelcourses, giventhattheyareusuallyplannedtobeconsiderablyshorterindurationthantheir mainlandeuropeancounterparts.forexample,thechairoftheuKcouncilforGraduate educationhasarguedthatthesegrossdifferencesdonotmatterbecauseuKmasters coursesare`moreintensive',andclaimedthattheoverallamountoflearningtimeis roughlyequalbetweentheuKandmainlandeurope(olcott,2010).Thisunsubstantiated claimcouldbecheckedbyrepeatingthesastryandbekhradniaundergraduatestudyof studyhours,citedabove,inmasterscoursesandaddingquestionstomeasuretheextent towhichstudentstakeadeepapproachtotheirstudies(seesection5.5.2below). 5.3 5.3.1. qualityofteaching qualityofteaching:experienceandtraining

Teacherswhohaveteachingqualifications(normallyaPostgraduatecertificatein highereducation,orsomethingsimilar)havebeenfoundtoberatedmorehighlyby theirstudentsthanteacherswhohavenosuchqualification(nasret al.,1996).This findingwasinacontextwhereobtainingsuchaqualificationwaslargelyvoluntary, andthosewhohavethequalificationmightbeconsideredtobedifferentinsomeway fromthosewhohavenot,andthiscouldbearguedtoinvalidatethecomparison.The differencemightconcerntheextentofprofessionalismorcommitmenttoteaching, butneverthelesstherewasnocontrolgroupinthestudy.alongitudinalstudythat overcomesthisobjectionhasexaminedtheimpactovertimeonstudents'ratings oftheirteachers,andonteachers'thinkingaboutteaching,of(mainly)compulsory initialtrainingduringtheirfirstyearofuniversityteaching,ineightcountries.itfound improvementsoneveryscaleofthe`studentevaluationofeducationalquality',a questionnairedevelopedintheus(marsh,1982)andtestedforuseintheuK(coffey andGibbs,2000),andimprovementsinthesophisticationofteachers'thinking(as measuredbythe`approachestoTeachinginventory',ameasureofteachingthat predictsthequalityofstudentlearning,Trigwellet al.,2004).Thisimprovement inmeasuresofteachingqualitycouldnotbeattributedtomerematurationor experienceasteachersinacontrolgroupininstitutionswithoutanyinitialtraining werefoundtogetworseovertheirfirstyear,onthesamemeasures(Gibbsand coffey,2004).ThequalityofschoolteachingintheuKisinpartunderpinnedby abeliefthatinitialorin-servicetrainingisessentialandthe,admittedlylimited, evidencesuggeststhatthesamebeliefwouldbejustifiedinuKhighereducation.

26

dimensionsofqualiT y

5.3.2.

qualityofteaching:researchrecord

Thereislittleornorelationshipbetweenmeasuresofthequalityorquantityof teachers'researchandmeasuresofthequalityoftheirteaching(forareviewof58 studiesoftheevidence,seehattieandmarsh,1996). ...the common belief that teaching and research were inextricably intertwined is an enduring myth. At best teaching and research are very loosely coupled. --hattieandmarsh,1996,p529 someexcellentresearchersmakeexcellentteachersandsomedonot.despite critiquesofthemeasuresofresearchandteachingthatarenormallyused,noneof thecriticshavemanagedtodeveloporusealternativemeasuresthatdemonstratea relationshipbetweenresearchandteaching.aminorityofundergraduatestudents havebeenreportedtovaluetheirteachersbeingactiveresearchersprovidedthis doesnotinterferewiththeirstudies(forexample,throughtheirteacherbeingabsent whileundertakingresearch)(lindsayet al.,2002),butthereisnoevidencethatthis improvestheirlearning. 5.3.3. qualityofteaching:judgedbystudents

despitethecommondisparagementofstudentratingsofteachers,theycanbe highlyreliable(inthesensethatstudentsagreewitheachotheraboutwhothe bestteachersare,agreewithteachers'peers,andmakethesamejudgementson differentoccasions)andrelativelyimmunefrombiasesofvariouskinds.students canreadilytellthedifferencebetweenteacherstheylikeandteacherswhothey thinkaregoodteachers,andthecommoncriticismthatstudentfeedbackissimply apopularityparadeislargelyunfounded(seemarsh,1987,forareviewofthevast literatureonthereliabilityandvalidityofstudentfeedbackquestionnaires).These observationsarebasedontheuseofthoroughlydevelopedandtestedfeedback questionnaires.however,intheuKalmostallsuchquestionnairesare`homegrown'andarelikelytobeofdoubtfulreliabilityandopentoallkindsofbiases. differentquestionnairesareusedindifferentinstitutionsandthereisthereforeno basisforcomparisonofthequalityofteachersbetweeninstitutionsorsubjects. ifstudentratingsofteacherswereconsideredausefulindicatorofcomparative qualitythentherewouldneedtobeanagreementforeveryonetousethesame questionnaire,suchasthestudentevaluationofeducationalquality(seeq) (marsh,1982;coffeyandGibbs,2000). severalwell-developed,reliable,studentfeedbackquestionnaireshavereasonable levelsofvalidityinthatscoresonscaleswithinthequestionnaires(involvingadding ratingsfromclustersofsimilarquestions)predictaspectsofstudentlearningprocess

27

ThehiGhereducaTionacademy

(suchastheextentofstudyingfollowingteaching),learningoutcomes(suchasgrades) andotherworthwhileconsequences(suchasthelikelihoodofstudentschoosingto studyfurthercourseswiththesameteacher).Theproportionofvarianceinsuch measuresoftheproductsofgoodteaching,thatisexplainedbystudentratings,varies acrossdifferentquestionnairescalesanddifferentmeasuresofproducts,butitis usuallyhighenoughtotakemeasuresofteachingbasedonstudentratingsseriously (abramiet al.,1990). Thereisanimportantdistinctiontobemadeherebetweenstudentratingsof theextenttowhichteachersengageinactivitiesthatareknowntoimprovelearning (suchasprovidingenoughpromptfeedbackonassignments),whichtendtobereliable andvalid,andglobaljudgementsofwhetherteachingis`good',whichareopentoall kindsofsubjectivevariationintheinterpretationofwhat`good'means.studentsalso changeovertimeintheirsophisticationaslearners,forexampleintheirconception oflearning(säljö,1979)andintheirconceptionofknowledge(Perry,1970).as theychange,sotheirconceptionsofwhat`goodteaching'consistsofevolve(van rossumet al.,1985).whatanunsophisticatedstudentmightconsidertobegood mightconsistoftheteacherprovidingallthecontentinlecturesandthentestingfor memoryofthatcontent,whileamoresophisticatedstudentmightseegoodteaching asinvolvingsupportingindependentlearningandthedevelopmentofapersonal stancetowardsknowledge.whatunsophisticatedstudentswanttheirteacherstodo isoftenbadfortheirlearningandrespondingtotheirglobalratingsuncriticallyisnot thewaytoimprovequality.whenasurveyreportsasingleglobalratingoftheextent towhichstudentthinkalltheteachingoverthreeyearsissimply`good',thesevery differentstudentconceptionsofgoodteachingaremuddledtogetherandtheaverage ratingisthenverydifficulttointerpret.incontrastso-called`lowinference'questions thatrefertospecificteacherbehaviours,suchasthepromptnessoftheirfeedback, aremucheasiertointerpret. 5.4 researchenvironment

aswehaveseenabove(insection5.3.2)thereisnorelationshipbetweenmeasures ofanindividualacademic'sresearchandmeasuresoftheirteaching.however,itcould bearguedthatitisnotindividualresearchers'teachingthatmattershere,butthe researchenvironmentgeneratedbythemajorityofteachersinadepartmentbeing researchactive.Thismightbeconsideredapresagevariable,butasweshallsee,what mattersistheeducationalprocess,notpriorresearchperformance. atthelevelofdepartmentswithinaninstitutionthesituationisthesameas itisatthelevelofindividualteachers.Thebestresearchdepartmentsmayormay notbethebestteachingdepartments:thereisnocorrelationbetweenmeasures ofadepartment'sresearchandmeasuresofitsteaching(ramsdenandmoses, 1992).Therearesuggestionsthattheremayberelationshipsbetweentheextentof

28

dimensionsofqualiT y

researchintensivenessofadepartmentandstudents'senseofbenefit,inawaythat isnotmediatedbytheteaching,butthisisinanycaseonlyforsomestudents(e.g. dunbar-GoddetandTrigwell,2006),andthisrelationshipappearstobeafunction ofthefocusofattentionofthestudentsratherthanaconsequenceofdifferentor betterteaching:ifstudentsareinterestedinscholarshiptheywillbemorelikelyto saythattheybenefitfromascholarlyenvironment. atthelevelofwholeinstitutions,thosewherethereisastrongorientation towardsresearchoftenrevealaweakemphasisonteaching,andviceversa­there isastrongnegativerelationshipinrelationtoinstitutionalprioritiesandthishas measurableeffectsoneducationalgains: ... a college whose faculty is research-orientated increases student dissatisfaction and impacts negatively on most measures of cognitive and affective development. --astin,1993,p363 Thereisevidencethattheresearchenvironmentcanimpactpositivelyon undergraduatestudents,inawaythatcanbemeasured,wheremaximisingthe benefitstoundergraduatesofresearchstrengthshasbeenmadeadeliberate priority.forexample,miTenablesabout80%ofitsundergraduatestoengageina realresearchprojectasaninternorjuniorresearchstudentplacedwitharesearch group,andtheyhavegoodevidenceconcerninghowstudentsbenefit(bergrenet al., 2007).Thenationalsciencefoundationintheushassimilarevidenceacrossmany institutionsthatsuchinitiatives: ... have a major impact on most participants' confidence and their understanding of research-related issues, increase their interest in careers in research and science and engineering, and lead them to raise their degree expectations. --ramaley,2004,citedinJenkins,2004 (seealsobauerandbennett,2003;hathawayet al.,2002.)Thekeypointhere isthatsuchbenefitshavetobedeliberatelyengineered­theydonotaccrueby magicsimplybecauseresearchisgoingonaswellasteaching.Theinstitutional indicatorofqualityinthesestudiesistheexistenceofanundergraduateresearch opportunitiesscheme,notthestrengthoftheinstitution'sresearch.similarly thepositiverelationshipfoundattheuniversityofoxfordbetweenstudents' experienceofresearch-activestaffandtheextenttowhichtheytakeadeep approachtolearning(Trigwell,2005)isaconsequenceofthecollegialsystem fosteringactiveinclusioninacommunityof(research)practice,notsimplyofthe existenceofaresearchenterprise. forthesereasonsdepartmentalraescoresorothermeasuresofresearch activityorperformanceintheenvironmentstudentsstudywithinarenot,ontheir own,validindictorsofeducationalquality.

29

ThehiGhereducaTionacademy

5.5

levelofintellectualchallenge

itcanbearguedthataninstitution,orevenanentirehighereducationsystem,is ofhigherqualitybecauseitoffersagreaterlevelofintellectualchallenge.inthis sectionthenotionofintellectualchallengeisoperationalisedinthreeways:challenge asdeterminedbythelevelofthecurriculumstudentsarestudying,challengeas indicatedbythedepthofapproachstudentstaketotheirstudies,andchallengeas reflectedinthelevelofstudentengagementwiththeirstudies. 5.5.1. levelofthecurriculum

insomesubjectareaswithrelativelyhierarchicalknowledge,formalcourse documentationcanbeveryinformativeaboutthelevelofthecurriculum.itis thenpossibletocomparethelevelofintellectualchallengebyexaminingcourse descriptionstoseehowfarthroughthesubjectstudentsareexpectedtoprogress. itisinprinciplepossibletostudycurriculumdocumentationwithinwell-defined disciplines,fromdifferentperiods,inordertoascertainwhethertherehasbeena wateringdownofintellectualdemandsovertime.Thereareseverallimitationstothis kindofstudy: -- documentationchangesovertimeinwhatisdescribedandinwhatdetail, andalsodiffersbetweeninstitutions; -- fashionschangeovertimewithinsubjectsaboutwhatshouldbeincluded, whichmakeschangesoflevelofdemanddifficulttojudge; -- eveninsupposedlyhierarchicalsubjectsitissometimesdifficulttoagreewhat isnecessarytostudyfirst,orwhatcountsasmoreadvanced.inmedicine, forexample,problem-basedlearningmaynowadaysconfrontfirst-year studentswithproblemsofacomplexitythatinpastcurricula,thathadclear divisionsbetweenpre-clinicalandclinicalstages,onlyfourth-yearstudents encountered.however,thefirst-yearstudentsnowtacklethesecomplex problemsatalowerlevel,withalesserknowledgebase.calibratingthelevel oftheproblemandthelevelatwhichtheyaretackledisnotstraightforward. neverthelessthereisprobablymorescopefordeterminingqualitybyexamining curriculumdocumentationthaniscurrentlyexploited,provideditisundertakenby thosewithindisciplinarycommunitiesthatunderstandthecomplexitiesinvolved. Thereareseveraltypologiesforeducationalobjectivesorlearningoutcomes (forexamplethatofbloomet al.,1956)thatcouldbeusedtocomparethelevel ofdifficultyofwhatstudentsaresupposedtodowiththecontent(forexample, rememberit,applyitorcritiqueit).itispossibletodistinguishlevelsofintellectual demandbetweendifferentlevelsofbTeccoursesinthesamesubjectbycomparing

30

dimensionsofqualiT y

theproportionofeducationalobjectivesspecifiedateachofanumberoflevels(such as`knowledge'and`synthesis').higherlevelsofcoursesaresupposedtohaveahigher proportionoftheircurriculaobjectivesatahigherlevelinthehierarchy.Proposals fornewcoursescanbeturneddowniftoomanyobjectivesaresetattoolowalevel. whereacurriculumauthorityhasthepowertoimposeacommonsetof definitionsandacommonformatofdocumentation,asbTecdoes,thisisjust aboutpossibletoimplement.whereeachinstitutionandeachdegreeprogramme hasthefreedomtospecifylearningoutcomesindependentlyofanycommon conceptualframeworkorsetofdefinitions,suchcomparisonsbecomeimpossible, exceptinanimpressionisticway.forlearningoutcomesthatconcerngenericskills, suchas`communication'and`groupwork',consistentjudgementoflevelsagainst standardsisevenmoredifficult,independentlyofexemplarsthatgobeyondcurricula documentation.alvernocollegeintheushasdemonstratedthatevenforsuch genericoutcomesorcapabilities,itispossibletodefinestandardsinunambiguous waysandtotrainteacherstousethesestandardsinconsistentways,butunless alverno'sapproach,orsomethinglikeit,weretobeadoptedbyall,comparison acrosscontextswouldstillnotbepossible. neverthelessthereisplentyofscopeforsubjectassociationstomakemoreuse ofcurriculumdocumentationinjudgingtheintellectualdemandsofprogrammes, atleastinprinciple.Tojudgetheminpracticeitisalsonecessarytoexaminethe productsofstudentlearning(seesection7.7). 5.5.2. depthofapproachtostudying

inthe1970sferencmartonandhiscolleaguesinGoteborgdistinguishedbetweena `surfaceapproach'tolearninginwhichstudentsintendtoreproducematerial,and a`deepapproach'inwhichstudentsintendtomakesenseofmaterial:adistinction betweenafocusofattentiononthesignorwhatissignified.Toillustratethe consequencesforstudentlearningoutcomes,astudentwhotakesasurfaceapproach toreadinganarticlewithaprinciple-examplestructure(suchasacasestudy)may remembertheexample,whilethestudentwhotakesadeepapproachismorelikely tounderstandtheprinciple(martonandwenestam,1978).asurfaceapproachhas beendemonstratedinawidevarietyofstudiestohavedepressinglylimitedand short-lastingconsequencesevenformemoryoffacts.adeepapproachisessential forlong-termandmeaningfuloutcomesfromhighereducation(seeGibbset al.(1982) andmartonet al.(1984)foroverviewsofthisliterature). studentsarenot`surfacestudents'or`deepstudents'­approachtolearning isinthemainacontext-dependentresponsebythestudenttoperceiveddemands ofthelearningcontext(ramsden,1979).Therelevancetodimensionsofqualityis thatitispossibletoidentifythosefeaturesofcoursesthatfosterasurfaceoradeep approach.studentstendtoadoptasurfaceapproachtoagreaterextentwhenthere

31

ThehiGhereducaTionacademy

is,forexample,anassessmentsystemthatrewardsmemorisation,suchassuperficial multiple-choice-questiontests.incontraststudentstendtoadoptadeepapproach, forexample,whentheyexperiencegoodfeedbackonassignments,andwhenthey haveaclearsenseofthegoalsofthecourseandthestandardsthatareintendedto beachieved.Theseinfluentialcharacteristicsofcoursesarethefocusofthecourse experiencequestionnaire(ceq)(ramsden,1999),originallydevelopedinstudies atlancasteruniversityinthe1970s,throughwhichstudentsindicatetheextentto whichthesecoursefeaturesareexperienced.reasonablycloserelationshipshave beenfoundbetweenscoresonscalesoftheceqandtheextenttowhichstudents takeadeepandsurfaceapproachtotheirstudies,andsoceqscalescoresthat focusoncertaincoursefeaturescanactasaroughproxyforeducationaloutcomes, becauseapproachpredictsoutcomestosomeextent.Theceqbecamethebasisof thequestionnaireusedannuallythroughoutaustralianhighereducationtomeasure comparativequalityofdegreeprogrammes,publishedinannualreportsaimedat students.ithasbeenusedforsomeyearswithinsomeinstitutionsasaperformance indicatorforallocatingaproportionoffundingforteachingtodepartments,asat theuniversityofsydney.ithasnowbeenadoptednationallyinaustraliaasone componentofperformanceindicatorsforallocatingovera$100millionofteaching funding(in2008)touniversitieseachyear.ithasbecomethedrivingforcebehind evidence-basedinstitutionaleffortstoimproveteachingthatfocusoncoursedesign ratherthanonindividualteacher'sskills(barrieandGinns,2007).amodifiedversion oftheceq(theosceq)hasbeenusedannuallyattheuniversityofoxford. itisoftenassumedthatthevalidityofthenationalstudentsurvey(nss)is basedonthesameresearchandevidence.uptoapointthisistrue.however,the characteristicofstudents'intellectualengagementwiththeirstudyingthatbestpredicts theirlearningoutcomes,theextenttowhichtheytakeadeepapproach,isnotincluded asascaleinthenss(andnorisitintheceq).somecharacteristicsofwhathavebeen foundtobeeffectivecourses,suchasconcerningfeedback,areincludedinthenss. however,mostofthescalesoftheoriginalversionoftheceqthatrelatesomewhat totheextenttowhichstudentstakeadeepapproach,suchas`clearGoalsand standards'or`appropriateworkload'arenotincludedinthenss(andneitherarethey inthemostrecentversionsoftheceq).infactbothquestionnaireslackmostofthe scalesthatwouldstrengthentheirvalidity.Themissingscalesarecurrentlyincludedas optionsinbothquestionnaires,butthismeansthatcomparabledataarenotpublished oravailableforcomparisonbetweeninstitutionsorcourses. evensomeofthesemissingscaleshaveasomewhattenuousclaimtovalidity today.forexampleinthe1970sitwasfoundthatifstudentsweregrossly overburdenedthentheymightabandonadeepapproachandadoptasurface approachtotheirstudies.however,30yearslaterexcessiveworkloadseemsa distantmemory(seesection5.2),sothe`appropriateworkload'scalenolonger seemslikelytopredict,toaworthwhileextent,whichstudentswilladoptasurface approach,andhencetheirlearningoutcomes.

32

dimensionsofqualiT y

Therehavebeennorecentstudiestoconfirmtheoriginalfindingsconcerning relationshipsbetweenfeaturesofcourses,studentresponsesandlearningoutcomes incurrentcontexts.Therehavebeennodirectstudiesofthevalidityofthenssin relationtoitsabilitytopredicteducationalgains.Therehavebeennostudiesthat demonstratethatifevidence-basedpracticesareadopted,andnssscoresimprove, thiswillbeassociatedwithimprovededucationalgains.forthatkindofevidencewe havetolooktomeasuresofstudentengagement. 5.5.3. studentengagement

Thereiscurrentlyahighlevelofinterestintheusinmeasuringstudentengagement asacrucialindicatorofeducationalquality,embodiedintheuseofaquestionnaire: thenationalsurveyofstudentengagement(nsse).Theresearchunderlyingthe nsse,anditswide-scaleuseandperceivedvalue,hasinvolvedthreemainstages. first,verylarge-scalestudiesexaminedahugerangeofprocessvariableson campusandalsoassessedahugerangeofmeasuresofeducationalgain,acrossavery widerangeofcoursesandinstitutions,toidentifywhichprocessvariablesrelateto anyofthemeasuresofgain.noresearchencompassingsomanyvariablesatonce, oronasimilarscale,hasbeenconductedoutsideoftheus.Thesevaststudieshave beenrepeatedoverthreedecadesandverysimilarfindingshaveidentifiedthesame fewprocessvariablesthataremostcloselylinkedtoeducationalgains(Pascarella andTerenzini,2005).essentiallythecrucialvariableis`studentengagement'andit hasprovedpossibletoidentifytheprocessvariablesinvolvedinengagingstudents, suchasthelevelofacademicchallenge,theextentofactiveandcollaborativelearning andtheextentandqualityofstudent-facultyinteraction.Theseprocessvariables areprominentintheinfluentialevidence-based`sevenprinciplesofgoodpracticein undergraduateeducation'elaboratedabove(chickeringandGamson,1987,1991). Theseprincipleshavebeenusedbymanyinstitutionsintheusasindicators ofqualityinreviewingandimprovingtheireducationalpractices,andtherehas beenmuchpoolingofinstitutionalexperienceinusingtheseprinciples.Pascarella et al.(2008)listaplethoraofstudiesthatdemonstratethatifyouactonthe`seven principlesofgoodpractice'thenthisimprovesstudentoutcomes. second,aquestionnairehasbeendevelopedthroughwhichstudentscan indicatetheextenttowhichthesecrucialprocessvariableshavebeenexperienced andthewaysinwhichtheyareengagedintheirstudying:thenationalsurveyof studentengagement(nsse,2007).Thisquestionnairehasbeenusedwidely­by 774universitiesandcollegesin2008alone.ithasalsobeenwidelyusedtomonitor studentengagementaschangeshavebeenmadetoeducationalpracticesinattempts toimprovestudentlearning.sufficientstudiesofappropriaterigourhavebeen publishedtomakeitpossibletobringthedatatogetherandmakegeneralisable conclusions.meta-analysisofstudiesofinnovationandchangesupporttheoriginal

33

ThehiGhereducaTionacademy

identificationbytheearlierresearchofapredictablerangeofeducationalprocesses that,whenadoptedorenhanced,makeameasurabledifferencetostudents engagement(carininet al.,2006). Thethirdstageofthisresearchhasinvolvedvaliditystudies­examiningifthere isarelationshipbetweenengagementandeducationaloutcomes:betweenscores onthensseandbothstudentperformanceandpsychometricmeasuressuchasof effectivereasoningandproblemsolving,`inclinationtoinquire'andlifelonglearning. anumberofstudieshaveshownclearlinksbetweennssescoresandoutcomes suchasfirst-yearsuccess(e.g.lanasaet al.,2007).Pascarellaet al.(2010)have gonefurtherandinamuchlargerstudy,involving19collegesofawidevarietyof types,demonstratedstrongrelationshipsbetweennssescalescoresandarangeof educationalgains,involvingbeforeandaftermeasurements.Theauthorsconcluded: NSSE results regarding educational practices and student experiences are good proxy measures for growth in important educational outcomes. inotherwordsifyouwanttoknowthe`valueadded'bystudents'higher educationexperiencethenthenssewillprovideagoodindicationwithoutneeding tousebeforeandaftermeasuresofwhathasbeenlearnt. itisinterestingtonote,withreferencetotheself-imposedlimitationsofthe nssandceq,thatthescaleonthenssethathastheclosestrelationshipwith educationalgainsconcerns`deeplearning'(Pascarellaet al.,2008). 5.6 formativeassessmentandfeedback

Theeducationalinterventioninschoolsthathasmoreimpactonstudentlearning thananyotherinvolvesimprovingformativeassessmentandespeciallytheprovision ofmore,betterandfasterfeedbackonstudentwork(blackandwiliam,1998; hattieandTimperley,2007).`Goodpracticeprovidespromptfeedback'isoneof theevidence-based`sevenprinciplesofgoodpracticeinundergraduateeducation' (seeabove).ondegreeprogrammeswherethevolumeofformativeassessmentis greater,studentstakeadeepapproachtotheirstudiestoagreaterextent(Gibbsand dunbar-Goddet,2007)anddeepapproachisagoodpredictoroflearningoutcomes (seesection5.5.2above).enhancedfeedbackcanalsoimprovestudentretention (yorke,2001). Thenumberofoccasionsduringathree-yearbachelorsprogrammeintheuK onwhichstudentsarerequiredtoundertakeanassignmentpurelyforthepurpose oflearning,withfeedbackbutwithoutmarks,varieswidelybetweeninstitutions. onestudyhasfoundarangefromtwiceinthreeyearsatoneenglishuniversity toover130timesatanother(Gibbsanddunbar-Goddet,2009).inanotheruK studyusingthesameassessmentauditmethodology(TesTa,2010),thevolumeof

34

dimensionsofqualiT y

writtenfeedbackonassignmentsoverthreeyearsvariedfrombelow3,000words perstudenttoabove15,000words,andfororalfeedbackvariedfrom12minutes peryearperstudenttoovertenhoursperyear(Jessopet al.,2010).Thesearemuch widervariationsbetweeninstitutionsthanexistintheirfundingperstudent,their ssrs,theirclasscontacthoursortheirindependentstudyhours.Theissueaddressed bythenssthatrevealsthegreatestareaofstudentdisquietisfeedback. asresourcesperstudenthavedeclinedtherehavebeeneconomiesofscalein teachingthataredifficulttoachieveinassessment:assessmentcostsgoupprettymuch inproportiontothenumberofstudents.Thisplacesenormoustimepressureson teachers.qualityassurancesystemsinmostinstitutionshavenotpreventedthevolume offormativeassessmentfromdecliningsubstantially,despitetheqaa Code of practice. anexceptionisTheopenuniversitywherethenumberofassignmentspermodule, andthevolumeandqualityoftutorfeedbackonallassignmentshavebeenmaintained over30years.Thishasbeenachievedacrossallcoursesbyformalrequirementsof theircourseapprovalprocessandbyseveralqualityassuranceprocesses.Theopen universityhasexceptionallyhighnssscoresforassessmentandfeedback. 5.7 5.7.1. otherprocessdimensionsofquality reputation

seekingtheviewsofresearchpeersisacommonmethodusedtojudgeadepartment oruniversity'sresearchqualityandthesamemethodologycouldinprinciplebe usedtojudgeeducationalquality.Thehighlyinfluentialuniversityrankingsystem intheusprovidedbytheusnewsandworldreport,`america'sbestcolleges', investsheavilyinsurveysofdeansandPresidentsinestablishingcollegereputations. however,thereputationalrankingthatderivesfromthesesurveyscorrelatesclosely withthesizeofinstitution'sfederalresearchgrants(GrahamandThompson,2001) andcanalsobepredictedbyundergraduateselectivity,perstudentexpenditure andnumberofdoctoralawardingdepartments(astin,1985),noneofwhichpredict educationalgains.reputationaldatahaveaverypoorreputationasavalidindicator ofeducationalquality. 5.7.2. P eerratings(wherethesefocusonprocessdimensions)

manyqualityassurancesystemsmakeuseofexpertpeerjudgementofthequality ofeducationalprovisioninadegreeprogramme,atthetimeofaperiodicreviewof somekind,basedonawiderangeofevidenceanddocumentationandsometimes includingobservationofteaching.Therelationshipbetweentheseratingsandthe evidenceonwhichtheyarebasedisnoteasytoestablishastheyareinherently

35

ThehiGhereducaTionacademy

subjectiveandglobal,andbasedondifferentcombinationsofevidence,withdifferent weightings,indifferentcontexts,bydifferentgroupsofpeers.however,theremaybe potentialfortheapplicationofprofessionalexpertiseinsuchsubjectivejudgements toreachmorevalidconclusionsthancouldbeachievedmerelyonthebasisof individualquantitativemeasures.ThisiswhatTeachingqualityassessment(Tqa) ratingsattemptedtoprovideinaquantitativeway.inleaguetablesinenglandthe sixfour-pointratingscalesinvolvedinTqahaveusuallybeencombinedintoasingle scoreoutof24,andinstitutionalaveragesoutof24havebeenusedasindicators ofeducationalquality.subsequentanalysisofaverageTqascoresforinstitutions hasrevealedthattheyareverylargelypredictableonthebasisofstudententry standards(a-levelpointsscores)andresearchperformance(raescores),together orseparately,withoutreferencetoanymeasuresofeducationalprocess(drennan andbeck,2001;yorke,1997,1998).inotherwords,Tqascoreslargelyreflect reputationalfactors.Thiswouldnotbeaterminalproblemifresearchperformance andqualityofstudentswerevalidindicatorsofeducationalqualitybut,aswehave seenabove,theyarenot.Theinabilityofreputationalfactorstoprovideavalid indicatorofeducationalqualityishighlightedaboveinsection5.7.1.Theinabilityof peerjudgementstobeimmunefromreputationalfactorsunderminestheircredibility. Tqascoreswerealsosubjecttootherconfoundingvariables,suchasinstitutional size,whichhavenotbeentakenintoaccounteitherinmoderatingoverallscores,or inleaguetablesbasedonTqascores(cooket al.,2006). 5.7.3. studentsupport

Theqaahavehighlighted,intheirreviewsofwhathasbeenlearntfrominstitutional audits(qaa,2003),theimportantroleplayedbyadequatestudentsupportservices ofvariouskinds:studyskillsdevelopment,counselling,englishlanguagesupport, supportforstudentswithspecialneeds,andsoon.Thereanumberofreasons whyitisdifficulttoestimatetheextenttowhichstudentservicesplayarolein educationaleffectivenessorgain.supportservicesareconfiguredinmanydifferent ways,forexamplesubsumedwithinacademicrolesorcentralisedingenericservice units.Theyaredescribedusingdifferentterminology:forexample,therearefewus equivalentsoftheuK'straditionalpersonaltutorrole,andfewuKequivalentsofthe roleof`studentadvising'intheus.Thismakescollatingevidenceacrosscontexts,or comparinglikewithlike,somewhatchallenging.dataconcerningthepositiveimpact ofstudentsupportfromlargeusstudiesaredifficulttorelatetothenatureofuK provision.Theimpactofsuchservicesalsorelatescloselytothenatureofstudent intake.slenderprovisionatoneinstitutionmightbeperfectlyadequatebecauseit onlyhasazephyrofdemandtodealwith,whileatanotherinstitutionevenextensive andprofessionallyrunsupportservicesmayfaceagaleofdemandandexpectations andsomayfallshortdespiteextensiveinstitutionalcommitment.Thereisclear

36

dimensionsofqualiT y

evidenceoftheroleofvariouskindsofstudentsupport,forexampleconcerningthe impactonstudentperformanceofthedevelopmentofstudents'studyskills(hattieet al.,1996).however,whatsupportservicesareappropriate,andhowtheymightbest bedelivered,canbehighlycontext-anddiscipline-specific.forthisreasonnogeneral empiricalconclusionswillbedrawnhere. 5.7.4. qualityenhancementprocesses

muchofthepastfocusofattentionofthecouncilfornationalacademicawards, andtodaythequalityassuranceagency,hasbeenonqualityprocesses,suchas theoperationoftheexternalexaminersystemandtheuseofstudentevaluation ofteaching,thatareintendedtoassurequality.Theassumptionisthatifsuch processesaresecurelyinplace,thenanadequatelevelofqualitycanbemoreor lessguaranteed.Thereissomeevidencetosupportthiskindofassumption.aswas discussedinsection4.1above,ininstitutionswherestudentengagementisfoundto behighandeducationalgainsarehigh,onefindsahigherthanaverageinvestmentof resourcesinqualityenhancementprocessessuchasfacultydevelopmentandteaching andlearningcentres(Gansemer-Topfet al.,2004).Thereisalsoevidencethatsome oftheprescribedqualityenhancementprocesseshaveapositivemeasurableimpact, butonlyundercertaincircumstances.forexample,collectingstudentfeedbackon teachinghaslittleornoimpactonimprovingteaching(weimerandlenze,1997) unlessitisaccompaniedbyotherprocesssuchastheteacherconsultingwithan educationalexpert,especiallywhenprecededbytheexpertobservingteachingand meetingstudents(Piccininet al.,1999). Theextentofinstitutionaladoptionofqualityenhancementprocessesthrough teachingandlearningstrategieshasbeendocumentedforenglishinstitutions, (hefce,2001;Gibbset al.,2000),butthereiscurrentlynoevidencethattheextent ofadoptionoftheseprocessesrelatestoanyothermeasuresofprocessorproduct.

37

ThehiGhereducaTionacademy

6.

P roducTdimensionsofeducaTionalqualiTy

6.1

s tudentperformanceanddegreeclassifications

intheuKthemeasuremostcommonlyusedtoindicatethequalityoftheoutcome ofhighereducationistheproportionofstudentsgaininguppersecondclassorfirst classdegrees.Theproportionofstudentswhogain`gooddegrees'hasincreased verymarkedlyovertime,althoughunevenlyacrossinstitutionsandsubjects(yorke, 2009).atthesametimepresageandprocessindicatorsofquality(suchasfunding perstudent,thequalityofstudentintake,classsize,ssrs,amountofclosecontact withteachersandamountoffeedbackonassignments)havedeclined.yorke(2009) suggestsawholelistofreasonswhythiscounter-intuitivephenomenonhasoccurred. forexample,theproportionofassessmentmarksderivedfromcourseworkhas increasedandcourseworkusuallyproduceshighermarksthanexaminations(Gibbs andlucas,1997).mostofthepossibleexplanationscurrentlylackdatathroughwhich theycouldbetested. Thekeyproblemappearstobethattherehasbeenlittletostopgradeinflation. Theexternalexaminersystemhasnotprovedcapableofmaintainingthestandards thatareappliedbymarkerstowhateverqualityofstudentworkisbeingassessed. asaconsequencedegreeclassificationscannotbetrustedasindicatorsofthe qualityofoutcomes.awholeraftofunjustifiablevariationsexistsinthewaystudent degreeclassificationsaregenerated.forexample,amathsstudentsismorethan threetimesaslikelytogainafirstclassdegreethanahistorystudent(yorkeet al.,2002;bridgeset al.,2002)andthereareidiosyncraticinstitutionalalgorithms foraddingmarksfromdifferentcourses(yorkeet al.,2008)thatcanmakeasmuch asadegreeclassificationdifferencetoindividualstudents(armstronget al.,1998). Thebestpredictorofthepatternofdegreeclassificationsofaninstitutionisthat theyhaveproducedthesamepatterninthepast(Johnes,1992),andinstitutions' historicalpatternsarenoteasilyexplicable. ithasbeenarguedthatthereisnolongeranymeaningfulsenseinwhichdegree standardsarecomparable(brown,2010).Therehasbeenpersistentcriticismofthe meaningandinterpretabilityofdegreeclassificationsasindicatorsofeducational outcomes(e.g.houseofcommons,2009)andtheseargumentshavebeenlargely accepted,e.g.bytheqaa(2006),andsotheargumentswillnotberehearsedhere. whatisclearisthatdegreeclassificationsdonotcurrentlyprovideasoundbasisfor indicatingthequalityofeducationaloutcomesofauKinstitution.

38

dimensionsofqualiT y

6.2

studentretentionandpersistence

Theopenuniversityandtheuniversityofoxfordhavecomparablenssstudent ratingsfortheperceivedqualityoftheireducationalprovision,butareatopposite endsofrankingsintermsofstudentretention,withabout98%ofentering undergraduatescompletinginthreeyearsatoxford,almostdoubletheproportion ofnewstudentscompletingaten-monthcourseatTheopenuniversity.student retention(inrelationtopersistingfromoneyeartothenextandcompletionrates withinnormaltimeframes)varyveryconsiderablyfromoneinstitutiontoanother evenwheneducationalprovisionisjudgedtobesimilarlyexcellentorsimilarlypoor. institutionalcomparisonsaremadedifficultbythevariednatureofstudentcohorts. broadly,nationalretentionratesvaryininverserelationtoageparticipationrates (oecd,2000):thebroadertherangeofstudentabilityisenteringhighereducation, theloweristheoverallretentionrate.inaddition,differentinstitutionstaketheir studentsfromdifferentsubsetsoftheoverallabilityrange. studentsvarynotjustintermsoftheirrecordofpasteducationalsuccess,but inothervariablesknowntoaffectretentionsuchaswhethertheyliveoncampus (chickering,1974)andwhethertheyareundertakingpaidworktosupporttheir studies(Paton-saltzbergandlindsay,1993). intheusitisnolongerthecasethatthemajorityofstudentsgainthecredits theyneedforaqualificationfromasingleinstitution.so`drop-out'isnotonlythe normbutis,formanystudents,expectedandevenplannedforastheyaccumulate creditswhereverandwheneverisconvenient.ThisisnotyetthenormintheuK, but`drop-out'doesnothavethesamemeaningorsignificanceforanincreasing proportionofstudentsasitdoesforpolicymakers(woodley,2004).itisnotsimply thatpart-timestudentscompleteatdifferentratesthandofull-timestudents,but that`retention'hasadifferentsignificanceforthem. avariableknowntoinfluenceretentioniswhetherstudentsaresocially andacademicallywellintegrated(Tinto,1975).socialandacademicintegrationis affectedbylivingoffcampus,livingathome,andtakingtimeouttoearnenough tocontinuestudying.Theprevalenceofthesevariablesisveryvariedacross institutions,anditisdifficulttotakeallsuchvariablesfullyintoaccountinjudging institutionalretentionperformance. studentvariablesalsoaffectretentionwithininstitutionsandaresoinfluentialthat intheuscommercialcompanies(suchasthenoel-levitzorganisation)offerservices toinstitutionstocollectmanagementinformationandotherstudentdataconcerning theireducationalqualifications,preparednessandattitudes,inordertopredictwhich studentsaremostlikelytodropoutsothatscarceadditionalsupportcanbedirected atthestudentsmostlikelytobenefit.amathematical,data-drivenapproachofthiskind atTheopenuniversityhasidentifiedverywidedifferencesbetweenenteringstudents inrelationtotheprobabilityofthemcompletingasinglecourse.Thispredictionhas beenusedtodecidewhichstudentstocontactandsupport,withmeasurablepositive

39

ThehiGhereducaTionacademy

consequencesforoverallretention(simpson,2003;Gibbset al.,2006).Thetypesof studentvariablesthatpredictdrop-outgowellbeyondthekindofdatathathefce haveavailabletocalculateinstitutionalbenchmarksforretention.soeventheextent towhichinstitutionsexceedorfallshortoftheirretentionbenchmarkscanonlybea crudeandincompletemeasureoftheireducationalquality. notalloftheinstitutionalvariationinretentionisduetostudentvariables. effortstoimproveretentionhavebeenevaluatedfor30yearsintheus,andwhile overallretentionrateshaveremainedlargelystatic,thishidessubstantialprogress inimprovingretentioninsomeinstitutions.agooddealisnowknownaboutwhat kindsofinstitutionaleffortsarelikelytoimproveretentionandpersistenceinthe us(barefoot,2004)and,withamuchlesserevidencebase,intheuK(yorke, 1999).makinggooduseofthoroughinformationaboutstudentssoastotarget timelyindividualisedsupportandinterventionisoneofthemosteffectivepractices. othereffectivepracticescloselyresemblethoseidentifiedasimprovingstudent performanceandeducationaloutcomesingeneral(lanasaet al.,2007),discussed insection5.5.3above.inparticular,collaborativeandinteractivelearningandclose contactwithteachersincreasessocialandacademicintegration.aspointedout above,suchinterventionshaveagreaterimpactonlessablestudents. ifvariationsbetweenstudents,andespeciallypsychologicalvariablessuchas motivationandcommitment,andsocialvariables,suchaswherestudentsliveand howmuchtimetheyhaveavailabletostudy,couldbefullytakenintoaccount,then retentionperformancecouldbeusedasanindicatorofeducationalquality.however, withthedatacurrentlyavailablethisisnotyetpracticable. 6.3 employabilityandgraduatedestinations

Theextenttowhichgraduatingstudentsareabletoobtainemploymentreasonably quickly,ingraduatejobs,infieldsrelevanttotheirdegreesubject,andwithasalary thatjustifiestheirinvestmentoftimeandmoneyintheirhighereducation,isa commonlyuseddimensionofquality.Thedifficultywithemployabilitydata,aswith retentiondata,istheirinterpretation.differentmethodsofcollectingdata,andin particularthetimingofthedatacollection,makesaconsiderabledifference,and theprocessusuallyreliesonsurveysinvolvingstudentself-reporting.however employabilitydataarecollected,interpretingdifferencesbetweeninstitutionsis problematicforawidevarietyofreasons(smithet al.,2000): -- itisaffectedbydegreeclassification,andstudentperformanceisclosely linkedtostudents'prioreducationalqualifications(whichalsoaffects employability),whichvariesgreatlybetweeninstitutions(seesection4.4 above).inaddition,degreeclassificationsvaryintheirmeaningsacross institutions(seesection6.1above).

40

dimensionsofqualiT y

-- itisgreatlyaffectedbyinstitutionalreputation,whichisaverypoor indicatorofeducationalquality(seesection5.7.1above).Thereislittle evidencethatemployershaveanaccurateandup-to-datepictureofthe educationalqualityoftheinstitutionstheyemploygraduatesfrom,especially iftheyderivethispicturefromcurrentlyinvalidleaguetables. -- itisaffectedbyregionandlocality,duetovariationsinlocalemployment marketsandtheproportionofstudentswholiveathomeandwhoareless likelytomoveawayeithertostudyortofindemployment. -- itchangesovertime,duetochangesintheemploymentmarket,and probablydifferentiallybetweeninstitutions,withoutanychangesinthe effectivenessofhighereducationinstitutions. -- itisaffectedbystudents'socialclass,andthemixofsocialclassvaries betweeninstitutions. -- itisaffectedbystudents'age,andageprofilesvarybetweeninstitutions. -- itisaffectedbystudentaffluence,withsomestudentsneedingtotakeany employmentquicklywhileotherscanaffordtowaitforgraduate-leveland subject-relevantemployment,ortocontinuetheireducation. -- itisaffectedbysubjectmix,withhigherlevelsofgraduateunemployment, non-graduate-levelemployment,andemploymentinfieldsunrelatedtothe degreesubject,moreofteninsomesubjectsthaninothers. interpretingthedifferencesthatexistbetweenemployabilityindicatorsinthe uKandmainlandeuropeisalsodifficultbecausethehighereducationsystems aredifferentincrucialways.forexample,abouthalfofuKgraduatesfromnonvocationalsubjectsfeltthattheirfirstjobswerenotappropriatetoadegree-level educationcomparedwithaboutaquarterofsuchgraduatesfromothereuropean countries.forvocationalsciencegraduatestheproportionwasonly17%and 10%respectively,butstillhigherfortheuKthanfortherestofeurope(brennan andTang,2008a,2008b).Theoveralldifference,acrossallsubjects,isprobably becauseeuropeangraduatestendtobeolder,muchmorelikelytohavemasterslevelqualificationsbythetimetheyenterthejobmarket,andmorelikelytohave experiencedwork-basededucationrelevanttotheirsubjectduringtheirmore extendededucation.ThereisalsoadifferentsubjectmixbetweentheuKand europewithmorenon-vocationalartsgraduatesintheuK(withunemployment ratestwicethatofsomeothersubjects).Thisemployabilitydifferencebetweenthe uKandeuropedisappearsaboutfiveyearsaftergraduation,atwhichpoint96%of uKgraduatesareinemployment(ibid.),whichsupportstheinterpretationthatthere areinfluentialdifferencesbetweentheuKandmainlandeuroperegardinggraduates' ageandvocationalexperienceatthetimeofgraduation. Thereareusexamplesofresearchthatvalidateinstitutionalemployability missionsbyidentifyinglong-termconsequencesforthewaygraduatesdemonstrate, intheirprofessionallives,abilitieslearntatcollege(e.g.mentkowskianddoherty,

41

ThehiGhereducaTionacademy

1984).ThereareonlyverymodeststudiesofthiskindintheuK(e.g.Jenkinset al., 2001),andcertainlynotenoughtomakeinstitutionalcomparisonsoreventovalidate institutionalclaimsabouttheefficacyoftheiremployabilitymissions. Thehighereducationstatisticsagencyisabletotakeintoaccountsome variables(subjectofstudy,qualificationsonentryandageonentry)insetting institutionalperformancebenchmarksforemployability,butnotothers.smithet al.(2000)havemadeamoremathematicallysophisticatedattempttotakemore variablesintoaccount,butstillleaveoutcrucialvariablesaboutwhichdataarenot easytoobtain.interpretinganinstitution'sgraduateemploymentperformancein relationtohefcebenchmarksisfraughtwithproblems. finally,theloosefitthatcharacterisestheuK'shighereducationanditsjobs markethasbeeninterpretedbysomecommentatorsnotasaproblem,butas providingflexibilityforgraduatestocopewithafluidemploymentmarketthat isconstantlychanginginrelationtothecapabilitiesthatarerequired.Thisissue concernsthedifferencebetweenexpertiseforefficiency,whichiswhatemployers recruitinggraduatesnormallydemand,andadaptableexpertise,thatenablesan individualtooperateeffectivelyinunpredictablenewsituations(schwartzet al., 2005).ittakesverydifferentkindsofeducationalprocesstodevelopthesetwo formsofexpertise.Thereisalackofevidenceaboutthelong-termconsequencesfor graduateemploymentofeithernarrowlyfocusedvocationaleducationoreducation thatemphasisesefficiencyingeneric`employabilityskills',ratherthanemphasising thehigherorderintellectualcapabilitiesinvolvedinadaptableexpertise.Thismakes relyingonhesa'sveryshort-termemploymentdataariskythingtodo.

42

dimensionsofqualiT y

7.

summaryandconclusions

7.1

Theimportanceofprocessvariables

muchofthisreportdemonstrateswhatcommentatorsintheushavebeenarguing formanyyears.Presagevariablessuchasfunding,researchperformanceandthe reputationthatenablesaninstitutiontohavehighlyselectiveentry,donotexplain muchofthevariationbetweeninstitutionsinrelationtoeducationalgains.measures ofeducationalproductsuchasgradesandcareerearningsreflectthesepresage variables,becausethebeststudentscompetetoenterthebestfundedandmost prestigiousinstitutionsandthequalityofstudentsisthebestpredictorofproducts. measuresofproductsuchasretentionandemployabilityarestronglyinfluencedbya raftofvariablesthatmakeinterpretinganinstitution'sperformanceextremelydifficult. Themostimportantconclusionofthisreportisthatwhatbestpredicts educationalgainismeasuresofeducationalprocess:whatinstitutionsdowiththeir resourcestomakethemostofwhateverstudentstheyhave.Theprocessvariables thatbestpredictgainsarenottodowiththefacilitiesthemselves,ortodowith studentsatisfactionwiththesefacilities,butconcernasmallrangeoffairlywellunderstoodpedagogicalpracticesthatengenderstudentengagement. intheuKwehavefewdataabouttheprevalenceoftheseeducationalpractices becausetheyarenotsystematicallydocumentedthroughqualityassurance systemsandnorarethey(inthemain)thefocusofthenss.Thebestmeasureof engagement,thensse,isusedonlytoaverylimitedextentintheuK. 7.2 Theimportanceofmultivariateanalysis

muchoftheuKdataaboutrelationshipsbetweenpresageandprocessvariables,or betweeneitherpresageorprocessvariablesandproductvariables,looksatonepair ofvariablesatatime­forexample,therelationshipbetweenameasureofresearch performance(e.g.therae)andameasureofteachingquality(e.g.Tqascores).such relationshipsareinvariablyconfoundedwithrelatedvariables,forexamplewiththe qualityofstudentsattractedtothehigh-statusinstitutionsthathavehighresearch performance.asaconsequencefewrelationshipsbetweentwovariablescanbe interpretedwithconfidence.ThefewuKstudiesthathaveexaminedanumberof variablesatatimeusingsomeformofmultivariateanalysis(e.g.drennanandbeck,

43

ThehiGhereducaTionacademy

2001;yorke,1998)haveconfirmedthatapparentlystrongrelationshipsbetweenpairsof variables(e.g.betweenameasureofresearchandameasureofteaching)areconfounded byothervariablesthatcouldequallyberesponsibleforapparentrelationships(e.g.a measureofqualityofstudentintake).intheustherehavebeenfarmore,largerand morecomplex,multivariateanalysesthattakeintoaccountawholeraftofvariables atthesametimeandwhich,asaconsequence,areabletoteaseoutthosevariables thatareconfoundedwithothersandthosethatarenot.wearethereforelargely dependentonusdataandanalysesforourunderstandingofthecomplexrelationships betweendimensionsofquality.someofthenecessarydatathatwouldallowamore comprehensivemultivariateanalysisintheuKhavealreadybeencollectedandcollated (forexamplebyhefce,hesa,thenssandbyhePi),butitcurrentlyresidesin differentdata-bases.itwouldbehelpfultocombinethesedatabasessoastoallow multivariateanalysis,andtoaligndatacollectionmethodstomakethiseasiertodo. 7.3 Theimportanceofeducationalgain

whilesomeuKdataincludemeasuresofeducationalproduct,thereareveryfewuK studiesthathaveincludedmeasuresofeducationalgain.Thismattersbecausethebest predictorofproductisthequalityofstudentsenteringtheinstitution,andthequality ofstudentsvariesgreatlybetweeninstitutions,sothatifyouonlyhaveameasureof product,suchasdegreeclassifications,ratherthanofgains,thenyoucannoteasily interpretdifferencesbetweeninstitutions.whenuKstudiesdoattempttomeasure gaintheyinvolvedifferentmeasuresonentrythanonleavinghighereducation(for examplea-levelpointscoresanddegreeclassifications,respectively).furthermore themostcommonmeasureofproduct,degreeclassification,variesinitsmeaningand standardacrosssubjectsandacrossinstitutions(yorke,2009).itisthereforedifficultto interpreteventhesecomparativemeasuresofgain.studiesintheusincontrastarefar morelikelytousepsychometricmeasuresofgenericeducationaloutcomes(suchasa testofcriticalthinking)withthesamemeasure,andwiththesamestandardsbeingused acrossdifferentsubjectsandinstitutions,andalsousingthesamemeasurebothbefore andafterexperiencingthreeorfouryearsofcollege.inthiswayareliablemeasureof educationalgain,andcomparisonbetweeninstitutionsinrelationtoeducationalgain, ispossible.againweareheavilydependentonusstudiesforevidenceofwhichquality dimensionspredicteducationalgain,andespeciallyonthevaststudies,andreviewsof evidence,undertakenbyastin(1977,1993)andPascarellaandTerenzini(1991,2005). 7.4 d imensionsofqualityindifferentkindsofinstitutions

relyingonusdatamightnotmatterifinstitutionsandeducationalprocesseswere essentiallythesameoneithersideoftheatlantic.however,itseemslikelythat

44

dimensionsofqualiT y

thedimensionsthatdefinequalityinavalidwayaredifferentindifferentkindsof institutions.forexample,evenwithintheusthequalityindicatorsthatappearvalid forlarge,national,researchuniversities(inthesensethattheypredicteducational performancetolerablywell)donotworkaswell,oratall,inregionalschoolsand non-selectivecolleges(schmitz,1993).similarlythenormallackofarelationship betweenanemphasisonresearchandanemphasisonteachingdoesnotseem toapplytoasmallgroupofwell-endowedliberalartscollegesthatemphasise closecontactbetweenteachersandstudents(astin,1993).differentpedagogical phenomenaarelikelytobesalientindifferentcontexts,withsomewhatdifferent patternsofrelationshipsbetweenprocessandproduct,dependentoncontext. itisnotjustthatdifferenteducationalprocessesmighthavemoreinfluenceon educationalgainsinsometypesofinstitutionthaninothers.measuresofeducational gainthemselvesmightalsoneedtobedifferentbetweeninstitutionsiftheyareto havemeaning.institutionalmissionsvary,particularlywithregardtotherelative importanceofemployabilityandsubjectknowledge.itwouldbesurprisingifthe samemeasuresofeducationalgainwereequallyappropriateinalluKinstitutions. forexample,Theopenuniversity'smission,emphasisingopenness,meansthatit wouldnotseektoincreasestudentretentionandperformancethroughincreasing selectivitybecausethatwouldreduceitsopenness.itsownindicatorsofquality aredistinctive,andaredifferentevenfromthoseusedbyhefceindeterminingits funding.Theproblemhereisthatfundingmechanismsaredrivenbyindicatorsof qualitythatcutacrossinstitutions'missions. usresearchhasdonewelltoidentifyanyconsistentpatternsatallacrossvaried contexts.however,thelimitsofwhatispossibletoconclude,onaverage,havebeen highlightedbythoseconductingtheresearch(Pascarella,2001).Thesamecaution shouldaccompanyextrapolationoffindingsaboutkeyindicatorsofqualityfrom varieduscontextstovarieduKcontexts. 7. 5 d imensionsofqualityindif ferentdepar tments

muchoftheliteraturecitedabove,andmostofthedebate,hasfocusedon institutionaldifferencesinquality.however,itisclearthatdepartmentscandiffer hugelywithinthesameinstitution.regardingnssscores,thereareinstitutions thathavethehighest-rateddepartmentinenglandinonesubjectandthelowest ratedinanothersubject,despitesharingthesameinstitutionalqualityindicators. educationalleadershipofdepartmentsmakesadifference,creatingculturesthatvalue teaching,thatengageinaconstantprocessofimprovingteaching,andthatcreate richandengaginglearningenvironments,tosomeextentwhatevertheinstitutional environmentandpresagevariables(ramsden,1998;Gibbset al.,2008b). interestinglythetwoinstitutionsfrequentlyreferredtointhisreport,and thatappearatthetopofthenssranking,theuniversityofoxfordandThe

45

ThehiGhereducaTionacademy

openuniversity,have`institutionalpedagogies':commonpatternsofteachingand assessmentacrossalldepartments.alltheirdepartmentsscoresimilarlyhighlyon thenss.Thismaybeinpartduetodeliberatepoliciestoprotecttheinstitutional pedagogy(whichisknowntoworkwell)fromoutsidepressures,withtheeffectof limitingdepartmentalvariationinteachingquality.Princetonuniversitysimilarlyhas anapproachtoqualityassurancethroughitscourseapprovalprocessthatisintended tomaintain`thePrincetonmodel'ratherthanencouragediverseinnovation. muchoftheemphasisintheuKonrecognisingandrewardingqualityinteaching hasfocusedonindividualteachers(e.g.throughthenationalTeachingfellowship scheme)orinstitutions(throughleaguetablesofonekindoranother).Thereare exampleselsewhereofnationalandinstitutionalschemestoidentifyeducationalquality (e.g.infinlandandattheuniversityofoslo,respectively)thatfocusondepartments, or`learningenvironments'atthelevelofadegreeprogramme(Gibbs,2008). 7.6 dimensionsofqualityindifferentsubjects

ThecarnegiefoundationfortheadvancementofTeachingestablishedalarge-scale initiativeontheassumptionthatthepedagogiesofdisciplinesaredifferent:that,for example,educationalqualityisachievedthroughdifferentstrategiesandpracticesin thecreativeartsthaninthesciences,anddifferentlyinenglishthaninlaw(huber andmorreale,2002).atasufficientlevelofabstractiontherearesimilarunderpinning educationalprinciplesacrossalldisciplines(Gibbs,1999),buttheyareembodiedin suchvariededucationalpractices,andaresalienttosuchdifferentextentsindifferent disciplines,thatdisciplinesineffectachieveeducationalqualityindifferentways.if youthenattempttomeasurequalityacrossdisciplines,forexamplebyusingthe courseexperiencequestionnaire,youfindthatsomedisciplinesemergeconsistently betterthanothers,acrossdifferentstudiesanddifferentinstitutions.eitherone hastoacceptthatcertainsubjectsarealwaystaughtlesswellthanothers,which seemshighlyunlikely,orthatdifferentmeasuresofqualityarebetteralignedwiththe consequencesofsome(disciplinary)pedagogicpracticesthanwithothers.Product measuresareparticularlysusceptibletodisciplinaryvariation.Thedistributionof degreeclassificationsismarkedlydifferentbetweensubjectsintheuK(yorkeet al., 2002;bridgeset al.,2002).employabilitymeansdifferentthings,andcomesaboutin differentways,followingthestudyofdifferentdisciplines.comparingqualitybetween disciplinesisfraughtwithdifficulties. itseemslikelythatmanystudentschoosesubjectsfirstandinstitutionssecond, nottheotherwayround.currentinstitutionalqualitydataandrankingsarethe wrongplacetostartifinformingpotentialstudentsisthepriority.furthermore thedefinitionofa`subject'intheinformationavailabletostudentsoftendoesnot correspondverycloselywiththeprogrammeinwhichstudentsmaybeinterested. brownet al.(2009)giveanexampleofa`subjectcategory'usedforreporting

46

dimensionsofqualiT y

nssratingsthatinoneinstitutionisanaveragedrawnfrom11differentdegree programmes.studentsneedgooddataaboutprogrammesmorethantheydoabout institutionsorevenaboutbroad`subjects',andthensscurrentlydoesnotprovide that,fortechnicalreasonsthatwillbedifficulttoovercome.Politicaldemandsfor `betterinformationforcustomers'cannotbemetwithcurrentdatagatheringand analysismethodspartlybecausetheyaggregatedataintoocoarseaway.once dataareaggregatedinafineenoughwaytobeuseful,therearethenboundtobe problemswithsamplesizes.Thisproblemmaybeintractableandisoneofanumber ofsimilarproblemsthatmakeitdifficulttoprovideinformationaboutqualityin accessibleandusableformsevenwhenithasbeencollated(brown,2007). Thequalityofindividualcoursesormodulesalsovarieswithindegree programmes,andtheextentofthisvariationmayberelatedtodegreecoherence. Thisreporthasfocusedoninstitutionsanddegreeprogrammesratherthanon variablesthatprimarilyaffectindividualcourses. 7.7 d imensionsofqualitythataredifficulttoquantify

mostofthisreporthasfocusedondimensionsofqualitythatarefairlyreadily operationalisableinawaythatenablesthemtobemeasuredquantitatively,sothat statisticalrelationshipscanbeestablishedwithotherdimensionsthataresimilarly easytomeasure.Thereareotherdimensionsofqualitythatareimportant,atleast insomecontexts,butthataredifficultorimpossibletoquantify.forexample, throughoutliteratureinvolvingcasestudiesofexcellentteachingatdepartmentlevel therearereferencestoaspectsofdepartmentalculture:whetherteachingisvalued andrewarded,whetherteachersregularlytalktoeachotheraboutteachingandits improvement,whetherinnovationinteachingissystematicallysupportedandfunded, whethereducationaleffectivenessisthesubjectofseriousscholarlyevaluation,and soon(hannanandsilver,2000).qualitiesofdepartmentalleadershipofteaching makeaconsiderabledifference(ramsden,1998;Gibbset al.,2008a),andsomeefforts havebeenmadetomeasureteachers'perceptionsbothofdepartmentalleadershipof teachingandoftheteachingenvironmentthatframesthekindofteachingandlearning thatislikelytotakeplace(e.g.ProsserandTrigwell,1997;martinet al.,2003). sometimeshighlyeffectiveeducationalsystemsaredrivenalmostentirelyby values,suchas`likingyoungpeople',virtuallyindependentlyofthepedagogicpractices employedortheresourcesavailable.inaninternationalstudyofdepartmentsthatwere identifiedbytheirinstitutionasofexceptionallyhighqualityinrelationtoteaching, studentsinoneofthedepartmentssaidthattheirteacherswerenotespecially goodbut that it didn't matter becausetheyfeltincludedinanexcitingcommunityof scholars(Gibbset al.,2008a).studiesatoxfordbrookesuniversityconcerningwhy somesubjectsregularlyproducedbetterstudentperformancethanothersfoundno differencesinanyquantitativemeasureofpresagevariables.however,aqualitative

47

ThehiGhereducaTionacademy

follow-upstudyfoundthatthehighperformingsubjectswerecharacterisedbyhealthy `communitiesofpractice'involvingmuchdiscussionofhowtosolveteachingproblems soastomaketheentireprogrammeworkwellforstudents.incontrast,subjects withconsistentlylowaveragemarkswerecharacterisedbyacorrespondinglackof talkingaboutteaching,andafragmentedfocusonindividualcourses(havnes,2008). itmaybedifficultorimpossibletomeasuresuchinfluentialvariablesinwaysthat allowsafecomparisonbetweencontexts,althoughitmaybepossibletomeasure theirconsequences,forexampleinrelationtostudentengagement. 7.8 evidenceoftheproductsoflearning

amongthemosttellingofallindicatorsofthequalityofeducationaloutcomesmust bestudents'final-yeardissertationsandprojectreports.itisadistinctivefeature ofuKhighereducation(andinthepastarequirementofthecnaaforhonours degreeclassification)thatstudentsundertakeaverysubstantialpieceofindependent studyintheirfinalyear.evenatusivyleagueinstitutionsundergraduatestudents wouldusuallyneedtotakeafourth,honours,yeartotacklesuchachallengingpiece ofwork.itisoftenaculminationandintegrationofalltheyhavelearnt,especially inappliedandcreativefieldsofstudy.Thereisanalmosttotallackofevidence concerningtherelativequalityofsuchproductsacrossinstitutions,withinsubjects. anattempt,forthisreport,toobtainsuchevidencefromsubjectcentreselicited notasingleexample,andthefewpublishedstudiesillustratetheembryonicnatureof efforts(e.g.woolfet al.,1999).dissertationsandprojectreportsareoftenarchived andareavailableforstudy­althoughcurrentlynotcomprehensivelyacrossall institutions.suchproductswouldbeamenabletosystematicpeerreviewwithineach subject'sacademiccommunity,inawaythattheexternalexaminersystemsignally failstodo(warren-Piper,1994).suchproductswouldalsobeamenabletoreview byeducationalresearchersusingagenericframeworkforcategorisingthequalityof learningoutcomessuchasthesolo(structureoftheobservedlearningoutcome) taxonomy(biggsandcollis,1982),whichiscapableofdistinguishinglevelsofquality acrossdifferentformsofassessmentproductwithinsubjects,andevenacrosssubjects. 7.9 T hepotentialforimprovedquality,andtheevaluationof improvedquality

Thelackofarelationshipbetweenresearchperformance,funding,ssrsandstudent selectivity,ontheonehand,andstudentengagementandeducationalgainson theother,thatmakesthesepresagevariablessuchpoorindicatorsofquality,is notinevitable­itisnotlikeaninvariantphysicallaw,i.e.itdoesnotapplytoall circumstances,foralltime.itisinpartaconsequenceofcomparativelywell-funded,

48

dimensionsofqualiT y

selective,research-orientedinstitutionsnotexploitingtheirpotentialadvantages tothefull.iftheyweretochangetheirpracticesinlinewithwhatisknownabout educationaleffectiveness,inotherwordsemphasiseprocessvariablesratherthan relyingonpresagevariablestodothejobontheirown,thenthisrelationshipcould change,providedthattheirpotentialadvantagesenabledthemtoimplementthe processesmorethoroughlythanothers. Thereisarapidlygrowingevidencebaseintheusabouttheimpacton educationalgainsofchangesinevidence-basededucationalprocesses(cf.carininet al.,2006).itisbecomingeverclearer,asaconsequence,whateducationalprocesses itissensibletointroducewithareasonablelikelihoodofimprovingeducationalgains. ThereisnosuchevidencebaseintheuK,orevenamethodologyforbuildingone. whileinenglandtherearesomedeliberateinstitutionaleffortstoimprovenss scorestheydonotalwaysappeartobebasedonanyvalidconceptualframework concerningwhatprocessesarelikelytoproducewhatoutcomesandthenssdoes notcurrentlyhavevalidityasanindicatorofqualityinthesensethatitpredicts outcomes,letalonegains.anexampleofanalternativeapproachintheuKisthe TesTaprojectbasedattheuniversityofwinchester(TesTa,2010),whichemploys establishedmethodologiesforauditingassessmentpracticesandformeasuring studentlearningresponsestothosepractices,bothbeforeandafterevidencebasedchangesinassessment,involvingwholedegreeprogrammes,acrossanumber ofinstitutions.Progresshasbeenmadeintheuspartlybecausealargenumber ofinstitutionshavebeenpreparedtoadoptthesameevidence-basedconceptual frameworkandusethesamevalidevaluationtools.Thisallowsmeaningfulevaluation datatobepooled,andsoenablingconclusionstobedrawnthatstandoutfromthe noiseofcontextualvariation.Thehighereducationacademyisstartingtotakea leadinidentifyinganddisseminatingsuitableevaluationtoolsandmethodologies,and creatinganevidenceinfrastructurewithinwhichdatafromlocallyconductedstudies couldbecollated.

49

ThehiGhereducaTionacademy

8.

abouTTheauThor

GrahamGibbshasspent35yearsinresearchanddevelopmentworktoimprovethe qualityofteaching,studentlearningandassessmentinhighereducation. hehasbeencentrallyinvolvedinaseriesofnationalteachingdevelopment initiatives,includingtheTeachingmorestudentsProjectandhefce'sinstitutional learningandTeachingstrategyinitiative,andintheco-ordinationofthefundfor thedevelopmentofTeachingandlearning.heisthefounderoftheimproving studentlearningsymposiumandoftheinternationalconsortiumforeducational developmentinhighereducation.hehasbeenawardedanhonorarydoctorateby sheffieldhallamuniversityforhisleadershipofthedevelopmentofteachinginthe uK,andbytheuniversityofutrechtforhisinternationalleadershipofeffortsto improveuniversityteaching. heretiredfromhispositionasProfessoranddirectoroftheoxfordlearning institute,attheuniversityofoxford,in2007.

50

dimensionsofqualiT y

9.

acKnowledGemenTs

inundertakingbackgroundresearchforthisreportireceivedsupportfrombahram bekhradnia,Johnbrennan,davidwatsonandmantzyorke,andalsofromroger brownwhoinadditionprovidedinvaluableguidanceinrevisingdrafts.Theyhelped metolocateitsfocusinrelationtoexistingliteratureandinrelationtothenature ofcurrentdebatesaboutqualityandstandardsinhighereducation.Theirwisdomis greatlyappreciated. iwouldalsoliketothankhighereducationacademystafffortheirsupport,and inparticulardrrachelsegalforhereditorialguidance

51

ThehiGhereducaTionacademy

10.

references

abrami,P.c.,d'appolonia,s.andcohen, P.a.(1990)validityofstudentratingsof instruction:whatweknowandwhatwedo not. Journal of Educational Psychology. 82(2), pp219­231. `adjwilson'.(2010)ifthecapfits[online].london: educationGuardian.availablefrom: www.guardian.co.uk/education/2010/may/25/ education-letters-gibb-adhd-tuition-fees [12august2010]. armstrong,m.,clarkson,P.andnoble,m. (1998)Modularity and credit frameworks: the NUCCAT survey and 1998 conference report. newcastle-upon-Tyne:northernuniversities consortiuumforcreditaccumulationand Transfer. ashby,e.,Plant,K.,ericsson,a.,hill,l.and asberg,K.(2005)whystudytimedoesnot predictgradepointaverageacrosscollege students:implicationsofdeliberatepractice foracademicperformance.Contemporary Educational Psychology.30(1),pp96­116. astin,a.(1977)Four critical years. sanfrancisco: Jossey-bass. astin,a.w.(1985)Achieving educational excellence: a critical assessment of priorities and practices in higher education.sanfranciscoandlondon: Jossey-bass. astin,a.(1993)What matters in college. san francisco:Jossey-bass. bales,r.f.,strodtbeck,f.l.,mills,T.m.and roseborough,m.(1951)channelsof communicationinsmallgroups.American Sociological Review.16(4),pp461­468. barefootb.o.(2004)highereducation'srevolving door:confrontingtheproblemofdrop-out atuscollegesanduniversities.Open Learning, 19,(1),pp65­77. barrie,s.andGinns,P.(2007)Thelinkingof nationalteachingperformanceindicators

toimprovementsinteachingandlearningin classrooms.Quality in Higher Education.13(3), pp275­286. bauer,K.w.andbennett,J.s.(2003)alumni Perceptionsusedtoassessundergraduate researchexperience.Journal of Higher Education. 74(2),pp210­230. bergren,m.,snover,l.andbreslow,l.(2007) undergraduateresearchopportunitiesat miT.Illuminatio.spring2007,pp6­8. biggs,J.b.(1993)fromtheorytopractice:a cognitivesystemsapproach. Higher Education Research and Development. 12(1),pp73­85. biggs,J.andcollis,K.f.(1982)Evaluating the quality of learning: the SOLO Taxonomy (Structure of the observed learning outcome). newyork: academicPress. black,P.andwiliam,d.(1998)assessmentand classroomlearning.Assessment in Education. 5(1),pp7­74. bloom,b.s.,englehart,m.d.,furst,e.J.,hill,w.h. andKrathwohl,d.r.(1956)Taxonomy of educational objectives. Handbook 1: Cognitive domain.newyork:davidmcKay. bound,J.andTurner,s.(2005)cohortcrowding: howresourcesaffectcollegiateattainment. Journal of Public Economics.91(5­6),pp877­899. bowden,r.(2000)fantasyhighereducation: universityandcollegeleaguetables. Quality in Higher Education.6(1),pp41­60. brennan,J.andTang,w.(2008a) The employment of UK graduates: comparisons with Europe. REFLEX Report to HEFCE No 1.bristol:hefce. brennan,J.andTang,w.(2008b) Subject differences in graduate employment across Europe. REFLEX Report to HEFCE No 2. bristol:hefce. brennan,J.,Patel,K.andTang,w.(2009) Diversity in the student learning experience and time devoted to study: a comparative analysis of the UK and European evidence.bristol:hefce.

52

dimensionsofqualiT y

bridges,P.,cooper,a.,evanson,P.,haines,c., Jenkins,d.,scurry,d.,woolf,h.andyorke,m. (2002)courseworkmarkshigh,examination markslow:discuss.Assessment and Evaluation in Higher Education.27(1),pp35­48. brown,r.(2006)leagueTables­dowehaveto livewiththem?Perspectives: Policy and Practice in Higher Education. 10(2),pp33­38. brown,r.(2007) The information fallacy.oxford: highereducationPolicyinstitute.available from:www.hepi.ac.uk/484-1291/Theinformation-fallacy.html[June2010]. brown,r.(2010) Comparability of degree standards? oxford:highereducationPolicyinstitute. availablefrom:www.hepi.ac.uk/455-1838/ comparability-of-degree-standards.html [June2010]. brown,r.,carpenter,c.,collins,r.andwinkwistnoble,l.(2009)recentdevelopmentsin informationaboutprogrammequality.Quality in Higher Education. 13(2),pp173­186. carinin,r.,Kuh,G.andKlein,s.(2006)student engagementandstudentlearning:testing thelinkages.Research in Higher Education. 47 (1),pp1­32. carney,c.mcneish,s.andmccoll,J.(2005)The impactofpart-timeemploymentonstudents' healthandacademicperformance:ascottish perspective. Journal of Further and Higher Education. 29(4),pp307­319. cheng,y.c.andTam,w.m.(1997)multi-models ofqualityineducation. Quality Assurance in Education. 5(1),pp22­31. chickering,a.w.(1974)Commuting versus resident students: Overcoming the educational inequities of living off campus. sanfrancisco:Jossey-bass. chickering,a.w.andGamson,Z.f.(1987a)Seven Principles for Good Practice in Undergraduate Education.racine,wi:TheJohnson foundationinc. chickering,a.w.andGamson,Z.f.(1987b)seven principlesforgoodpracticeinundergraduate education.AAHE Bulletin. 39(7),pp3­7. chickering,a.w.andGamson,Z.f.(1991) Applying the seven principles for good practice in undergraduate education.sanfrancisco: Jossey-bass. clarke,m.(2002)someguidelinesforacademic qualityrankings. Higher Education in Europe. 27(4),pp443­459.

coffey,m.andGibbs,G.(2000)Theevaluationof thestudentevaluationofeducationalquality questionnaireinuKhighereducation. Assessment and Evaluation in Higher Education. 26(1),pp89­93. cook,r.,butcher,i.andraeside,r.(2006) recountingthescores:ananalysisofthe qaasubjectreviewgrades1995­2001. Quality in Higher Education.12(2),pp135­144. curtis,s.andshani,n.(2002)Theeffectof takingpaidemploymentduringterm-timeon students'academicstudies.Journal of Further and Higher Education.26(2),pp129­138. curtis,s.andwilliams,J.(2002)Thereluctant workforce:undergraduates'part-time employment.Education and Training. 44(1), pp5­10. dochy,f.,segers,m.,vandenbossche,P.and Gijbels,d.(2003)effectsofproblem-based learning:ameta-analysis.Learning and Instruction. 13(5),pp533­568. drennan,l.T.andbeck,m.(2001)Teaching qualityPerformanceindicators­key influencesontheuKuniversities'scores. Quality Assurance. 9(2),pp92­102. dunbar-Goddet,h.andTrigwell,K.(2006) A study of the relations between student learning and research-active teachers. Paperpresentedatthe14thinternational improvingstudentlearningsymposium, bath,4­6september. eccles,c.(2002)Theuseofuniversityrankings intheunitedKingdom.Higher Education in Europe. 27(4),pp423­432. ehrenberg,r.G.(2006) What's Happening in Public Higher Education?westport,cT:Praeger. ewell,P.(2008)nocorrelation:musingsonsome mythsaboutquality.Change.novemberdecember2008,40(6),pp8­13. fearnley,s.(1995)classsize:theerosiveeffect ofrecruitmentnumbersonperformance. Quality in Higher Education. 1(1),pp59­65. feldman,K.(1984)classsizeandcollegestudents' evaluationsofteachersandcourses:acloser look. Research in Higher Education.21(1), pp45­116. finnie,r.andusher,a.(2005)Measuring the Quality of Post-secondary Education: Concepts, Current Practices and a Strategic Plan. Kingston, on:canadianPolicyresearchnetworks.

53

ThehiGhereducaTionacademy

ford,J.,bosworth,d.andwilson,r.(1995)Parttimeworkandfull-timehighereducation. Studies in Higher Education.20(2),pp187­202. Gansemer-Topf,a.,saunders,K.,schuh,J. andshelley,m.(2004)A study of resource expenditure and allocation at DEEP colleges. ames,ia:educationalleadershipandPolicy studies,iowastateuniversity. Gardiner,l.f.(1997)Redesigning higher education: producing dramatic gains in student learning. ASHE-ERIC Higher Education Report 7. washingtondc:associationforthestudyof highereducation. Gibbs,G.(1995)nationalscalefaculty developmentforteachinglargeclasses. in:wright,a.(ed.)Teaching Improvement Practices.newyork:anker. Gibbs,G.(1999)arethepedagogiesofthe disciplinesreallydifferent?in:rust,c. (ed.)Improving Student Learning Through the Disciplines. oxford:oxfordcentreforstaff andlearningdevelopment. Gibbs,G.(2008)Designing teaching award schemes. york:highereducationacademy. Gibbs,G.(2010)The assessment of group work: lessons from the literature.oxford: assessmentstandardsKnowledgeexchange. availablefrom:www.brookes.ac.uk/aske/ documents/brookes%20groupwork%20 Gibbs%20dec%2009.pdf[June2010]. Gibbs,G.andcoffey,m.(2004)Theimpactof trainingofuniversityteachersontheirteaching skills,theirapproachtoteachingandthe approachtolearningoftheirstudents.Active Learning in Higher Education.5(1),pp87­100. Gibbs,G.anddunbar-Goddet,h.(2007)The effects of programme assessment environments on student learning. york:highereducationacademy. Gibbs,G.anddunbar-Goddet,h.(2009) characterisingprogramme-levelassessment environmentsthatsupportlearning. Assessment and Evaluation in Higher Education. 34(4),pp481­489. Gibbs,G.andJenkins,a.(eds.)(1992)Teaching Large Classes: maintaining quality with reduced resources. london:KoganPage. Gibbs,G.andlucas,l.(1997)coursework assessment,classsizeandstudent performance:1984­94. Journal of Further and Higher Education.21(2),pp183­192.

Gibbs,G.,habeshaw,T.andyorke,m.(2000) institutionallearningandteachingstrategies inenglishhighereducation.Higher Education. 40(3),pp351­372. Gibbs,G.,Knapper,c.andPicinnin,s.(2008a) Departmental leadership for quality teaching: an international comparative study of effective practice. london:leadership foundation.availablefrom:www.lfhe. ac.uk/research/projects/gibbsoxford.html [may2010]. Gibbs,G.,Knapper,c.andPicinnin,s.(2008b) disciplinaryandcontextuallyappropriate approachestoleadershipofteachingin research-intensiveacademicdepartmentsin highereducation.Higher Education Quarterly. 62(4),pp416­436. Gibbs,G.,lucas,l.andsimonite,v.(1996)class sizeandstudentperformance:1984­94. Studies in Higher Education. 21(3),pp261­273. Gibbs,G.,morgan,a.andTaylor,e.(1982) areviewoftheresearchofference martonandtheGoteborgGroup:a phenomenologicalresearchperspective onlearning.Higher Education. 11(2), pp123­145. Gibbs,G.,regan,P.andsimpson,o.(2006) improvingstudentretentionthrough evidencebasedproactivesystemsatthe openuniversity(uK).College Student Retention. 8(3),pp359­376. Glass,G.v.andsmith,m.l.(1978) Meta-Analysis of Research on the Relationship of Class Size and Achievement. sanfrancisco:farwest laboratoryforeducationalresearchand development. Glass,G.v.andsmith,m.l.(1979)meta-analysis ofresearchontherealationshipofclasssizeandachievement.Evaluation and Policy Analysis. 1,pp2­16. Graham,a.andThompson,n.(2001)broken ranks:usnews'collegerankingsmeasure everythingbutwhatmatters.andmost universitiesdon'tseemtomind.Washington Monthly. 33(4),pp9­14. Grunig,s.G.(1997)research,reputationand resources:theeffectofresearchactivityon perceptionsofundergraduateeducationand institutionalresourceacquisition. Journal of Higher Education. 33(9),pp9­14.

54

dimensionsofqualiT y

hannan,a.andsilver,h.(2000) Innovating in Higher Education: teaching, learning and institutional cultures.buckingham:Thesociety forresearchintohighereducation/open universityPress. harvey,l.andGreen,d.(1993)definingquality. Assessment and Evaluation in Higher Education. 18(1),pp9­34. hathaway,r.s.,nagda,b.a.andGregerman,s.r. (2002)Therelationshipofundergraduate researchparticipationtograduateand professionaleducationpursuit:anempirical study. Journal of College Student Development. 43(5),pp614­631. hattie,J.andmarsh,h.w.(1996)Therelationship betweenresearchandteaching:ametaanalysis.Review of Educational Research.66(4), pp507­542. hattie,J.andTimperley,h.(2007)Thepowerof feedback.Review of Educational Research.77 (1),pp81­112. hattie,J.,bibbs,J.andPurdie,n.(1996)effects oflearningskillsinterventionsonstudent learning:ameta-analysis.Review of Educational Research.66(2),pp99­136. havnes,a.(2008),There is a bigger story behind. An analysis of mark average variation across Programmes. europeanasssociationforresearch intolearningandinstructionassessment conference.universityofnorthumbria. hefce(2001)Analysis of Strategies for Learning and Teaching. Report 01/37a.bristol:higher educationfundingcouncilforengland. hePi(2006)The Academic Experience of Students in English Universities (2006 Report). oxford: highereducationPolicyinstitute. hePi(2007) The Academic Experience of Students in English Universities (2007 Report). oxford: highereducationPolicyinstitute. hochschul-informations-system(2005)Eurostudent 2005: Social and Economic conditions of student life in Europe 2005. hannover:his. hoskins,s.,newstead,s.e.anddennis,i.(1997) degreePerformanceasafunctionofage, Gender,Priorqualificationsanddiscipline studied.Assessment & Evaluation in Higher Education. 22(3),pp317­328. houseofcommonsinnovations,universities, scienceandskillscommittee(2009) Students and Universities. Eleventh Report

of Session 2008-09. Volume 1.london:The stationeryoffice. huber,m.T.andmorreale,s.(2002)Disciplinary styles in the scholarship of teaching and learning: exploring common ground. washingtondc: americanassociationforhighereducation andthecarnegiefoundationforthe advancementofTeaching. hunt,a.,lincoln,i.andwalker,a.(2004)Termtimeemploymentandacademicattainment: evidencefromalarge-scalesurveyof undergraduatesatnorthumbriauniversity. Journal of Further and Higher Education.28(1), pp3­18. innis,K.andshaw,m.(1997)howdostudents spendtheirtime?Quality Assurance in Education. 5(2),pp85­89. Jenkins,a.(2004)A guide to the research on teaching-research relations. york:higher educationacademy. Jenkins,a.,Jones,l.andward,a.(2001)The long-termeffectofadegreeonGraduate lives.Studies in Higher Education. 26(2), pp147­161. Jessop,T.&el-hakim,y.(2010)Evaluating and improving the learning environments created by assessment at programme level: theory and methodology. europeanassociationfor researchintolearningandinstruction assessmentconference,universityof northumbria,June2010. Johnes,G.(1992)Performanceindicatorsinhigher education:asurveyofrecentwork.Oxford Review of Economic Policy. 8(2),pp19­33. Jongbloed,b.w.a.andJ.J.vossensteyn(2001) KeepingupPerformances:aninternational surveyofperformancebasedfundingin highereducation.Journal of Higher Education Policy and Management. 23(2),pp127­145. Kehm,b.m.andstensaker,b.(2009) University rankings, diversity and the landscape of higher education. rotterdam:sensePublishers. Kuh,G.d.andPascarella,e.T.(2004)what doesinstitutionalselectivitytellusabout educationalquality?Change.septemberoctober2004,36(5),pp52­58. lanasa,s.,olson,e.andalleman,n.(2007)The impactofon-campusstudentgrowthon first-yearengagementandsuccess. Research in Higher Education. 48(8),pp941­966.

55

ThehiGhereducaTionacademy

lindsay,r.andPaton-saltzberg,r.(1987) resourcechangesandacademicperformance atanenglishPolytechnic. Studies in Higher Education. 12(2),pp213­27. lindsay,r.,breen,r.andJenkins,a.(2002) academicresearchandteachingquality:the viewsofundergraduateandpostgraduate students.Studies in Higher Education. 27(3), pp309­327. lucas,l.,Jones,o.,Gibbs,G.,hughes,s.and wisker,G.(1996)The effects of course design features on student learning in large classes at three institutions: a comparative study.Paperpresentedatthe4th internationalimprovingstudentlearning symposium,bath. marsh,h.w.(1982)seeq:areliable,validand usefulinstrumentforcollectingstudents' evaluationsofuniversityteaching.british Journal of Educational Psychology. 52,pp77­95. marsh,h.w.(1987)students'evaluationsof universityteaching:researchfindings, methodologicalissues,anddirectionsfor futureresearch.International Journal of Educational Research.1(3),(entireissue). martin,e.,Trigwell,K.,Prosser,m.andramsden, P.(2003)variationintheexperienceof leadershipofteachinginhighereducation. Studies in Higher Education. 28(3),pp247­259. marton,f.andwenestam,c.(1978)qualitative differencesintheunderstandingand retentionofthemainpointinsometexts basedontheprinciple-examplestructure. in:Gruneberg,m.m.,morris,P.e.andsykes, r.n.(eds.)Practical aspects of memory. london:academicPress. marton,f.,hounsell,d.andentwistle,n.(1984) The experience of learning.edinburgh:scottish academicPress. mentkowski,m.anddoherty,a.(1984)Careering After College: Establishing the Validity of Abilities Learned in College for Later Careering and Professional Performance. Final Report to the National Institute of Education.milwaukee,wi: alvernocollege. nasr,a.,Gillett,m.andbooth,e.(1996) lecturers'teachingqualificationsand theirteachingperformance. Research and Development in Higher Education. 18, pp576­581.

nchems(2003)Do DEEP institutions spend more or differently than their peers? boulder,co: nationalcentreforhighereducation managementsystems. nus(2008)NUS Student Experience Report. london:nationalunionofstudents. oecd(2000)Education at a Glance 2000. Paris: organisationforeconomiccooperationand development. olcott,d.(2010)Parforthecourse.Times Higher Education.8april,p32. Pascarella,e.T.(1980)student-facultyinformal contactandcollegeoutcomes.Review of Educational Research.50(4),pp545­595. Pascarella,e.T.(2001)identifyingexcellencein undergraduateeducation:areweevenclose? Change.33(3),pp19­23. Pascarella,e.T.andTerenzini,P.(1991) How college affects students.sanfrancisco:Jossey-bass. Pascarella,e.T.andTerenzini,P.(2005)How college affects students: a third decade of research, Volume 2. sanfrancisco:Jossey-bass. Pascarella,e.T.,cruce,T.,umbach,P.,wolniak, G.,Kuh,G.,carini,r.,hayek,J.,Gonyea,r. andZhao,c.(2006)institutionalselectivity andgoodpracticesinundergraduate education:howstrongisthelink?Journal of Higher Education.77(2),pp251­285. Pascarella,e.T.,seifert,T.a.andblaich,c.(2008) Validation of the NSSE benchmarks and deep approaches to learning against liberal arts outcomes. Paperpresentedattheannual meetingoftheassociationforthestudyof highereducation,Jacksonville,fl.available from:www.education.uiowa.edu/crue/ publications/index.htm[march2010]. Pascarella,e.T.,seifert,T.a.andblaich,c.(2010) howeffectivearethenssebenchmarksin predictingimportanteducationaloutcomes? Change. January-february2010,42(1), pp16­22.availablefrom:www.changemag. org/index.html[march2010]. Paton-saltzberg,r.andlindsay,r.(1993)The Effects of Paid Employment on the Academic Performance of Full-time Students in Higher Education. oxford:oxfordPolytechnic. Patrick,J.P.andstanley,e.c.(1998)Teachingand researchqualityindicatorsandtheshaping ofhighereducation. Research in Higher Education.39(1),pp19­41.

56

dimensionsofqualiT y

Perry,w.G.(1970)Forms of intellectual and ethical development in the college years: a scheme. newyork:holt,rhinehartandwinston. Piccinin,s.,cristi,c.andmccoy,m.(1999)The impactofindividualconsultationonstudent ratingsofteaching. International Journal of Academic Development. 4(2),pp75­88. Prosser,m.andTrigwell,K.(1997)Perceptionsof theteachingenvironmentanditsrelationship toapproachestoteaching.British Journal of Educational Psychology. 67,pp25­35. qaa(2003)Learning from Subject Review 1993­ 2001. Gloucester:qualityassuranceagency forhighereducation. qaa(2006) Background Briefing Note: The classification of degree awards. Gloucester:qualityassurance agencyforhighereducation. qaa(2009)Thematic enquiries into concerns about academic quality and standards in higher education in England. Gloucester:quality assuranceagencyforhighereducation. ramsden,P.(1979)studentlearningand perceptionsoftheacademicenvironment. Higher Education. 8(4),pp411­427. ramsden,P.(1998) Learning to lead in higher education. london:routledge. ramsden,P.(1999)aperformanceindicatorof teachingqualityinhighereducation:the courseexperiencequestionnaire.Studies in Higher Education. 16(2),pp129­150. ramsden,P.andmoses,i.(1992)associationsbetween researchandteachinginaustralianhigher education.Higher Education.23(3),pp273­295. säljö,r.(1979)learningaboutlearning.Higher Education. 8(4),pp443­451. sastry,T.andbekhradnia,b.(2007)The academic experience of students in English universities. london:highereducationPolicyinstitute. schmitz,c.(1993)assessingthevalidityofhigher educationindicators. Journal of Higher Education. 64(5),pp503-521. schomburg,h.andTeichler,u.(2006)Higher education and graduate employment in Europe: Results from graduate surveys from twelve countries. dordrecht:springer. schwartz,d.l.,bransford,J.d.andsears,d.(2005) Efficiency and innovation in transfer. stanford, ca:stanforduniversity.availablefrom:www. stanford.edu/~danls/efficiency%20and%20 innovation%204_2004.pdf[april2010].

simpson,o.(2003)Student retention in open and distance learning. london:routledgefalmer. smith,m.l.andGlass,G.v.(1979)Relationship of class-size to classroom processes, teacher satisfaction and pupil affect: a meta-analysis. sanfrancisco,ca:farwestlaboratoryfor educationalresearchanddevelopment. smith,J.andnaylor,r.(2005)schoolingeffectson subsequentuniversityperformance.Economics of Education Review. 24(5),pp549­562. smith,J.,mcKnight,a.andnaylor,r.(2000) Graduateemployability:policyand performanceinhighereducationintheuK. The Economic Journal. 110(464),pp382­411. stinebrickner,r.andstinebrickner,T.(2008) Thecausaleffectofstudyingonacademic Performance.The B.E. Journal of Economic Analysis and Policy. 8(1),article14. svanum,s.andbigatti,s.m.(2006)Theinfluences ofcourseeffortandoutsideactivitieson Gradesinacollegecourse. Journal of College Student Development.47(5),pp564­576. svensson,l.(1977)onqualitativedifferences inlearning:iii­studyskillandlearning. British Journal of Educational Psychology. 47, pp223­243. Terenzini,P.T,andPascarella,e.T.(1994)living myths:undergraduateeducationinamerica. Change. 26(1),pp28­32. TesTa(2010)Transforming the Experience of Students Through Assessment. available from:www.winchester.ac.uk/studyhere/ excellenceinlearningandTeaching/research/ Pages/TesTa.aspx[12august2010]. Tinto,v.(1975)dropoutfromhighereducation: atheoreticalsynthesisofrecentresearch. Review of Educational Research. 45(1), pp89­125. Thompson,n.(2000)Playingwithnumbers:how usnewsmis-measureshighereducationand whatwecandoaboutit.Washington Monthly. 32(9),pp16­23. Trigwell,K.(2005)Teaching­researchrelations, cross-disciplinarycollegialityandstudent learning.Higher Education. 49(3),pp235­254. Trigwell,K.andashwin,P.(2004)Undergraduate students' experience at the University of Oxford.oxford:oxfordlearninginstitute. availablefrom:www.learning.ox.ac.uk/oli. php?page=365[april2010].

57

ThehiGhereducaTionacademy

Trigwell,K.&Prosser,m.(2004)Thedevelopment anduseoftheapproachestoTeaching inventory.Educational Psychology Review, 16,4, pp409­424. usher,a.andsavino,m.(2006)A world of difference: a global survey of university league tables. Toronto, on:educationalPolicyinstitute.available from:www.educationalpolicy.org/pdf/world-ofdifference-200602162.pdf[12august2010]. vanrossum,e.J.,deijkers,r.andhamer,r. (1985)students'learningconceptionsand theirinterpretationofsignificanteducational concepts.Higher Education.14(6),pp617­641. vos,P.(1991) Curriculum Control of Learning Processes in Higher Education.13th internationalforumonhighereducationof theeuropeanassociationforinstitutional research.edinburgh. warren-Piper,d.(1994) Are Professors Professional? Theorganisationofuniversityexaminations. london:JessicaKingsley. weimer,m.andlenze,l.f.(1997)instructional interventions:areviewofliteratureon effortstoimproveinstruction.in:Perry,r.P. andsmart,J.c.(eds.) Effective Teaching in Higher Education: Research and Practice. new york:agathonPress. wood,K.,limsky,a.s.,andstraus,m.a.(1974). classsizeandstudentevaluationsoffaculty. Journal of Higher Education.43,pp524­34. woodley,a.(2004)conceptualisingstudent drop-outinparttimedistanceeducation: pathologisingthenormal?Open Learning.19 (1),pp47­63. woolf,h.,cooper,a.,bourdillon,b.,bridges,P., collymore,d.,haines,c.,Turner,d.and yorke,m.(1999)benchmarkingacademic standardsinhistory:anempiricalexercise. Quality in Higher Education. 5(2),pp145­154. yorke,m.(1997)agoodleaguetableguide?Quality Assurance in Higher Education. 5(2),pp61­72. yorke,m.(1998)TheTimes`leagueTable'of universities,1997:astatisticalappraisal. Quality in Education.6(1),pp58­60. yorke,m.(1999)Leaving early: undergraduate noncompletion in higher education.london:falmer. yorke,m.(2001)formativeassessmentandits relevancetoretention.Higher Education Research and Development. 20(2),pp115­126. yorke,m.(2009)Trends in honours degree

classifications, 1994­95 to 2006­07, for England, Wales and Northern Ireland. york: highereducationacademy. yorke,m.,bridges,P.andwoolf,h.(2000)mark distributionsandmarkingpracticesinuK highereducation;somechallengingissues. Active Learning in Higher Education. 1(1), pp7­27. yorke,m.,barnett,G.,bridges,P.,evanson, P.,haines,c.,Jenkins,d.,Knight,P.,scurry, d.,stowell,m.andwoolf,h.(2002)does gradingmethodinfluencehonoursdegree classification? Assessment and Evaluation in Higher Education. 27(3),pp269­279. yorke,m.,woolf,h.,stowell,m.,allen,r., haines,c.,redding,m.,scurry,d.,Taylorrussell,G.,Turnbull,w.andwalker,w. (2008)enigmaticvariations:honoursdegree assessmentregulationsintheuK.Higher Education Quarterly. 63(3),pp157­180.

58

dimensionsofqualiT y

59

ThehiGhereducaTionacademy

60

dimensionsofqualiT y

61

The Higher Education Academy supports the sector in providing the best possible learning experience for all students. It does this by: -- providing national leadership in developing and disseminating evidence-informed practice about enhancing the student learning experience -- operating as an independent broker, enabling expertise to be shared across institutions and subject areas -- working at multiple levels, with individual academics, subject communities, departments, faculties and institutions -- working across all parts of the UK, recognising the distinctive policy contexts and priorities of the devolved administrations but also provising opportunities to share expertise among them. The Academy is an independent organisation funded by grants from the four UK higher education funding bodies, subscriptions from higher education institutions, and grant and contract income for specific initiatives.

www.heacademy.ac.uk

Information

64 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

980910