Blog of the International Sociological Association (ISA)

Germans Boycott University Rankings

June 21, 2013 39 Comments

Scientific Evaluation, Yes – CHE Ranking, No

Methodological Problems and Political Implications of the CHE University Ranking

German Sociological Association Statement

June 2013 (long version)

The results of the CHE (Centre for Higher Education Development) University Ranking, a subject-level classification covering a range of academic disciplines, have been published each spring since 1998. The ranking has acquired high public visibility by virtue of the fact that it has been published in the weekly newspaper DIE ZEIT and in the annual ZEIT Studienführer (Study Guide) since 2005.

Doubts about the professional quality of the CHE Ranking have been voiced repeatedly within the field of sociology since it was first implemented. However, in view of the informational needs of prospective students of sociology, sociological institutes have participated in the data collection for the ranking. Rather than neglecting to mention it here, we self-critically acknowledge that sociology and the social sciences have been officially represented on the CHE Advisory Board in the past and that they may not have exercised, and availed of, their influence and their supervisory responsibilities – or at least may not have done so effectively enough.

However, since the middle of last year, mounting professional and science-policy-related misgivings on the part of a number of sociological institutes have led to a rethink. In June 2011, the Institute of Sociology at the University of Jena – which had consistently received very good ratings from the CHE – decided that it no longer wished to participate in the CHE Ranking. This prompted the Board of the German Sociological Association (GSA) to undertake a thorough analysis of the CHE Ranking. After studying the available documentation and conducting a lengthy discussion with the representatives of the Centre for Higher Education Development responsible for the ranking, the GSA Board arrived at the appraisal and the recommendations documented below. At its meeting on 20 April 2012, the GSA Council endorsed this appraisal and unanimously adopted the recommendations ensuing therefrom.

Professional and Science-Policy-Related Appraisal of the CHE Ranking

Firstly, the CHE Ranking has a number of serious methodological weaknesses and empirical gaps. Secondly, the summary assessment practice and the specific publication formats of the ranking systematically invite misinterpretations. Both aspects will be discussed in greater detail here.

Professional Appraisal: Research Indicators

For a number of years, at least, the quality of the research conducted at the individual faculties was measured on the basis of publication databases that not only the German Council of Science and Humanities (Wissenschaftsrat), but, meanwhile, also the CHE itself, deems to be an unsuitable, or – in the case of sociology, at least – an insufficiently meaningful indicator. As an alternative, the CHE now measures research performance on the basis of external research funding per (budgeted) academic staff member. When doing so – and without any further differentiation – Higher Education Pact positions, for example, which were created expressly not for research purposes but rather to cope with teaching loads, are also included in the divisor of the external funding values. In effect, this means that – in purely arithmetical terms – as the teaching load of an institute increases (in the area of teacher training, for example), its per capita research performance, which the CHE claims to “measure”, deteriorates. It is obvious that the universities particularly affected are those that, because of the region in which they are located, have taken in a large number of students within the framework of the Higher Education Pact. Thus, the “burden of proof” of the quality of research of an individual institute is borne almost entirely by the subjective criterion of that institute’s research reputation among fellow academics at other  – in the logic of the ranking, rival – institutions. Anyone who has ever participated in the CHE survey of professors will be aware of its lack of methodological sophistication and the undifferentiated nature of its contents. The informational value of such sweeping faculty-specific judgements for prospective students, as the intended target audience of the ranking, is definitely questionable.

Professional Appraisal: Teaching Indicators

For this specific target audience the central criterion for the choice of a possible study location is obviously the quality of teaching at the various sociological institutes. However, this indicator is measured by the CHE largely on the basis of a student survey characterised by (a) low response rates (19.3% in sociology in the last round), (b) a small number of participants (at every third university, less than 30 students from the subject area in question), and (c) completely unexplained survey selectivity, with the result that the danger of responses biased by careless or inattentive response behaviour is correspondingly high. The CHE is well aware of the fact that by no means all universities draw a genuinely random sample with a calculable probability of selection. Moreover, a self-administered questionnaire survey with no systematic reminders and no nonresponse study can claim practically no validity. By the end of his or her basic training in methodology at the latest, any student of sociology would recognize that the survey is simply absurd. Therefore, massive doubts must be expressed with regard to the results of the CHE student survey – which is often described in discussions about the ranking as an opportunity for student participation qua evaluation.

Moreover, important, if not decisive, parameters for the assessment of the study situation – parameters that cannot be influenced by the teaching staff – are not included in the analysis (or the evaluation) at all. These parameters include, for example, (a) the respective faculty-student ratio (the ratio of the teaching load of faculty employed in budgeted positions to the number of students), (b) the associated arithmetical (and actual) class sizes, and (c) the efficiency of examination offices. Furthermore, the CHE forgoes the collection of qualitative data that are, or would be, extremely relevant for the assessment of the quality of teaching at the individual locations and for prospective students’ choice of study programmes, for example, the areas of focus and specialization offered by the various sociological study programmes, and the systematic linking of teaching with the research conducted at the institute in question – irrespective of the external-funding intensity or reputational standing of that research. Such an inadequate, extremely selective, and factually misleading data situation renders absolutely untenable the construction of a ranking of institutes with regard to their teaching performance.

Science-Policy-Related Appraisal: Evaluation Practice and Publication Formats

The basic problem with the university ranking is that the Centre for Higher Education Development aims to construct a ranking of institutes with regard to their teaching performance, and actually “succeeds” in doing so, namely by dividing sociological institutes on the basis of extremely doubtful data into “good” and “bad” – or “better” and “worse” – institutes, and listing them hierarchically with spurious accuracy.  Because of the sweeping evaluation practice and simplistic modes of presentation, the publication formats of the ranking invite systematic misconceptions about the situation in sociology.

The CHE collects data for a total of approximately eighteen indicators of research and teaching quality in the field of sociology, and these indicators are also published in the online version of the ranking. However, for a description of the individual indicators and their derivation, readers are referred to the small print, which most people are unlikely to understand. In the print version published in DIE ZEIT and in the ZEIT Studienführer (Study Guide), however, these eighteen indicators are not combined to form indices. Rather, only five or six indicators are selectively presented. This fact is neither discernible from a cursory reading, nor is any explanation given for the selection that has been made. Moreover, for both the quality of research and the quality of teaching, only the subjective evaluations from what we have shown to be methodologically extremely questionable surveys are presented. In particular, the simplistic ranking by means of traffic-light symbols (recently modified to green, yellow, and blue) obscures the remarkable paucity of the database; in some cases, a single binary-coded response to a questionnaire item can yield a traffic light symbol signalling “good” or “bad” performance. The CHE Ranking – willingly bowing to the presentational demands of the mass media – gives the impression of unequivocal, reliable assessments, which are by no means covered by the available data. Here, systematic differentiations and thick descriptions would clearly be indicated and appropriate.

It is indeed disturbing in itself that the CHE Ranking thus misleads the very group whose interests, according to its authors, it is primarily supposed to serve, namely prospective students of sociology, who could, indeed, benefit from having accurate information about individual study locations when choosing a university and a study programme. It is perhaps a blessing in disguise, therefore, that – as far as teachers of sociology can ascertain – hardly any of the students who are now studying sociology at German universities, at any rate, allowed themselves to be decisively influenced by the CHE Ranking. Obviously, only a small minority of prospective students take serious note of the ranking – and that is a good thing.

On the other hand – and quite apart from its lack of informational value – the CHE Ranking has a very problematic effect on science policy. Therefore, if we are to believe the declared intentions of its authors, the ranking serves de facto a purpose for which it was not “actually” intended. However, in higher-education-policy reality, the CHE Ranking invites – or, indeed, practically demands – extremely simplistic interpretations on the part of faculty- and university management and ministerial bureaucracies. This may lead to structural decisions that have grave consequences for sociology, as an academic discipline, and its study programmes at individual locations – decisions that may well be objectively unfounded.

In view of the danger of such political uses of the CHE Ranking, it appears all the more remarkable that the persons responsible for the ranking at the CHE are unwilling to limit themselves to an informational function – however incomplete and unsatisfactory its implementation may be. They maintain that they cannot do without the construction of a ranking of the sociological institutes in Germany. At the preliminary meeting with those responsible for the ranking at the CHE, the German Sociological Association representatives were told quite openly that it would not be possible for the discipline to satisfy its own informational intentions within the framework of the procedure organized by the CHE, while at the same time avoiding the obligatory assessment and ranking. Thus, it became quite clear to the GSA that the CHE at least accepts the possibility that the university ranking will be politicized. The authors of the ranking claim that it merely depicts existing differences in quality between the sociological faculties. However, in the opinion of the German Sociological Association, there are strong grounds for assuming that the CHE Ranking contributes significantly to the construction of “difference” and, thus, to creating divisions in the university landscape in the field of sociology.

In the worst case, therefore, the ranking will act as a self-fulfilling prophecy in the long term. Faculties labelled on a supposedly sound empirical basis as “good” or “bad” may actually become so in the long run because of the structural policy decisions and – perhaps one day, after all – changing student flows prompted by their rankings. More than any other academic discipline, sociology is aware of the way in which such social definitions of situations influence action. It therefore feels both a scientific obligation to draw attention to the far-reaching consequences of actions based on incorrect definitions of situations, and a scientific responsibility not to contribute to such consequences any longer.

Recommendations Concerning the Handling of the CHE Ranking

Firstly, because the CHE Ranking has serious methodological and empirical deficiencies, secondly, because it withholds vital information from prospective students, as its declared target audience, and, thirdly, because it gives rise to wrong decisions on the part of science-policy decision-makers, sociology must take a stand against this presentation of its teaching and research performance in the public sphere constructed by the media. On the basis of this appraisal and the justifications thereof outlined above, the Board and the Council of the German Sociological Association have arrived at the following recommendations:

1.  Because our analyses and the discussion of the considerable methodological deficiencies with the CHE representatives responsible for the ranking yielded no prospect of significant improvements in the CHE Ranking in the future, we hereby declare that this evaluation does not meet the basic quality requirements of empirical social research. As a professional sociological society, we call on the sociological institutes at German universities not to give the impression any longer that they support an empirical procedure that sociology must reject on professional grounds. In concrete terms, this means that the sociological institutes should defend and explain this resolution and its professional justifications vis-à-vis their faculty- and university managers and their students, and, in particular, that they should not take part in the collection of data for the next CHE Ranking of sociology.

2.  The GSA calls on science-policy decision-makers at university and ministerial level not to rely any longer on appraisals and information derived from the CHE Ranking when deliberating on, and undertaking interventions for, the development of sociology at the discipline’s various university locations. More reliable information than that provided by the ranking already exists; in individual cases, occasion-specific evaluations should be conducted, for which both suitable concepts and unbiased institutions are available.

3.  As an empirically oriented social science discipline, sociology claims to be particularly competent in the assessment of all kinds of empirical social research – including evaluations such as the CHE Ranking. In the present case, this competency implies a responsibility to recommend other disciplines, which are perhaps less sensitive in this regard, not to participate in the CHE Ranking any longer. After all, the grave deficiencies and misuses of this ranking that have been observed in the case of sociology are equally characteristic of its application to other disciplines.

4.  Sociology is a discipline that is proficient in evaluation in every sense of the word. For this reason, it made itself available in 2006 for a pilot study on the rating (and precisely not the ranking) of research performance conducted by the Council of Science and Humanities (Wissenschaftsrat). In a process characterised by considerable social and technical complexity, this scientific rating demonstrated in an exemplary way the minimum requirements that a reliable and valid scientific evaluation must fulfil. To further meet the specific and justified desire on the part of prospective students of sociology for assistance in choosing a course of study and a study location, the GSA will develop a publicly accessible information package, which will also feature descriptions of the sociology programmes offered by German universities.

This statement, a summary thereof, and the latest information on the GSA’s science-policy initiative launched herewith are available online at www.soziologie.de/che.

Germany

39 Comments → “Germans Boycott University Rankings”

  1. Nice post !! This information about individual study locations when choosing a university and a study programme. I like so much.

  2. MONSTER Beats 11 months ago   Reply

    These kind of similar styles MONSTER Beats http://www.kumport.com.tr/images/images/monster-beats-by-dr-dre.html will be hand-crafted through a lot of special Germans Boycott University Rankings companies hiring essentially an identical techniques since early Tiffany Incandescent bulbs. Instantly following 2 decades including behave about.

  3. entreprenörskap 10 months ago   Reply

    That is actually attractive, Prepared to a really suffered writer. We have enrolled with the give plus be expecting enjoying your current incredible write-ups. Furthermore, I have got propagated your blog within our bebo.

    • blog sites 1 month ago   Reply

      My partner and I stumbled over here from a different web page and thought I should check things out.
      I like what I see so i am just following you.
      Look forward to looking at your web page for a second time.

    • study french online 1 month ago   Reply

      wonderful points altogether, you just won a new reader.
      What could you suggest about your post that you simply made a few days in the past?
      Any sure?

    • Amazing issues here. I am very happy to look your article.
      Thank you so much and I’m looking forward to contact you.
      Will you please drop me a e-mail?

    • Hi to all, as I am really eager of reading this blog’s post to be updated daily.

      It contains pleasant data.

    • jeu pour party magog 4 weeks ago   Reply

      It’s difficult to find knowledgeable people in this particular subject, but you seem
      like you know what you’re talking about! Thanks

    • Hey There. I found your blog using msn. This is a really well written article.
      I’ll be sure to bookmark it and return to read more
      of your useful info. Thanks for the post. I will definitely return.

    • What’s Taking place i am new to this, I stumbled upon this I
      have discovered It absolutely useful and it has helped me out loads.
      I’m hoping to give a contribution & assist different customers
      like its helped me. Good job.

    • tumblr.com 3 weeks ago   Reply

      What i don’t realize is in fact how you are now not really a lot more
      smartly-preferred than you may be now. You’re very intelligent.
      You already know thus significantly relating to this topic, produced me
      individually believe it from a lot of varied angles.
      Its like men and women are not fascinated unless it’s one
      thing to accomplish with Girl gaga! Your individual stuffs excellent.
      At all times maintain it up!

    • Hello there! This is my first visit to your blog! We are a group of volunteers and starting
      a new project in a community in the same niche. Your
      blog provided us beneficial information to work on.
      You have done a marvellous job!

  4. my friend, that is an good post.

  5. estudiar online 9 months ago   Reply

    A low monthly membership fee gives students unlimited access to the question-based clep study courses on the website.
    The technology has been planned with the understanding
    that every child has a different learning style
    and ability of learning. 2 million students were enrolled in online courses during 2007.

    • online blogging sites 1 month ago   Reply

      Hello to all, because I am actually keen of reading this website’s post to be updated on a regular basis.
      It consists of nice information.

    • free online blog 1 month ago   Reply

      Admiring the time and energy you put into your site and detailed information you present.
      It’s nice to come across a blog every once in a while that isn’t the same old rehashed material.
      Excellent read! I’ve bookmarked your site and I’m including your RSS feeds to my Google account.

    • Hi there! Quick question that’s totally off
      topic. Do you know how to make your site mobile friendly?
      My weblog looks weird when viewing from my iphone4. I’m trying
      to find a template or plugin that might be able to correct this issue.
      If you have any recommendations, please share. Cheers!

    • This is my first time pay a quick visit at here and i
      am truly impressed to read all at single place.

    • Great goods from you, man. I’ve understand your stuff previous to and you’re
      just too magnificent. I actually like what you’ve acquired here, really
      like what you’re stating and the way in which you
      say it. You make it entertaining and you still take care of to keep it sensible.
      I can’t wait to read far more from you. This is really a wonderful web
      site.

    • In fact when someone doesn’t know afterward its up to other viewers that they will assist, so here it occurs.

    • roof installation costs 3 weeks ago   Reply

      Hello There. I found your blog using msn. This is a very well written article.
      I’ll be sure to bookmark it and return to read more of your useful info.
      Thanks for the post. I will definitely return.

    • party de noel bromont 2 weeks ago   Reply

      You actually make it appear really easy with your presentation however I in finding this matter to be really one thing that I
      believe I would never understand. It seems too complex and very broad for
      me. I am looking forward for your next submit, I’ll attempt to get the hold of it!

    • this old house roofing 1 week ago   Reply

      Hey there I am so grateful I found your web
      site, I really found you by mistake, while I was searching on Aol for something else, Nonetheless I am here
      now and would just like to say cheers for a tremendous post and a all round enjoyable
      blog (I also love the theme/design), I don’t have time to look over it all at the moment but I have book-marked it and also added in your RSS
      feeds, so when I have time I will be back to read a lot more, Please do keep up the great
      job.

    • Hi, after reading this remarkable piece of writing i am too
      glad to share my knowledge here with friends.

    • We are a gaggle of volunteers and starting a new scheme in our community.
      Your site offered us with useful information to work on. You
      have performed an impressive job and our entire group will be thankful to you.

    • soirée casino estrie 2 days ago   Reply

      Because the admin of this site is working, no uncertainty very
      rapidly it will be famous, due to its feature contents.

  6. kolumne 9 months ago   Reply

    Hi there!

    This is a good post Germans Boycott University Rankings. It seems online copy had far surpassed print news. Even proffesional journalists can’t make it up to what the web writers can provide in terms of informativeness, and conciseness. Reading current affairs online has completely replaced newspaper news for me.

    Best,
    Carrejo Starns

  7. Dionysios Gouvias 7 months ago   Reply

    University ranking is becoming a commonplace even in my country Greece. Its HE institution are undergoing a radical overhaul, which is based on ‘external assessment’, combined with drastic cuts in public funding and closing down of whole departments (or amalgamation between existing ones). The latter, however, have not (so far) emerged as a result of ‘external assessment’ reports, but rather as an arbitrary decision by the Greek government (don’t forget that Greece is under the Troika’s financial grip!), which disclosed its decision at the end of summer 2012, and collected data (…) within a time-limit of less than 2 months!!! The so-called ‘consultation’ with the HE institutions and their representatives was a total sham, since the government rejected all the proposals put forward by the former! Instead they invented ‘quality-related’ indicators and algorithms that judge a ratio of faculty/adminstrative personnel of 1:60 as one of ‘high quality’!!!… Following that criteria, they put on temporary ‘suspension’ around 1350 administrative persons at the largest Greek Universities of the country. Of those, very few would be able to ‘reassigned’ to another post in the public sector!…

    These developments are very interesting for academics around the world, and the afomentioned situation (of Germany or Greece) is not unique in a performance-based accountability-driven reform wave that sweeps across the European –and only– continent. You can read a relevant story about the Danish universities, at: http://www.universityworldnews.com/article.php?story=20140124130958826 (Over the Christmas period Morten Østergaard, Denmark’s minister for science, innovation and higher education, told a leading newspaper there were far too many courses at universities and the number would be significantly cut…).

    Dionysios Gouvias
    Assistant Professor in Education Policy.
    Department of Pre-school Education and Educational Policy,
    University of the Aegean.

  8. It’s hard to find well-informed people for this topic,
    but you seem like you know what you’re talking about!
    Thanks

    • our website 2 months ago   Reply

      I’m really enjoying the design and layout of your website.
      It’s a very easy on the eyes which makes it much more pleasant for me to come here and visit more
      often. Did you hire out a designer to create your theme?
      Exceptional work!

    • My spouse and I stumbled over here by a different page and thought I might check things out.

      I like what I see so now i am following you. Look forward
      to checking out your web page yet again.

  9. コヨーテから始まった幻覚はまだ続いているらしい。マンションの前にあるはずの道路はない。歩道橋も公園も、公園脇にある地下鉄出口も、そして道路を隔てた向こうに広がる中層の団地も、何もない。平らな土の上に、茶色に縮れたセイタカ

  10. 663 1 month ago   Reply

    materials manufacture, provides the taste is also diverse, is most likely the overseas land taste Congo leather, the tough Scotchgrain leather, and top quality exquisite British Matt the glove

  11. access-support.com 1 month ago   Reply

    Hi, i think tat i saww you visitged my site so i came tto “return the favor”.I am trying to find things
    to enhance my website!I suppose its ok to use
    some of your ideas!!

  12. fountain of youth 3 weeks ago   Reply

    I am sure this post has touched alll the
    internet viewers, its really really nice post on building
    up new webpage.

  13. stopthesound.com 3 weeks ago   Reply

    At this moment I aam going away to do my breakfast,
    later than having my breakfast coming over again to rread additional news.

    GSI Great Southern Insulation
    Toll Freee Call: 1-800-763-1844
    Local Phone: 954-763-1844
    Or fax uus at: 954-763-1873

    Our office iis located at:

    GSI Corp.
    923 S.E. 20 Street, Suite D2
    Forrt Lauderdale, FL 33316
    stopthesound

  14. google.com 2 weeks ago   Reply

    Thanks designed for sharing such a fastidious idea, article
    is nice, thats why i have read it fully

Trackbacks For This Post

  1. Ninth Level Ireland » Blog Archive » Germans Boycott University Rankings - 1 year ago

    [...] “The results of the CHE (Centre for Higher Education Development) University Ranking, a subject-level classification covering a range of academic disciplines, have been published each spring since 1998 …” (more) [...]

  2. Bernard Rentier, Recteur » Pourquoi les « Rankings » m’exaspèrent - 1 year ago

    [...] non anglophones, incontestablement dédaignées par tous les classements mondiaux, et un appel au boycott dans différents pays, on a assisté à des réactions visant à tenter de remonter dans les [...]

Leave a Reply