Sir Drummond is a specialist on the works of Romantic poet Lord Byron, and on leaving Oxford in 1972 became lecturer in English and Comparative Literature at the University of Warwick. He returned to Glasgow in 1980 as a lecturer in English Literature, becoming Senior Lecturer in 1989 and titular Professor in 1995. From 1991 to 1995, he was Dean of the Faculty of Arts, and in 1995 became Vice-Principal.
In 2000, he left Glasgow to become Principal of Royal Holloway, University of London and Pro-Vice-Chancellor of the University of London, and in 2002 became Vice Chancellor of the University of Liverpool. Although head of the University, he continued to teach an undergraduate class on Byron, and was also chairman of the Liverpool Culture Company, directing the city's preparations to be European Capital of Culture in 2008. He also served as President of Universities UK, a committee of university heads, from 2005 to 2007. He retired from Liverpool in 2008, and from October 2011 became Master of Balliol College, Oxford.
He is an acknowledged expert on Lord Byron's work and is Vice-President of the Byron Society. He was editor of The Byron Journal from 1978 to 1988 and has been co-editor of journal Romanticism since 1995. He is a member of the Steering Group of the Council for College and University English, a Fellow of the English Association, and an elected Fellow of the Royal Society of the Arts (1995) and of the Royal Society of Edinburgh (2008).