And yet we need to recognise that despite these very real problems, the trajectory of higher education as an institution is utterly different from the one these bleak pictures convey. Beyond the din of the latest protest about sexual violence on campus or the latest controversial speaker whose mere presence on university premises provokes an uproar, some remarkably positive trends have left American universities much bigger and stronger, and in a more dominant position – both domestically and internationally – than ever before.

The success of US higher education goes far beyond the two dozen American universities that dominate the world tables. The vast majority of the top 200 research universities are stronger than ever, and the system as a whole has shown an amazing resilience through recession and expansion alike. In my new book, Two Cheers for Higher Education: Why American Universities are Stronger 
than Ever – And How to Meet the Challenges They Face, I have traced the contours of American higher education from 1980 to the present – and, despite the validity of some of the gloom and doom stories we see every day, a very different picture emerges. I seek to paint this picture not because I want to sweep the problems of higher education under the rug but instead to set these daily challenges in a broader – and frankly more positive – context.

American research universities have grown more vigorous over the past four decades, both financially and intellectually. They have done so by incorporating multiple growth logics in an interconnected and flexible way.

The traditional structures and purposes of colleges and universities are intended to produce two outcomes: the expansion of knowledge, principally in the disciplines but also at their interstices, and the development of students’ cognitive capacities and knowledge of subject matter. In the years after 1980, two movements hit colleges and universities with great force. One was the drive to use university research to advance economic development through the invention of new technologies with commercial potential. The other was the push to use colleges and universities as instruments of social inclusion, providing opportunities to previously marginalised groups, including women, racial-ethnic minorities, and members of the LGBTQ community. These moves were driven by external parties, such as the Business-Higher Education Forum and the great philanthropic foundations, and by campus constituencies that benefited from their advance.

My argument is that these movements created a special kind of dynamism because of the strength of partisan commitments to them, backed by high levels of patronage. The innovation movement fostered a stronger embrace of entrepreneurship; the rise of engineering and biomedicine as the two centres of exceptional dynamism in universities; new ideas about economic development related to partnership between universities, industry and government; and the creation of new high-tech clusters of firms surrounding universities. It also contributed to the growth of interdisciplinary initiatives on campus, as a result of the underlying assumption that the solution of technological problems required the skills of investigators from many disciplines.

The inclusion movement, meanwhile, fostered the expansion of the curriculum to include the experiences of marginalised peoples from the US and those from nonWestern cultures; commitments to the diversification of the student body and the faculty; attention to intergroup relations on campus as a measure of the new concept of “campus climate”; and interventions to help disadvantaged groups succeed. It, too, contributed to the growth of interdisciplinary initiatives as means to knit together networks of colleagues with common interests in diversity and social change.

The rise of these two dynamic forces created a contest in which the traditions of academic professionalism both accommodated and resisted pressures to shift attention towards technological innovation and social inclusion. The dean of the engineering school promoted colleagues who made fundamental advances in their specialty areas, but also encouraged those who worked with industry and sponsored programmes for minorities and women in engineering. The chair of the sociology department celebrated scholars who accumulated influence through the citation of their research, while at the same time seeking to diversify his faculty and graduate student body and adding new “self-supporting” master’s degree programmes in applied social statistics or human relations management.

Even as they accommodated the growing interest in use-inspired research, the majority of faculty researchers continued to focus on solving problems identified by colleagues in their disciplinary communities. And even as they accommodated the push for social inclusion, colleges and universities also found means to preserve their traditional role in the identification of talent, most often found in students from relatively advantaged backgrounds.

Occasional tensions arose, as when faculty entrepreneurs seemed to flout their academic responsibilities in favour of building their enterprises, or when the racial or gender backgrounds of candidates seemed to supersede their scholarly achievements as a basis for advancement. But accommodation was the norm.

The dynamic forces of technological innovation and social inclusion have not been the only fuel for expansion. Patronage was essential, just as it was when the Medici family financed the flowering of the arts and scholarship in 14th- and 15th-century Florence. Neither the renaissance of applied science nor the extensions of educational opportunity during the period would have occurred without multibillion-dollar investments by the federal government and the 50 states – the greatest of university patrons – or the multimillion-dollar investments by thousands of contemporary Medicis.

The 1950s and early 1960s are often described as a “golden age” of American universities because of the building spree of that era and the strong levels of state support for expansion. Scholars tend to deride the period after 1980 as a neoliberal disaster, but the evidence suggests otherwise. It could more accurately be considered a second golden age. Between 1980 and 2010, for example, research expenditures grew by more than nine times in constant 2010 dollars, publications catalogued in the Web of Science grew nearly fourfold, and Web of Science citations grew by at least 250 per cent.

Few sectors were as important to the emerging knowledge society as universities, and the federal government supported their development with high, if never fully sufficient, funding. Such funding, estimated at about $30 billion (£23.6 billion) in 2015, was crucial to the research activities and infrastructure on university campuses. So too was the financial aid system, which expended approximately $65 billion in grants, loans and indirect tax benefits by 2015. Both support systems have trended sharply upward in constant dollars since the 1980s, including during recessionary periods.

Measures of university impact show a similar picture. Universities do not hold a monopoly on knowledge production – far from it – but the research they produce has contributed to momentous improvements in human well-being. A decade ago, in his book The Great American University, former Columbia University provost Jonathan R. Cole provided an overview of the most fundamental of these contributions. Among those he highlighted included the gene-splicing technology of Herbert Boyer and Stanley Cohen that led directly to the creation of a multibillion-dollar biotechnology industry and to drugs to treat heart disease, strokes, haemophilia, rheumatoid arthritis, thyroid cancer, asthma, non-Hodgkin’s lymphoma and diabetes, among others. Other biomedical discoveries discussed by Cole have led to new ways to suppress cancer tumours, to prevent smoking, to replace broken or damaged joints, to improve hearing through cochlear implants, to allow damaged hearts to beat regularly, and to detect previously undetectable bodily ailments through magnetic resonance imaging.

The big picture: US higher education research funding and number of degrees awarded

In the physical sciences the development of lasers has led to applications that range from eye surgery to audio CDs. Such familiar products as light-emitting diodes, bar codes, radar and the pacemaker were developed by university researchers. Many advances in information technology were also the product of university researchers, including the design of the first high-powered computer (which led to computer-aided design and manufacturing), the first web browsers, and packet-network switching, which created the architectural foundation for the internet. Another entirely new industry may be in the making as a result of university researchers’ breakthroughs in creating much more flexible and lighter materials through nanoscale technologies.

When I examined 50 of the most important inventions of our era chosen by a panel of experts, I found that academics made essential contributions to 40 per cent of them, an impressive achievement given that investments in academic R&D have run between 0.2 and 0.4 per cent of US GDP in recent years, and with no countries breaking the 1 per cent barrier. And life-altering new inventions come regularly, most recently in the gene-editing work of scientists at the University of California, Berkeley and the Massachusetts Institute of Technology and the quantum computing breakthroughs produced at Yale University.

The contributions of university researchers go well beyond these high-profile inventions. I examined highly cited work in the social sciences and found that many concepts that have worked their way into public consciousness – and some that have transformed practices – have their origins in these publications. They include emotional intelligence, self-efficacy, social capital, transaction costs, relationship-based (as opposed to principle-based) moral reasoning, network embeddedness, communities of practice and stakeholders (as opposed to shareholders).

In a study of people cited as authorities in New York Times reporting during a week in January 2017, I found that academics were mentioned most frequently. This is because of both the breadth of topics they investigate (they commented on everything from investments in robotics technology to the history of popular music) and because of the norms of careful and critical discourse that distinguish their work.

We can add to this sense of the university’s growing importance the weight of the tens of thousands of leaders it has helped prepare for positions of responsibility and the tens of millions it has helped equip for occupations requiring well-informed judgement and cognitive skills. When we consider the options to pursue more complete personal development, in terms of creativity and powers of reflection, our thoughts naturally turn to the time and challenges offered by higher study. In a society with few other avenues for social mobility, higher education is also the path that leads out of economic marginality for hundreds of thousands of young people every year.

These contributions would not have been possible without the strong demand for a college education among young people. College has become nearly a necessity in the minds of most Americans, the only good option for young people hoping to secure good jobs. The growth in both undergraduate and graduate enrolments was steady from the 1990s on, through recession and prosperity alike. Nor did the rate of increase slow in the face of rising tuition costs. One consequence of the larger population of baccalaureates is that postgraduate degrees have also become more common. Some 25 million Americans hold advanced degrees (master’s and above), the combined size of the five largest US cities; doctorate degree holders alone could populate Los Angeles.

The large majority of these highly educated people – 85 per cent – move into “knowledge sector” industries, such as aeronautics, entertainment, finance and internet services. If you define a knowledge-sector industry as one in which 10 per cent of employees have postgraduate degrees, then the four dozen major knowledge-sector industries now account for as much as 50 per cent of US gross domestic product, a contribution that has risen steadily since 1980. The US cannot be accurately characterised as a post-industrial society, but large swathes of its economy can be characterised in that way.

Recognising the strength of US universities does not mean minimising their problems. A phase shift in the quality of undergraduate teaching and learning will be required to reach the higher levels of achievement that are within reach. Too many college classrooms remain mired in a style of instruction that tests students’ capacity to memorise presented material, and little else. College classrooms will need to become research-centred places where students are expected to bring evidence to bear to evaluate arguments; where they are presented with information that allows them to explore hypotheses; where they are required to make their arguments both in writing and orally; and where detailed feedback is given.

Cost is the other fundamental concern. Investment in higher education by the 50 states has shown declines during recessionary periods and only partial restorations thereafter, and universities have had to rely on tuition increases (and enrolling more international and out-of-state students) to make up the balance. If states’ disinclination to invest in higher education means tuition cannot be reduced, the second-best scenario is to marshal federal, state and institutional support for low-income students, and to use merit-based scholarships very sparingly. Most students have not accumulated unmanageable debt, and college remains a good investment for the great majority, repaying many times over the costs of loans. However, repayment at a time when earnings are low is a burden for many students who have taken degrees in less remunerative fields or who have had problems finding well-paying jobs. Well-designed and easy-to-use income-contingent loans could provide a sensible approach to the student debt problem.

Universities can and should intensify their contributions to society by building stronger bridges to both private firms and lower-income households. But as higher education institutions draw closer to the new engines of growth, it will be more important than ever for scientists and scholars to maintain sentry duty over the two most important aspects of what universities offer: teaching that adds substantial value to students’ cognitive development and rigorously reviewed, illuminating research in all fields of formal and abstract knowledge. By doing so, professors will ensure that their institutions continue to follow the injunction that is at the heart of the academic mission: Let there be light!

Steven Brint is distinguished professor of sociology and public policy at the University of California, Riverside. Two Cheers for Higher Education: Why American Universities are Stronger than Ever – And How to Meet the Challenges They Face is published this month by Princeton University Press.