Australia's national health insurance system, Medicare, had a difficult birth. Under the leadership of Gough Whitlam, the Labor Party went to the 1972 election with a commitment to "a universal health insurance system". Yet, once the Whitlam government had been elected, the Senate, still effectively controlled by parties on the right, refused to pass the bills to implement it.

A monumental struggle ensued. The only way open for the government was to present the bills to the Senate a second time and, after a second rejection, to call another election and subject the bills to an unprecedented joint sitting of both Houses of Parliament.

This was a high-risk move. Even though Labor had won a strong majority of the vote in 1972, a system of unevenly sized electorates favouring the Coalition had kept Labor out of office in three previous elections where it had won the popular vote. But Whitlam took that risk, going into the 1974 election (just 17 months after the 1972 election) with a slim nine-seat majority and facing intense opposition from private health insurers and the Australian Medical Association.

Even after it won that subsequent election, the Coalition parties mounted a High Court challenge, questioning the legitimacy of a joint sitting. Once the High Court had cleared the way, legislation setting up Medibank (the forerunner of Medicare) was passed (along with electoral reform bills).

The easy path for the Whitlam government would have been to give up at the Senate's first rejection, keeping faith with the electorate by saying it tried. But that was not the way Whitlam or his most trusted ministers worked.

Whatever views people hold on Whitlam's policies, there is broad agreement that he took risks in pushing his reformist agenda – "crash or crash through" to use a term popular at the time. His administration did crash spectacularly in 1975, but even his bitterest ideological enemies admitted he left an indelible mark on Australian politics and on society as a whole. When Whitlam died in 2014, journalist Mark Kenny wrote in his eulogy "there are worse things than losing power, such as not using it while you're there".

Kenny's observation is a reminder of the work of Harvard psychologist David McClelland, who wrote about two motivations driving people to seek political office. Some seek high office as an end in itself, enjoying the prestige and recognition, and will do everything to cling to it. Others, however, are motivated to pursue an agenda. Whitlam clearly belonged to the second camp.

We are apt to be cynical about politicians' motives, assuming they will do anything to stay in office, yielding to petulant interest groups and not daring to incur short-term political costs to achieve long-term benefits – benefits that may accrue to governments of other persuasions. There is certainly confirming evidence, such as the Gillard government's capitulation to mining interests in watering down the minerals resource rent tax, and to licensed clubs in rejecting gambling reforms. More recently, the Abbott government wasted no time in attending to the interests of the coal and financial services industries, rapidly moving to abolish the mining tax, revert carbon pricing and restore the ability of financial advisers to cream off hidden commissions.

But there are counter-examples, where governments have been far more gutsy. In the 1980s the Hawke government introduced compulsory superannuation, believing it would take 40 or more years to relieve pressure on the budget for pensions. In the late 1990s, the Howard government, which had already shown courage by introducing strong gun control laws in its first term, took a high risk in introducing a goods and services tax, and only just scraped back to office in the following election.

Perhaps the most famous counter-example in Australian politics, ranking alongside America's Sam Houston, was South Australian premier Raymond Steele Hall, who, having won the 1968 state election, went on to demolish the electoral formula that had kept his conservative party in office for so many years – a formula that had effectively given country voters twice the representation of Adelaide voters. His reforms meant certain electoral suicide on a point of principle: his government was voted out in 1970.

It is in no way inevitable that governments become captured by business interests, or that they go to water in the face of well-funded campaigns. Indeed, reformist-minded governments may well gain respect if they are seen to be standing up for the public interest against sectoral interests.

If we sit back passively observing politics as if we are watching Yes Minister or The West Wing on television; if we nod in agreement when radio shock jocks or partisan journalists make outrageous claims about government; if we confine our political interests to personality clashes and petty scandals used as fillers for tabloids, then it's little wonder that the big end of town takes over the political agenda. To lose faith in democracy is to lose faith in ourselves. If we can't be trusted to use our hard-won freedoms to hold democratically elected governments to account, is there any reason to think that we would be more successful in holding corporate power in check?

☆ ☆ ☆ ☆ ☆

Just as corrosive to the public interest is a belief that governments are intrinsically incompetent – a belief that whatever the government does, even if it is well intentioned, the private sector will always do it better.

Markets work well in some situations but not in others. Where markets fail, it's not usually a question of greed or incompetence: we make no claim that those who work in the public sector are more virtuous or more competent than those who work in the private sector. We are all cut from the same cloth. It's about the limits of the competitive market model.

In a work for Britain's National Endowment for Science, Technology and the Arts, Will Hutton and Philippe Schneider referred to "the assumption that government efforts to correct market failures will only magnify them. In this world view, governments and regulators are blind giants, all chubby thumbs and slight fingers, incapable of eliciting common underlying information about market needs, or of taking efficient action even when they do. They are risk-averse, bureaucratic, overmanned and never deliver their intended outcomes. They are vulnerable to influence and capture by powerful organised groups, so that the benefits they distribute are skewed. One of the reasons for persevering with market solutions is because government 'solutions' are no solution at all."

It is not hard to find public scandals that reinforce such a view of the relative competence of government and of markets. By design, democratic governments are more accountable and transparent than businesses. Investigative journalists looking to uncover waste or maladministration have a lot more access to the public sector than to the private sector. And apart from the occasional story about a child who slips through the net of child protection services or a "horror intersection" on a highway, there is very little reporting on areas where governments failed to act, and even less on how revenue may be raised to fund better child protection or highways. Tax rises are usually greeted by front-page headlines in alarmingly large font and sympathetic profiles of the impact on a family with 1.9 kids and a quarter-acre block somewhere in middle Australia. By contrast, carefully considered articles on the need for higher taxes are typically relegated to the opinion pages.

The nuanced reports of official auditors and independent analysts on controversial incidents get even less attention, for three reasons. First, they are generally released long after the problem that made the story "hot". Second, they often reveal that a particular failure was exceptional, not systemic. And third, most audit reports simply find that public servants have been generally doing their work competently and effectively. Hardly the stuff of tabloid headlines. When audit reports find that waste has occurred because functions have been inappropriately privatised, as was the case when the Commonwealth decided to sell and rent back its real estate, media outlets tend not to run the stories.

A recent example of reporting that put the government in the worst possible light was coverage of the homeowner insulation program. This hastily developed scheme was intended to simultaneously reduce carbon dioxide emissions and boost employment after the 2008 global financial crisis. A scandal arose when four workers employed by insulation contractors were electrocuted on the job, and when subsequent fires were attributed to badly installed insulation. In the face of the overwhelmingly negative media treatment and fierce attacks in Parliament, the government abandoned the scheme.

The popular perception was that it had been an unmitigated "debacle" (the description repeatedly used by an ABC journalist), and that these deaths were the direct responsibility of the government.

The official report from the Auditor-General, released months later on the basis of far more comprehensive analysis than there was space for in the newspapers, criticised some features of the scheme, but was much more measured. It identified a lack of basic technical knowledge on the part of its administrators, a lack of knowledge of the industry (the public servants running the scheme were almost all stationed in Canberra), a failure to learn from similar programs in other jurisdictions, and a payment system that encouraged fraud and did not prevent poor quality and dangerous work. It noted that the fires and deaths of installers were "associated with installations under the program" but it did not blame the government for them. Later research revealed that the rate of fires and deaths while the program was in operation, despite the presence of some disreputable practitioners, was actually lower than the normal rate in the industry.

Certainly, installing foil insulation in houses with poor wiring is a dangerous trade that shouldn't be done by untrained people. But then, if an inexperienced driver dies on a public road, we don't blame the government for building the road. The real story is more complex than a government bungle. The main message from the Auditor-General was that the government had put too much faith in an industry subject to too little government regulation. We could take away the message that governments should, if anything, be more actively involved in such programs. The scheme had been exploited by lax suppliers, bringing disrepute to the industry and government simultaneously.

The insulation scheme reminds us that we need to be careful evaluating all of government based on journalists' gut reactions to an alleged debacle. Looking at the bigger picture, as engaged citizens, we must ask how widespread and severe are problems of mismanagement in public administration. Were the problems of the home insulation scheme peculiar to that situation or do they indicate system-wide problems? If so, are these problems intrinsic to the public sector? Could the private sector have done better? And, more broadly, what would have been the consequences if the government had not responded to the financial crisis with practical stimulus programs? The stimulus package managed to contain the post-financial crisis rise in unemployment to less than two percentage points. Had the jobless rate risen by three or four percentage points, as it did in countries that did not respond so rapidly, another 200,000 or 300,000 Australians would have become unemployed.

Australians get good value-for-money from their governments – federal state and local. There are some specific management shortcomings in government in Australia, but there is nothing to suggest they are intrinsic to the public sector.

Ian McAuley is a former public servant and lecturer in public sector financial management. Miriam Lyons was co-founded and executive director of the Centre for Policy Development. This is an extract from their book Governomics: Can We Afford Small Government? (Melbourne University Publishing, May 2015. RRP: $32.99)