In the early 1990s, Senator Patrick Moynihan campaigned for the abolition of the CIA. The brilliant campaigner thought the US Department of State should take over its Intelligence functions. For him, the age of secrecy was over.

In a New York Times opinion piece, Moynihan wrote:

For 30 years the intelligence community systematically misinformed successive presidents as to the size and growth of the Soviet economy … Somehow our analysts had internalised a Soviet view of the world.

In the speech introducing his Abolition of the CIA bill in January 1995, Moynihan cited British author John le Carré’s scorn for the idea that the CIA had contributed to victory in the cold war against the Soviet Union of Leonid Brezhnev and his successors. “The Soviet Empire did not fall apart because the spooks had bugged the man’s room in the Kremlin or put broken glass in Mrs Brezhnev’s bath,” Le Carré had written.

This was one of the CIA’s lowest points since its establishment in 1947 (my new book marks the agency’s 75th anniversary). It was created with two key goals in mind: thwarting Soviet expansionism, and preventing another surprise attack like that carried out by the Japanese on Pearl Harbour during the second world war. While Moynihan’s campaign to shut down the CIA did not ultimately prevail, there was certainly a widespread perception that the agency was no longer fit for purpose and should be curtailed.

Throughout the cold war, many had regarded fighting communism as the CIA’s raison d’être. With the collapse of the Soviet Union, the agency’s role was less clear, and it came under heavy criticism for having distorted intelligence and “blatantly pandered” to one ideological viewpoint: blind anti-communism. Without the cold war, Moynihan predicted, the CIA would become “a kind of retirement programme for a cadre of cold warriors not really needed any longer”.

Three decades on, however, Vladimir Putin’s invasion of Ukraine has put Russia’s threat to the stability of the world back at the top of the US foreign agenda. With a formidable Kremlinologist now in charge of the CIA and Donald Trump out of the presidential picture (for the moment, at least), the agency might be expected to be an influential player in the US response to this “new cold war”. But how much does Washington trust the CIA these days – and how much influence does it really have on events in Ukraine? To shed light on these questions, we need to go back to the early days of the Ronald Reagan presidency.

‘Stay the f-ck out of my business’

As US president from 1981 to 1989, the neoconservative Reagan unleashed the CIA from restrictions that had been imposed on it during the reforming post-Vietnam 1970s.

Like other anti-communists, Reagan saw the agency as a prime weapon in weakening the Soviet Union, which he famously denounced as the “evil empire”, and preventing the worldwide spread of communism. The new US president was convinced that in opposing an unethical foe, one could not afford to be too scrupulous. He chose as his CIA director Bill Casey, a veteran of intelligence in the second world war – a time when it had been “gloves off” for dirty tricksters.

An outright cold warrior, Casey resuscitated old CIA habits, running covert operations against the left-leaning – but democratically elected – Sandinista government in Nicaragua from December 1981 to the ceasefire of March 1988. Even the veteran conservative senator Barry Goldwater admitted he was “pissed off” when, in 1984, the CIA mined Nicaragua’s harbours without informing Congress. Accosted with this oversight, the uncompromising Casey replied: “The business of Congress is to stay the fuck out of my business.”

The CIA worked closely with the Contras, right-wing terrorists who sought to overthrow the Sandinista government. The agency trained these guerrillas in secret camps in adjacent countries and organised munition drops from planes stationed in clandestine bases. In one initiative, a contracted CIA operative wrote a manual for the Contras explaining how to assassinate individuals on one’s own side – skulls had to be fractured in just the right way – and then blame the enemy.

A disapproving US Congress banned these weapons drops and cut off the necessary funds. To get around this, arms were illegally supplied to Iran (then at war with Iraq) via Israel – paid for by covert Iranian financial assistance to the Contras. However, fearing the wrath of Congress should this ruse be discovered (as it later was), the Reagan administration bypassed the CIA in administering the Iran-Contra scam. While the president had not lost confidence in the agency, this was a sign that the CIA was becoming increasingly toxic in the eyes of Congress – making it too risky to deploy its spooks in the customary manner.

On the threat posed by the Soviet Union, though, there was far greater accord. CIA director Casey lined up with the secretary of defence, Caspar Weinberger, and the majority of Reagan’s cabinet in adopting an intransigent stance towards Moscow. They were supported by the CIA’s senior Russia expert, Bob Gates, who having gained his PhD in Russian affairs without ever visiting the country, proclaimed that the Soviet Union was an example of “oriental despotism”.

A keen boy scout in his youth, Gates – whether out of conviction or career calculation – glued himself to the American flag and offered no challenge to any president who wanted to play up the Moscow menace. Under Reagan, Casey and Gates, the CIA worked tirelessly to undermine the Soviet Union – secretly supporting Poland’s opposition movement Solidarity, and engaging in acts of economic sabotage against the Soviet economy.

Indeed, according to Republican partisans who argued that President Reagan won the cold war (the “victory thesis”), the US launched its Strategic Defense Initiative (SDI or “Star Wars”) with the aim of forcing Moscow to respond, thus ruining the Soviet economy and bringing about the collapse of communism. SDI was a multi-billion-dollar space defence system designed to intercept and destroy incoming enemy missiles. According to the victory thesis, Gates’ exaggerated estimates of Soviet military might were not an instance of unthinking anti-communism but rather, a cunning ploy designed to persuade Congress to fund the Star Wars bluff.

Gates would go on to lead the CIA from 1991-93, the years when Senator Moynihan was campaigning for its abolition. The Senate confirmation hearings that preceded Gates’ tenure would be the occasion for some bitter denunciations from erstwhile colleagues. Gates later recalled that these charges of 1980s intelligence distortion “truly imperilled my confirmation”.

Jennifer Lynn Gaudemans, who in 1989 had left the CIA’s Office of Soviet Analysis (Sova) in a disillusioned state of mind, accused Gates of seeing Soviet conspiracies around every corner, and of “blatantly pandering to one ideological viewpoint”.

At the Senate hearings, Gaudemans testified that Sova analysts were deeply upset when Gates suppressed their findings that the Soviet Union was not, in fact, orchestrating mischief in Iran, Libya and Syria. She claimed he had denied them even the opportunity to publish dissenting footnotes. Sova division chiefs were, she said, routinely dismissed for being “too soft” on issues such as Soviet policy in the developing world, and arms control.

But while the agency’s analysts had problems with Gates, more powerful individuals – not least, the US secretary of state George Shultz – were prepared to listen. Sova-generated data and findings made their way on to the desks of US negotiators.

On November 18 1985, the eve of Reagan’s summit meeting with Soviet leader Mikhail Gorbachev in Geneva, the president and his negotiators received an intelligence assessment to the effect that, while Gorbachev was repairing the economic damage of the Brezhnev era, he would not meet his growth targets. Because of this and the acute nationalist discontent in Poland, CIA analysts told Reagan that Gorbachev was ready to deal with the US.

Through such insights, the agency played an important role in ending the “old” cold war, culminating in the dissolution of the 

Intelligence to please

A decade later, the US’s confident post-cold war demeanour changed at a stroke when two hijacked planes flew into the World Trade Center on September 11, 2001. And the CIA would be the fall guy.

The attack masterminded by Osama bin Laden glaringly exposed the CIA’s inability to uphold its founding mission of preventing another Pearl Harbour-style attack on the US. Under renewed pressure to justify its existence, the agency succumbed to the demands of the George W Bush administration in the “war on terror” that arose from the ashes of 9/11.

As the US government desperately sought a rationale for invading Iraq, a deal was struck. Senior leaders of the agency may squirm at the charge, but the CIA supplied intelligence to please in exchange for the right to survive. Its leadership endorsed the mythical charge that Iraq was developing weapons of mass destruction (WMD). And when the ensuing war was a disaster, the CIA took the hit for having delivered that faulty intelligence.

Even in the early days of the Iraq war, however, the Intelligence Reform and Terrorism Prevention Act of 2004 had already stripped the agency of its central role in evaluating intelligence, handing the job to a new and independent director of national intelligence, John Negroponte.

With the role of the CIA thus diminished, the US intelligence community became an unresolved puzzle. Demoralised CIA personnel threw up their hands in despair. CIA veteran Art Hulnick, now teaching intelligence studies at Boston University, was at a loss to explain to his students the new arrangements for analysing intelligence. Hulnick complained of an overreaction to what he termed the “threat du jour”.

Resources were being poured into the huge and unwieldy Department of Homeland Security; the Department of Defence was poaching assets from the CIA; and the agency had even lost its monopoly on preparing the president’s daily briefing (the first item on the president’s desk each morning, memorably described by Michelle Obama as the “death, destruction and horrible things book”.)

By the mid-2000s, intelligence work was being heavily outsourced to private businesses in accordance with the ideology of the George W Bush administration. Private recruiters such as Blackwater were appearing at the CIA HQ’s cafeteria in Langley, Virginia, hiring personnel with promises of big salary increases before sometimes subcontracting them back to the agency at inflated rates.

The CIA had never been a fainting lily but now, in the interests of its own survival, its directors agreed to engage in unsavoury practices including torture, illegal kidnapping, and execution-by-drone without trial. Waterboarding, whereby water is poured over a cloth on the victim’s face to produce a sensation of drowning, was a common practice in the agency’s “dark sites” – secret interrogation centres in Poland, Egypt and other countries around the world where kidnapped suspects were held.

Investigative journalism and persistently curious congressional committees are staples of American democracy, and these dubious practices were bound to come to light – with the aid of whistleblowers such as Edward Snowden. Snowden had worked for the CIA as a highly regarded computer security expert before moving to a private subcontractor engaged by the US foreign signals intelligence organisation, the National Security Agency (NSA).

In 2013, Snowden leaked numerous files to the Guardian and Washington Post before fleeing to Russia in order to evade rendition by the CIA. His revelations about US internal surveillance practices infuriated the guardians of America’s secrets, and fed the fears of those who deplored the use of dirty tricks abroad – and the development of a “secret state” at home. Snowden was accused of having revealed the identities of CIA personnel on active duty to the possible detriment of their safety – a form of treason (should it be proved) that was a deeply sensitive matter within CIA headquarters. It was fortunate for the agency, though, that the main thrust of Snowden’s revelations was about the NSA’s role in global surveillance.

An end to CIA ‘groupthink’

By 2007, while the Iraq war grew mired, the Bush administration was talking loudly about another familiar Middle Eastern foe: Iran.

In 1953, the CIA had conspired to overthrow the country’s democratically elected but mildly leftist government headed by Mohammad Mossadegh. There followed a period of despotic royal rule by the last shah of Iran, Mohammad Reza Pahlavi. His overthrow in 1979 saw a period of priestly mullah rule a