It's a trifle difficult, perhaps, to get terribly worked up these days about whether or not there was a ''Fifth Man.'' You remember: a fifth Cambridge- educated, highly placed Soviet ''mole'' in British Intelligence to go along with wild Guy Burgess, neurotic Donald Maclean, daring Kim Philby (the last to defect, in '63), and unflappable Sir Anthony Blunt, who secretly confessed in 1964 and was unmasked but barely punished in '79.
Still, for those inclined to care, first-time novelist Cape (himself a Cambridge grad) churns up a serviceable, if overblown, fuss with The Cambridge Theorem. He moves the action back to 1981, before glasnost threatened to defrost Cold War spy fiction. He makes the Fifth Man a Nobel Prize-winning scientist, still cozily incognito, still capable of damaging the interests of the United States and the United Kingdom. And he takes for his hero a rebellious Cambridge copper (and Willie Nelson fan) named Derek Smailes, who is stubbornly investigating the ''suicide'' of an eccentric student obsessed with exposing the Fifth Man. Lots of drawbacks here including a sluggish pace and a headache- inducing wrap-up but enough sense of place and character to lull Anglophiles into semi-submission. B-