David Baddiel was six years old when his mother told him death was like a long sleep from which you never wake up. “I think from that point,” he says, “I never really wanted to go to sleep again.” That night, he lay on the top bunk of his bed, fervently praying – “probably” the first and last time he has prayed with any sincerity – that “my life as it was in Dollis Hill in 1971 would still somehow continue after death”.

More than half a century later Baddiel is still an insomniac, and he’s still terrified by the prospect of dying. “I don’t quite believe anyone who says they’re not,” he says. That childhood memory, and that conviction, is what kicks off his latest book, The God Desire, which delivers in a brisk 110-odd pages what Baddiel considers “an absolutely slam-dunk argument” against the existence of God.

That sounds hilariously hubristic, but Baddiel very fervently wishes he was wrong. The fact that belief in God is a readymade cure for the fear of death (and the sense of human insignificance, but mostly the fear of death) is the heart of his argument. It’s exactly how badly we want God to exist, he suggests, that makes it a racing certainty we’ve made Him up. God is, so to speak, too good to be true.

Yet it troubles and surprises him that favourite authors including John Updike, whose work he “worships”, and others he considers intellectual peers such as his friend Frank Skinner, a Catholic, could be believers. He describes being astonished at Skinner’s conviction that he would burn in hellfire for living with his girlfriend after his divorce: “I had not recognised, not in any visceral way, what that meant for him. That was my own failing.” In some ways the book feels like an attempt to understand the phenomenon as much as to rebut it.

He doesn’t spend much time wrestling with Aquinas, Kant, Gödel and co either, dismissing many gnarly points of logical debate as “late-night, sixth-form” dead ends and saying it’s “pointless on both sides” to use logic and reason to argue about a being that supposedly transcends them. Did he not feel a bit intimidated, though, by the fact that some of the greatest thinkers in human history have spent entire careers hacking through the weeds on this one?

“Well, no. I mean, I’ve read The God Delusion, I’ve read John Gray’s Seven Types of Atheism. When I couldn’t sleep, I was listening to a The Rest Is History [podcast hosted by Tom Holland and Dominic Sandbrook] about the Enlightenment, and they were talking about Voltaire being the first atheist who wrote properly about atheism, and I did think: ‘Hmm, I haven’t really read Voltaire and I’ve written a book about atheism. That’s probably shit … ’ But as far as I’m concerned, if it’s readable and accessible and makes people intellectually entertained for however long it is, I don’t care that much that I clearly haven’t read the huge tracts on this elsewhere. But yeah, you’re right: there is some chutzpah in it.”

Still, there’s also humility – because he’s not saying human beings have no need for God. In this he differentiates himself from the “Billy big-bollocks” swagger of the noughties new atheists – Sam Harris, Daniel Dennett, Christopher Hitchens and Richard Dawkins – who rejected not only the truth-claims of religion but its consolations. They dismiss as childish the longing for a cosmic parent, but Baddiel writes: “I’m happy to admit to my own babyishness.” He’d love an omnipotent being to take a personal interest in his life and assure him that he has more to look forward to than a meaningless death and a yawning infinity of extinction. That’s where The God Desire connects to his previous book, his sinuously argued and erudite essay on the left’s blind spot for antisemitism, Jews Don’t Count.

“I did notice that most well-known atheists seem to be not members of a minority,” he says. “A lot of ethnic minority experience is associated with religion. In the middle of the book, I try and disentangle what that means. This is partly why my tone towards religion isn’t dismissive, like I think Richard Dawkins is dismissive, because I know it’s part of identity. And being dismissive of identity, especially now, is kind of stupid; and also just uncomplex, in terms of understanding what humanity is.”

The most moving aspect of the book, then, is the discussion of his own relationship to Judaism. A central concern of Jews Don’t Count was to bat away the canard that antisemitism is religious intolerance rather than racism – as he points out, the Gestapo didn’t spare Jewish atheists – and in that book he made a point of talking about Jewishness rather than Judaism. His childhood, though, contained the latter as well as the former – to the extent that he was able to write to Tom Stoppard correcting a mistake in Leopoldstadt, his play about 20th-century Jewry and the Holocaust, about why parsley is eaten in the Seder (it’s hard to tell whether he’s more cock-a-hoop at being friends with Tom Stoppard or at one-upping the great man on Judaica).

Baddiel’s parents weren’t observant. When it was time for the prayers his dad would say: “Can we get through the olly-wolly-polly and get on with the food?” His mother observed the Seder rituals “not out of a great sense of religion” but as “a family thing”. Her parents, who were refugees from the Nazis, used to have Seder nights at the Baddiel home until his father knocked it on the head: “He was pretty curmudgeonly. Eventually, he wouldn’t let those grandparents stay at our house: ‘They can go and stay at a fucking hotel.’ Which is not great of him, given that they had been refugees.”

Yet Baddiel and his two brothers were sent to the North West London Jewish Day School “because that was the nearest school that we could go to where we probably wouldn’t get beaten up for being Jewish … that meant that I went to a school where I learned Hebrew, I said blessings before every meal, I had to wear religious garb, and I was inculcated in a very Jewish way – which was weird, because I’d come home and they’d make me bacon sandwiches. Although it’s messed up, it’s a very central building block of my identity.” Baddiel wrestles with the question, then, of why as a self-described “militant atheist” he can be so moved by the words of the Kaddish, or find himself sobbing in his seat – even as other theatregoers “collected their coats and programmes” – after the end of Leopoldstadt.

“To explain what you mean by being an atheist Jew is complicated,” he says, “and I’m drawn to complexity. The book, I think, to some extent comes from trying to explain what that is.” On reading it, he says, Stoppard told him, “I’m really enjoying your conversation with yourself”, which is on-brand for Baddiel (and a bit on-brand for the courteous, sphinx-like Stoppard). He’s a great one for conversations with, and about, himself. Where “character comedians” hate being themselves on stage, Baddiel never tried to create a gap between his public image and his private one. Baddiel and Skinner Unplanned, he says, was an exercise in “let’s see how close as possible we can get to who we actually are on TV”.

But, as he observes, a public image always involves a series of misprisions. And in any case he contains multitudes, slaloming cheerfully between highbrow and lowbrow. His early work with Rob Newman set a million teenagers saying “You see that pair of pants? That’s you, that is” and his 90s partnership with Frank Skinner helped bring about the “New Lad”. But he has also written with grace and subtlety about David Foster Wallace and the Roth/Updike generation, created a feature film about a Jewish jihadi and a play about quantum physics, published comic novels and literary novels and children’s books, done standup shows he sees as halfway to Ted talks, and a documentary about his father’s dementia. Now, wearing his (in his phrase) “Mr Jew mantle”, he appears on heavyweight TV shows and publishes monographs in the TLS.

There is a “thread” connecting these things, he says. “I’ve only ever done things – and this is really true about me – that I am immediately engaged with. I don’t get asked to do that much. It still fucks me off sometimes: I’d quite like for someone to say, ‘Do you want to do this? Do you want a part in this?’ That virtually never happens to me. What happens is, I feel like I really want to talk about this, or I really want to write this joke.

“So to go back to why I wrote this book, it’s been me for some time, the belief that God is projection of desire. I feel that desire very strongly, and yet I’m an atheist. How do I make sense of that? That’s as much part of me as thinking it would be funny, wouldn’t it, if two old history professors spoke like I used to at school. They feel both like things that I think in the front of my head quite intensely.” Those front-of-head things come out in the most public way possible.

“I’ll tell you what I think it is, if you wanted a psychological explanation,” he says. “I do come from a sort of lower-middle-class immigrant background. And I think even though I am fine, and I could probably retire, a tiny part of me thinks: ‘No, no, I need to work and I’ve had an idea. And if I don’t do it, someone else will.’”

Does all this work, this frantic covering of the bases, also help stave off the death-fear? “I think it does. Although that leads to an interesting question at my time of life. Writing is hard, and spending all day doing it is hard. I don’t have all that much time left. Should I not be, y’know, travelling the world or having endless pampering or whatever … before I’m too old or too demented to appreciate it? But there is also the anxiety – a separate anxiety – of ‘No, but I still maybe have stuff to say’.”

Baddiel sees the “God desire” as manifesting itself in non-religious ways other than work. “Football fills a God-shaped hole, I think. Because it makes you feel connected to something besides yourself. It is, in a small way, eternal. If you’ve been going to Chelsea, as I have, for 40 years, you think: ‘I have watched players come and go and die. And I’m still here. And I feel connected to the a priori idea of Chelsea and football, which is sort of beyond the here and now. It’s identity, and it’s tribalism, and it’s opposition to other tribes. It feels very religious.”

I wonder, too, whether being famous scratches some of the itch for significance that he identifies as a root of the God desire. “It probably does,” he says. “I mean, people want to be famous to be rich, which I’m sure is in there – but I think more, people want to be noticed. And wanting to be noticed is definitely associated with the God Desire.

“At one point, I talk about God being the ultimate parent figure, because He is both providing and can sort out your life for you but He’s also sometimes angry and whatever. I used to say – and I’ve had a lot of therapy – that if you asked me why I felt the need to get up on stage, it would be because my mother’s favourite child was my younger brother, Dan. I don’t think I’m angry about that at all, but I think somewhere deep in me was a need to say: ‘But me! Over here, me, me! You’re not noticing me!’ That’s a psychoanalytical parental thing, but if you expand it, yeah: God definitely provides a witness. With fame, you feel witnessed.”

Not always witnessed in the way you’d like, though, and that’s perhaps one of the reasons that, for someone as rich and successful and accomplished as he is, Baddiel seems to sit uneasily with himself. Even though he sees how the non-existence of God could give you a carpe diem attitude, he’s “plagued by anxieties and weaknesses … that stop me Yolo-ing my way through the world”.

As a comedian, Baddiel has specialised in what we now call overshare. It’s all on the surface with him: the ego – he namedrops and mocks himself for namedropping – and its fragility alike. (It seems to me indicative, and endearingly guileless, that the first time we met in person he quoted verbatim from a sniffy review I’d written about one of his novels a decade previously. He mentions it again when we meet for this interview: apparently Stoppard liked the book, so there.)

Social media, to which he has been quite addicted and which both rewards and punishes overshare, has probably exacerbated this. He dedicated one standup show to his trolls, and another to fame itself, where he was funny about constantly being recognised and mistaken for someone else. Also, social media never forgets. After Jews Don’t Count came out, Twitter blew up with 90s footage from Fantasty Football League, the TV show he presented with Skinner, of Baddiel blacking up to mock the footballer Jason Lee. Baddiel acknowledges that that is terrible now and was terrible at the time, and went on Lee’s podcast to apologise in person, but his social media critics don’t see that as the end of it. He says he has “trained himself” not to look when he’s trending on Twitter.

“I don’t want to talk deeply about Jason Lee for a very specific reason,” he says now. Newspaper interviews with Lee on the day his film went out set social media going again, he says, and “in a way, it was problematic for me because I was very happy to do the interview with Jason and the apology, but I was worried that my film was about antisemitism; and I felt that if the papers and social media decided that the bit with Jason Lee was the main thing about it, what are they doing there? It’s suddenly not about Jews and antisemitism any more: it’s about another form of racism. Right?”

Since then, “Jason did a Facebook post in which he says he really wants to move on. He feels I’ve done the apology, that that was useful for both of us, and constructive, and he doesn’t want it to be a constant. I feel that too. I will deal with it if I have to. But who is it serving, constantly coming back? If it’s not serving Jason Lee, why am I continually being asked about it?”

Another question, not much addressed in the book, is where does he think this “God desire” actually comes from? If it isn’t put there by God, is it somehow an adaptive trait?

“I’m just making this up now, but in evolutionary terms, it’s possible that without God, we’d all be fucking depressed all the time,” he says. “And if we were all fucking depressed all the time, we’d be committing suicide more and … an animal being depressed would allow them to be easily beaten by animals who didn’t get depressed. And that’s survival of the fittest, right?”

“So once humans get to the point where they realise we’re going to die, life is meaningless, it’s all shit, they would all be depressed. So we wouldn’t exist as a species if we were all depressed. We need something to keep us going.”

Very uplifting, I say. Baddiel laughs. And what is there but to laugh in any case? I ask him at one point whether he expects his argument to change any minds.

“I think it’s absolutely convincing,” he says proudly, while admitting that he doesn’t expect any true believers to find it all that persuasive. The God desire, as he sees it, is hardwired and it’s strong – and there’s simply not much getting round it.