Lily Allen: from chart-topping handbag kid to the heart of London’s West End

There, in the background, wearing drop pearl earrings, is 13-year-old Lily Allen dressed up as a little lady-in-waiting. Cinema audiences watching Cate Blanchett as Elizabeth when the film of that name came out in 1998 might have been concentrating on the queen’s courtly dancing in the middle of the frame, but yes, it really was Allen playing a mini royal favourite in director Shekhar Kapur’s lavish production.

Now, more than two decades later, the 36-year-old singer-songwriter is taking centre stage as an actress in the West End, appearing in a spooky new play, 2:22 – A Ghost Story, which opens this week.

Speaking about the show, Allen has said: “I get to explore mature subject matter, be a woman with a real point of view and show the West End audiences how much I love live performance and being in front of an audience.”

To Allen’s friends and family it has never been any surprise she became an entertainer, whether atop the charts as an Ivor Novello award-winning composer, or now in a straight drama. Neither is it a shock that she relishes playing “a woman with a real point of view”. From the beginning, her strikingly developed attitude was one of the key things this Hammersmith-born girl had going on. Speaking to the Observer’s Miranda Sawyer at the start of her journey to fame in 2006, Allen said her confident singing performance in a school assembly had come as a relief to all: at last, something she could do with all that precocity.

There were many schools. More than a dozen. Allen’s parents were both in show business and constantly on the move. Keith Allen, her father, has built a career on charismatic, unsettling screen appearances, from Comic Strip films to serial killer dramas. Last week it was revealed he is soon to appear alongside Vinnie Jones in a gangster film called Rise of the Footsoldier: Origins.

Her mother, film producer Alison Owen, travelled for work regularly, and was often required to uproot her three children, Sarah, Lily, and Alfie, who is now an actor. For much of her childhood her mother’s one-time partner, Harry Enfield, was an extra dad, following Owen’s split from Allen’s father in 1989. The singer and Enfield remain friends and were spotted walking around London together earlier this year. Rumours that her teenage years were the basis of Enfield and Kathy Burke’s perennial pubescents “Kevin and Perry” are partly true, though Allen claims her elder sister was the real inspiration.

“I was a handbag kid, so I don’t think I was in one place for long enough to take in my surroundings,” she told Sawyer in that first interview, at a point she was being hailed as the poster girl for MySpace, the early social media site. Her ideal future, she said, involved retiring from the entertainment business at 30. “That’s what I want, to have a really exciting block of about 10, 15 years, then marry someone with enough money, get a house in the country and have kids. I really want to spend lots of time with my kids and sit round the table every night and make Sunday roast and grow nice flowers.”

Sawyer, a huge fan of Allen’s talent, said that as an interviewer you didn’t have to be a child psychologist to spot the roots of Allen’s yearning for stability in her shaky childhood.

Far from retired, Allen, now a mother-of-two, is to make her West End debut at the Noël Coward theatre on Tuesday. It is not her only prominent acting role, as she joined her brother and a starry cast a couple of years ago to appear as Elizabeth Taylor in the film of Caitlin Moran’s How to Build a Girl. This time, though, she will be on stage live in a cast of actors, including Hadley Fraser, Julia Chan and Jake Wood. The chance to have such an experience has been a big part of the draw: “The live performance aspect of my career has always been the most thrilling part for me, connecting with a group of individuals and the spontaneity of whatever happens on the night.”

Allen is playing a young newly-wed who believes her home is haunted in a play by Danny Robbins, creator of the recent hit podcast series The Battersea Poltergeist.

“Danny’s play is a brilliant investigation into the ghosts that haunt us and the hows and whys they come to be,” Allen has said. “It’s everything I love; wit, a meditation on marriage and relationships and family all hinging on a frightening plot.”

Curtain-up follows recent headlines about Allen’s public celebration of two years of sobriety. She tweeted a picture of a unicorn cake baked by a friend to mark the occasion. Twitter, Allen has admitted, may be the only troublesome addiction she has yet to beat.

It is possibly not too self-serving to say that the Observer has helped Allen mark several staging posts in her life. Asked recently what she thought she would be doing in 10 years time she replied: “Probably being interviewed by Miranda Sawyer, again … it’s pretty much once a decade. I don’t know… my kids will be 15 and 16, so I’ll probably be out there beating up [their] boyfriends or girlfriends.”

For her part, Sawyer’s admiration of Allen has only grown. (“She is brilliant: honest and funny and clever, what’s not to like?”) But aside from that first prescient interview, Allen also chose to speak to this newspaper in 2016 when she revealed the terrible impact of a stalker.

“I can’t stress how much it affected me,” Allen said two years ago. “After he broke in, I became completely isolated from my entire social group, and my family. I was too scared to go to any public events, because of course people would know that I was there. And I’d just split up from my husband, so I was really very, very alone. I was legitimately terrified. I just stayed at home, for about two-and-a-half years. I went to the studio and that was it.”

The multi-award winning singer with five million albums sold around the world, not to mention that infectious debut single Smile forever in her backpack, has taken on several other social demons in recent years. The Grenfell Tower fire prompted a period of campaigning for justice and, more controversially, greater honesty about the official death toll, which she initially believed was deliberately kept low.

Her work on this and other social justice issues, such as the #MeToo movement protecting women in the entertainment industry, has impressed more than one high profile politician. Jeremy Corbyn is a professed fan and so is David Lammy, who paid tribute to her this weekend. “Lily Allen has brought joy to so many households across Britain through her music, as well as showing unwavering commitment to campaigning on the most righteous causes,” Lammy said this weekend. “She has never let her fame get in the way of her feminism or her support for marginalised communities in this country and abroad. She personifies genuine allyship in an industry where all too often looking the other way is considered certain path to success.”

Related: Lily Allen: ‘I was pretty brazen with all my behaviour. I just didn’t care'

And what of that teenage plan to kickback with her kids in family bliss? Well, shortly after her last album No Shame was nominated for a Mercury prize and her memoir My Thoughts Exactly became a bestseller, Allen agreed to marry for a second time. The wedding in Las Vegas last year to American actor David Harbour was partly prompted by a comment from her two daughters from her first marriage to Sam Cooper, Marnie and Ethel. Harbour, 46, said he was determined to be much more than “just some guy in our lives”. As to Allen’s early retirement scheme, she isn’t drawing down the payments yet.

• 2:22 – A Ghost Story is at the Noël Coward theatre, London, from 3 August.