Mr Bray was escorted from a meeting of the right-wing Bruges Group outside the secure zone in Manchester city centre after interrupting a speaker.
Wearing a blue pro-EU T-shirt, Mr Bray said: “Where are the Brexit benefits?”
After briefly grappling with an audience member, he was escorted from the venue, continuing to protest and describing the attendees as “losers”.
The meeting at the Friends’ Meeting House on the subject of whether the Conservatives could win the next election had been due to be addressed by party vice-chair Lee Anderson but the Ashfield MP had pulled out at short notice.
Mr Bray has staged a long-running protest against Brexit, with a weekly demonstration in Parliament Square while the House of Commons is sitting, and has come to Manchester to protest at the Conservative conference.
During debates on Brexit, he was often seen and heard in the background of news broadcasts from near Parliament.