The Multimodal Brain Orchestra performed its world premiere on Thursday.

星期四世界首次上演大脑控制的管弦乐表演。

Brains took the place of violins in the novel orchestra

Led by an "emotional conductor" and a traditional one, music and video change in time with the performers' brain waves and heart rate.

According to the work's producer, the orchestra（管弦乐队） aims to "see what the brain can do without the body".

The orchestra's premiere（初次公演） performance closed the Science Beyond Fiction conference in Prague.

The project is the creation of the Synthetic¹, Perceptive², Emotive and Cognitive³ Systems (SPECS) group at the University Pompeu Fabra in Barcelona.

"Only recently we have come to appreciate more the tight coupling between mind, brain and body," Paul Verschure, head of the project, told the audience.

"But we can wonder what the mind and brain would be capable of if it would be directly interfaced⁴ to the world, bypassing（分路作用） the body."

The piece that the orchestra performed, Xmotion, has been scored, giving it an underlying⁵ structure.

But it is the performers who control variations of visuals, sounds, frequencies, and volumes in the overall piece.

Model result  

"Everything is built to fulfil the circumplex model, which was worked out by psychologists that study emotions," says Anna Mura, a biologist who is the producer of the project. "How we feel and what we feel fits into a circle cut into four quadrants（象限）, labelled by 'arousal（激励，鼓励）' and 'valence（化合价）', that is, how much you're excited and how badly or positively⁶ you're excited."

To that end, four performers were fitted with caps littered with electrodes（电极） that take a real-time electroencephalograph - an image of the brain's electrical activity.

"There is a first violin, a second violin and so on, except that instead of violins they are brains," says Dr Mura.

The graphs of those brain waves are projected onto one of two large screens above the orchestra. The performers launch sounds or affect their frequencies and modulations（调音，调节） based on two well-characterised effects seen in EEGs: the steady-state visually evoked⁸ potential (SSVEP), and the so-called P300 signal.

Two of the performers were given a task to watch a screen in front of them, with flashing rows and columns of letters, and told to look for a particular letter.

When expectation is fulfilled, 300 thousandths of a second later, a signal known as the P300 appears in the EEG.

A similar strategy has been employed by Mick Grierson at Goldsmiths, University of London to generate individual notes.

In the Multimodal Brain Orchestra, the P300 signal is registered - with a dot demarcating（划分，划界） it on the EEG trace projected to the audience, so that they can see the effect of the performer's thought - in turn launching a sound or recorded instrument.

Two more performers were given boxes with four lights flashing at different frequencies. The SSVEP is a brain signal that comes about when visual stimulus⁹ in the retina（视网膜） at a given frequency causes the brain to synchronize¹⁰, so that frequency appears in the EEG.

Given a cue from the conductor, the performers switch their attention from one flashing frequency to another.

One of them affects the volume of a given sound - known to influence the level of arousal（激励，鼓励） in the circumplex model - and the other affects a certain modulation⁷ of that sound, which is known to influence the valence, how positive or negative the arousal is emotionally.

'Emotional experience'

Adjacent to（靠近） the EEG-capped players, the "emotional conductor" sits comfortably, wearing a pair of virtual reality glasses.

She is being shown images from a series created by artist Behdad Rezazadeh while her heart rate and skin conductance（皮肤电传导） are being measured. Her heart rate is plotted along with the EEG traces.

As her mood changes, so does the visual experience - Rezazedeh's images are blurred¹¹ and changed in line with the changing biological measures of the conductor.

"This performance is like her emotional experience," said Dr Mura. "What we want to show here is the use of your brain without your body. Embodiment - we should get rid of it sometimes."

While this artistic¹² interpretation¹³ of the biometric（生物计量的） technology is a fascinating look into the merger¹⁴ of technology with medicine, the overall goals of the group have far greater implications.

"People believe that to understand how we feel will help us to understand what consciousness is all about. This is the technology that is going in that direction; we cannot explain consciousness with this but we are at least exploring the surface of it."