Controller und Tastaturen sind out - wir benutzen unsere Hirnströme!
Die englische Streamerin Perri Karyal hat es neustens geschafft, das Videospiel "Elden Ring" mithilfe ihres Hirnes zu spielen - dafür nutzt sie EEG, also die elektrischen Ströme der eigenen Hirnaktivität. Dafür musste sie nicht nur im Digitalen programmieren lernen, sondern auch in ihrem eigenen Gehirn!
Wie sie das geschafft hat, und inwiefern diese Technik in Zukunft sogar im medizinischen Feld könnte genutzt werden, haben wir im Interview mit Perri Karyal besprochen:
If you're not too keen on German, then you can listen to the whole interview in English here:
Im Videospiel "Elden Ring" hat Perri Karyal mittlerweile zwei Eingaben, welche sie mit ihren Hirnströmen aktivieren kann - heilen und attackieren. Dafür musste sie aber lange ein Programm trainieren, dass dieses auch versteht, was sie mit ihrer Hirnaktivität auslösen will. Wie sie sagt, habe der ganze Prozess einige Monate gedauert, bis dann alles gestanden ist und zum Streamen geeignet war.
Dabei kann sie auch nicht einfach an eine wahllose Sache denken - sie müsse ein sehr genaues und konstantes Bild vor Augen haben, dass ihre Hirnaktivität stets gleich aussieht. Hierbei durfte Perri auch nicht abgelenkt sein - nur schon eine ungewollte Bewegung würde eine Aufnahme unbrauchbar machen. Insbesondere das habe ihr Probleme beschaffen, da sie selbst auch Symptome von ADHS habe. Perri sagt aber auch, dass sie tatsächlich das Gefühl habe, dass dieser konstante Fokus ihr geholfen hat, mit diesen Symptomen umzugehen - sie denkt, dass sie sich mittlerweile auch nebst diesem Projekt besser und länger auf eine einzelne Sache konzentrieren kann. Natürlich ist hier nichts bewiesen - sie sagt selbst, dass sie nicht wisse, ob der Placebo-Effekt hier eine Rolle spielt.
Trotzdem ist das Potenzial da, dass eine solche Funktion in der Zukunft vielleicht auch im medizinischen Wesen Platz finden könnte - wenn nicht durch Fokustraining, dann durch eine Unterstützung der Mobilität. Wenn jemand die eigenen Hände oder Finger nicht benutzen kann, dann könnte die Funktion, mit dem eigenen Hirn das Spiel zu steuern, einen grossen Vorteil bieten.
"And if I can bring a little bit of awareness to possibilities and the fact that that's important, then that would be pretty cool."
Momentan sei das aber noch nicht wahrscheinlich, meint Perri. Für sie sei lediglich das Aufstellen und Installieren des EEG-Geräts und der Programme zu aufwendig, als dass es helfen könnte. Sie fokussiert sich jetzt also vorerst "nur" aufs Gaming - und zwar vor allem auf das Spiel "Elden Ring".
Wie oben erwähnt, kann sie in diesem Spiel mittlerweile mit ihrem Hirn ihren Charakter heilen und Gegner angreifen. Dafür hat sie auch klare Bilder im Kopf, welche dies ermöglichen.
Der Input für die Attacke sei schnell klar gewesen - hier stellt sie sich vor, dass sie etwas Schweres wegstosst. Der Input zur Heilung war hier aufwendiger - dieser sei durch mehrere Änderungen gegangen. Schlussendlich kam Perri aber die Idee, sich einen Teller vorzustellen, welcher sich in eine Richtung dreht.
Aber auch hier steckt noch mehr dahinter - der Teller muss sich nämlich nicht nur immer gleich schnell drehen, sondern muss auch gleich gross und schnell sein. Deshalb hat Perri Karyal den Teller genau ausgemessen und dreht ihn im Rhythmus von "You Spin Me Right Round".
In nächster Zeit hat Perri Karyal das Ziel, Elden Ring komplett ohne ihre Hände zu spielen - hierbei betont sie aber, dass sie nicht alles nur mit ihrem Hirn machen werde. Beispielsweise würde sie gern mit Augen-Tracking arbeiten - je nach dem wo sie auf ihrem Bildschirm hinschaut, geschieht was anderes im Game selbst.
Das letzte Mal haben wir also auf jeden Fall nicht von der Streamerin Perri Karyal und ihren Projekten gehört.
And if you'd rather read the interview with Perri Karyal, you can do that here as well <3
Trojaner: You've originally studied psychology and have a master's degree in that field. Have your interests in psychology and gaming always been kind of entangled with each other?
Perri Karyal: Yeah. I mean, the interest in psychology came first. I'm fairly new to gaming and so that's always been a passion completely, and then when I first started thinking about gaming, because I was friends with the the, like, stereotypical gamer boys, I always assumed it was this silly little entertainment-throw- away thing that you don't really need to get involved with.
Then when I started getting into it myself through streaming and stuff, I realised that so many games have such depth - I mean, Elden Ring is one of them, but there are even some games that are geared towards the psychological stuff. Like Superliminal or some horror games, and it's just amazing how much care and attention goes into it.
So, since I realised that, it's been years since it's been completely entangled, as you said.
Trojaner: And has there been a moment where your interests have shifted towards using brain activity to play video games, or has it been more of a natural commitment for you?
Perri Karyal: It did come really naturally. I've been working with EEG in my master's course, so I knew that it was there and that it existed. But I thought that it was this massive machine that you could never ever touch unless you had a medical degree and thousands of pounds.
But then I kind of accidentally stumbled upon the fact that you could get little [EEG machines] that weren’t that expensive.
By that point I was talking about different psychology topics on stream, and sometimes we were playing games that were that theme. So, I thought that [the EEG] could be really cool; I originally had it as an overlay on stream and played horror games so you could see my live brain activity during the horror game.
Yeah, that was really, really fun. And it was this educational-ish tool bringing a little bit of awareness to the fact that nobody knows anything about psychology, which needs to change for so many reasons.
And then I realised, OK, you can record patterns [which the EEG sees in your brain], which means if you can do that, you can key bind it because you can key bind anything if you try hard enough. And that's really, really cool.
So, one cool thing led to another, until here we are.
Trojaner: So how long would you say was this process until you'd realised that you were actually able to control a game with your brain activity?
Perri Karyal: Ohh, I think it it took a while - I kind of ignored that feature for a little bit ‘cause I thought it was way beyond my skill set because a degree in psychology does not mean you're going to be amazing at Java. It was completely new.
Once I decided that was something I was going to try to do, it took maybe a few months to get it actually working because I had to really train up the visualisations to recognise the pattern of brain activity and it took hours and hours and hours and then the programming didn't work and nothing worked and I didn't know why a million times over and then the day I was in Minecraft and the day that Steve hit a block... It was just great. I think I screamed.
But even then, it was temperamental. It happened to me sometimes, but it was totally unreliable, so I couldn't bring that to stream yet.
It just was a process. I mean, we've only just added our second input - so instead of just attacking, I'm now attacking and healing, and still, that's somewhat unreliable, but it is getting better every single time so that’s really exciting.
Trojaner: So not only did you have to learn to code and programme things, but you also had to like, train your brain in a whole new way!
Perri Karyal: Yeah, absolutely. And I have some symptoms of ADHD, so that was hard. It took probably twice as long as it should have because I kept getting distracted when I was supposed to be recording.
You have to be doing the same thing for the whole amount of time that you're training it. And I kept getting distracted and looking around the room, so then I had to delete that trial and start again.
Then, sometimes if I wouldn’t notice I was doing that – like absentmindedly tapping my foot – that would completely mess up the brain activity. But I wouldn’t notice! So, I’d save those trials and it would completely mess up the whole thing, so I have to restart from the beginning, hundreds of trials in.
Trojaner: Do you think it's maybe helped you with your ADHD?
Perri Karyal: See, I genuinely do think so, but I don't know if that's backed up with science. I don't know if it’s just placebo, but I do think that I am kind of better at attention shifting now and better at getting my focus right. I think it has applied to other things and organisation and stuff, but I don’t know.
Trojaner: A similar question - do you think this kind of technology of being able to control something like a game or a programme with your brain could be used in maybe even a medical setting?
Perri Karyal: Oh, that's interesting. Well, they are currently using EEG mind control stuff with wheelchairs and prosthetic limbs. They have done that, so that's something like that.
For accessibility in gaming, going hands free and making hands free controllers would be awesome. Like I say, I don't know if it actually helps your focus, maybe it does.
I think the way that I'm doing it right now, it takes more dexterity to set it up and to train it, and to actually do it, and even just putting it on after you've trained than just playing the game with the controller, so I wouldn't recommend it for anything accessibility based now, but maybe in the future.
That would be really exciting. And if I can bring a little bit of awareness to possibilities and the fact that that's important, then that would be pretty cool.
Trojaner: And the EEG, as you said, uses your brain signals to then control the game that you're playing. It does sound pretty abstract when you hear it for the first time and super high tech and like it wouldn't even be possible in the furthest dreams. But how did you connect your brain to the game that you are playing?
Perri Karyal: Once you break it down, it’s actually pretty simple. I mean, I say that, but I guess I’m used to it, so maybe it just seems simple.
All it is, basically: Your brain is a buzz. I mean, you probably know this, but your brain is a buzz with electrical activity constantly everywhere, all the time.
And there are different kinds of brain waves that are just divided by frequency. It's not a big deal.
Basically, all [the EEG program] does, is remember your specific brain activity when you’re doing a specific thing. We look different regardless – like, me talking to you is gonna look different from me doing anything else. So, all it does is record that.
And then, after a certain threshold of “oh, you're doing the same thing again. I recognise this”, all you can do with that is to just key bind it.
There’s a Node-RED compatibility with the brain computer interface I'm using and so you just put that in and then do some Java stuff. I had a bit of help with that because like I say, that's not my specialty at all. But then you do some Java stuff to make it work and you just link it. Once it recognises that, that's it.
Once we figured out how to do it, it wasn’t that hard anymore.
Trojaner: And you've said that for you, you imagine something falling on your face, I think when you want to make your character attack, is that still the case for you?
Perri Karyal: Ohh, I've actually changed that now. So, my attacking was, and still is, imagining pushing something heavy forward. But my second input for the healing item went to so many reiterations.
So, the first one was me trying to drink something, but because it involved my body, I couldn't do the same thing every single time consistently. So, I changed that to pulling something towards me, but that was too similar to pushing something away from me.
And then I tried pulling something towards me from the sky as it's falling down, but then I realised while doing that I was looking away from the screen and so I couldn't see what was happening in the game!
Now I’ve changed that and this one is working way better. It’s really, really exciting – it's a plate spinning! But even then, I was forgetting the speed of the plate spinning and how big the plate was.
Trojaner: Oh, so every little thing changes your whole brain activity?
Perri Karyal: Yeah, and it makes it you have to make sure you're doing the same thing over and over and over again, otherwise it will get really confused. So, now what I've done is I have measured out the exact size of the plate, the exact distance, and I spin it to the rhythm of “you spin me right round baby right round...”
Trojaner: That's amazing.
Perri Karyal: And that seems to work! I don’t know, there might be another iteration later.
Trojaner: That's great. If it works, it works.
Perri Karyal: Yeah, absolutely. I mean, I’m still learning, and I don’t know much about the specific details. Maybe that’s a terrible idea, but you’re right, it’s working, so that’s good.
Trojaner: And you've mostly used this technique in Elden Ring. Did you choose that game just because you played a lot, or did you also see the possibility of it being easier in that game than in another game?
Perri Karyal: Uh, well, it would be easier to a certain extent because you can take it slow; you can be a mage and stand away and have the time to focus sometimes.
So that was definitely part of my consideration. I originally tried it in Minecraft obviously, because I think everyone's first thought is always Minecraft.
Platformers are things that would probably be way too difficult at this point. The main reason I chose Elden Ring, I kind of I saw other people play it and I knew a little bit about it, but this is actually my first playthrough, which, people call me crazy for doing that, but I kind of thought Elden Ring and Dark Souls in general is one of those things where it's such a challenge and it's such a massive sense of achievement that people can really follow it.
Also, the community kind of has precedents for doing it in weird, challenging ways. I mean, there's Miss Niko with her dance pad, and now she's playing with the guitar and there was a guy with banana controllers. Like, that's more impressive than what I'm doing. The bananas? I don't understand how we did the banana controllers. That's wild!
Trojaner: And now you're doing it with your brain.
Perri Karyal: But the brain stuff makes sense to me. The bananas? No idea. But very, very cool.
So, I kind of wanted to jump into that. Also, it's so difficult that people would be more impressed presumably.
Trojaner: And do different games work differently with being controlled by your brain or do most games, or even all games, have the same ground structure?
Perri Karyal: It does have the same ground structure because my visualisations aren't necessarily linked to a specific thing happening in game.
They're kind of a little bit separate, so you can sort of do the same thing. The thing that changes is if there is too much happening on the screen then it can be really difficult. So, some bosses in Elden Ring can be really, really difficult.
Like Rennala, because you're constantly spinning around and have lots of enemies coming at you, and because I'm imagining a plate spinning and the screen is spinning, that's not good.
Right now, I'm playing a little bit of Hogwarts legacy and the thing that changes is the programming a little bit, because if you're attacking in Elden Ring, I set it so that it attacks once every two seconds, whereas in Hogwarts, you kind of have to keep spamming the spell like over and over and over again, so that doesn't work at all. That's the thing that changes, just like subtle tweaks like that.
Trojaner: I'd already be at my last question. You've said that you plan on playing and beating Elden Ring hands free by the end of this year. Do you think this step will be a lot harder than the process that you have already made?
Perri Karyal: I think it's definitely gonna get progressively harder as we go. It's already harder at two inputs. And when I say hands-free, it's not all going to be with visualisations, ‘cause I think that would not be impossible, but very, very tricky, and maybe not necessarily with the resources that I have right now.
But I'm gonna try and do as many EEG brain commands as I can, but there's also different things you can do with it.
You can do gyro controls and movement, which I'll probably use to move, and there's other things; I'm starting to use voice commands for some things as well, which I kind of think that's cheating because that's way too easy, so I am going to get rid of those and swap them for something else later. I'm kind of just trying to use as many things as I can and I'm integrating it with other technology that I've kind of had on the stream for ages.
Chat has the ability to press keys on my keyboard and click my mouse and stuff. I let them have a go at Godfrey by themselves and they couldn't get in the door and that was kind of like my way of saying “hey, listen, it’s hard!”
So, I was like “you guys go!”
To be fair, they did hit and they did do better than I did the first time, but I’ll never admit that.
Trojaner: They'll never know.
Perri Karyal: They'll never know.
Trojaner: Alright, if there's anything else you would like to say on your experiment or project or achievements, then I'd love to hear it.
Perri Karyal: Eye tracking is a thing that I might do, because that’s another thing that’s used in psychology a lot. I know that other gaming streamers are using that.
But potentially even that as a way to move, instead of the gyro – looking around the screen and following something. I haven't messed with that. I know about it from my degree, but I've not used it the same way that I've used EEG, so that would be brand new, and I don't really know what's possible, but that could be fun.
And then it's more your face. You know, if you use voice commands, like I say, that's a bit cheap, and if you use gyro again that's maybe a bit cheating, but if it's your eyes, that's still technically your brain.
