Motion fusion on Smartphone
Just imagine: you, two people, are playing MassAffect, adding up your two movements in real time. Indeed MassAffect does just that, instantly adding the two movements together to make a game!
Yet if you think for a moment, adding two movements is impossible. It simply doesn’t exist. So it’s an unrealistic proposition? Yes, in everyday life at any rate, and except in mathematics, where manipulating this sum means playing with the center of mass. In the image below, the center of mass, or barycenter, is the midpoint between the positions of the two moving players.
Images of the first prototype programmed from the initial idea in 2015 by Maxime Bouton. The point to the left of player 1 leaves the red trace. The point to the right of player 2 leaves the green trace. The point in the center (1+2) leaves the gray trace in real time, and is at all times the mid-point of 1 and 2. Imagine, the combinations are endless. They’re also astonishing.
But with the MassAffect game, you don’t need to do math to play with the barycenter. “Come as you are!” Just come as you are.
Enter the playground, in fact the screen of your smartphone connected to your partner’s by WiFi, and let’s see how you learn to move together to control this barycenter, the sum of your movements in space.
To play oneself, a MassAffect app was produced by Paul Peter Arslan for Google Play.
MassAffect can be realized in many different ways: on the screens of two smartphones or tablets, in the real space of a connected gym, in a virtual space using VR virtual reality… The rules of the game in MassAffect can be varied ad infinitum. But the challenge proposed by MassAffect always remains the same: to draw in pairs, simultaneously, by manipulating, in real time, the barycenter of two points. The trajectory of the barycenter is the result of joint improvisation by two people. The movement of the barycenter is manipulated by the two players, each on their own screen. It’s simple and surprisingly fascinating! Let’s find out why!
Introduction
You control the movement of the point on the right with one finger. Your partner, the movement of the point on the left. MassAffect sums up your movement and that of your partner. The result, visible simultaneously on the 2 screens, is the movement of the middle point, the sum of the movements of the other two points. And if the game rule you’ve chosen says that “the point on the left and the point on the right must move on the same circle”, is it possible to draw a square at their midpoint? The answer is yes. But you’ll need to coordinate carefully.
But let’s go back to the beginning: can we talk about the sum of two movements? Does it exist? Is it common? In everyday life, we think it doesn’t exist. Movements don’t add up. In maths, on the other hand, the sum of two movements is, if not obvious, at least very clear. If you know the associated maths, there’s no ambiguity in this vector equation:
A is the position of the first player. B is the position of the second player. G is the position of the barycenter, here simply the midpoint. This is what the concepts of barycenter and center of mass are all about. Point G is simply the graphic representation on the screen of the barycenter of point A and point B.
“Come as you are“. You don’t need to be a surveyor!
But to play with MassAffect, you don’t have to be a mathematician or a fanatical physicist. You really don’t. On the contrary, it’s perhaps an advantage not to be. Let us know if you play. Enter as you both are on the field formed by the screens of your two smartphones. The game welcomes you. Show us how you create an amazing team that invents its own way of playing. In fact, this game is original. Nobody knows yet what the best strategies are. You can even invent a multitude of new game scenarios, or Gameplays. How will players develop the best strategies for playing MassAffect? How can they coordinate their movements so that there are no mistakes, no misdirections in the trajectory of the midpoint ?
Joint improvisation to create
We’re not the only ones, and not the first, to have asked ourselves this kind of question. MassAffect recalls this study from 2011 entitled:
“The mirror game as a paradigm for studying the dynamics of two people improvising motion together”
According to the authors in the article mentioned: “A one-dimensional mirror game shows complex, coordinated improvised movements. In the mirror game, the motion of the two players moving handles on tracks, is sampled at 50 Hz and at a spatial resolution of 0.94 mm.” Image from the article.
Ariel Lindner, INSERM research director and then Director of Research at CRI Paris, introduced us to this research after we had created MassAffect. It was carried out by researchers at the Weizman Institute of Science (Israel) and Harvard Medical School (USA). The rule in this game is based on a mirror effect: players copy each other. It’s also a mathematical rule, in this case based on symmetry.
“Joint improvisation is the creative action of two or more people without a script or designated leader. Examples include improvisation, theater and music, and everyday activities such as conversation. In joint improvisation, a new action is created from the interaction between people… We measured the hand movements of two people copying each other, with high temporal and spatial resolution. We focused on actors and musicians who are experts in joint improvisation. We discovered that actors can jointly create unprecedented complex movements without a designated leader, synchronized to less than 40 ms.“
One of the lessons of this study of joint improvisation is that you don’t need to appoint a leader. In this type of improvisation, it’s best to collaborate in real time. A priori, MassAffect puts players in similar conditions. At any rate, that’s one of the questions.
A brief look back. In their article “The mirror game as a paradigm for studying the dynamics of two people improvising motion together” in the journal Proceedings of the National Academy of Sciences in 2011, Lior Noy, Erez Dekel and Uri Alon begin their introduction with: “Joint improvisation is the creative action of two or more people without a script or designated leader.” We have therefore translated “joint improvisation” as “improvisation conjointe”, which we feel is the right translation for our purpose. By the way, this sentence, which starts their article, is perfect for describing MassAffect’s potential!
MassAffect is born: a Science&Design workshop
In spring 2015, we had organized a workshop with design students as part of a collaboration between the Ecole Supérieure d’Art et de Design, ESAD Grenoble-Valence and Grenoble Alpes University. We decided to explore movement and gesture using smartphone sensors. Our phones carry all the sensors needed for this measurement, which they carry out every 10 msec, faster in fact than in the experiment carried out in Israel.
Entering into a collaboration between science and design is always surprising, even jarring. No real surprise if we follow the designer matali crasset who, in August 2020, stressed on the France Inter program “le design est partout”:
“Design is a skill that must combine critical thinking, creativity and practicality.
On a subject as broad as the one we’d chosen, and therefore quite ill-defined, the resonance between science and design is strong, but tension sets in. And in practice, it didn’t start well. How could 5 design students and a physics teacher understand each other and work together? Actually, we did that on purpose. It’s a lot more interesting, and a lot more fun. But we had to get specific. Hard. And the combination of movements took center stage. One of the students, Maxime Bouton, now a designer, finally broke the lock with an embryonic game: “I move the right point on my screen, you move the left point on yours. The middle point is the result, at every moment“. Simple. However, it took a moment for the physics teacher to realize that Maxime had constructed this proposition without reference to the barycenter, which he had never learned, or had completely forgotten. Not that it mattered, since he had rediscovered the notion and some of its properties by playing on a screen with a point on the right, a point on the left and a point in the middle. At the time of this workshop, MassAffect was not yet called MassAffect, but it had been created.
MassAffect in action!
The surprise in Maxime’s proposal is the immediate discovery of the immense play space that opens up. When you move the right and left points simultaneously but independently, the middle point draws unexpected shapes that are not so easy to grasp. Incidentally, it seems obvious to us that the middle point can draw any closed boundary on the screen, whereas the right and left points are constrained to move along any closed boundary. It seems obvious to us from the many examples we’ve tested. But none of us has taken a serious look at whether the result is that general. If you have a demonstration, we’d be grateful!
What kind of gameplay?
The notion of Gameplay is essential when working on the creation of digital games. It can be defined by following Marc Goetzmann and Thibaud Zuppinger (2016):
“Gameplay is the articulation between game, the structures and rules of the game, and play, the way in which the player appropriates the possibilities of the game by developing his or her own strategies, in response to the constraints imposed by the “constitutive” rules of the game”.
Following this definition, a multitude of Gameplays framing the movements of the midpoint are possible in MassAffect. But in all cases, the central challenge offered by MassAffect is to succeed in this joint improvisation: to draw together in real time, coordinating precisely to achieve the proposed goal.
In this gif, the red cross shown here motionless (it’s actually rotating around the sun) is the center of mass of the planet Pluto and its satellite Charon. The large disk represents Pluto and the smaller Charon. The Pluto-Charon gameplay is very well defined by the movement of this planet and its satellite.
See Plato and Charon’s movement here: https://fr.wikipedia.org/wiki/Barycentre#/media/Fichier:Orbit2.gif
Keeping the barycentre fixed while the two points, the one on the right and the one on the left, move, is therefore one of the possible MassAffect gameplays: how can the points up and down here move without the middle point moving?
MassAffect, a pure design. Pong comes to mind
Nolan Bushnell and Allan Alcorn’s Pong has been universally known since the 70s. It has always had a very frugal graphic design. It appears as basic, simple and immediate as possible. This has probably contributed to its worldwide success. The video of MassAffect earlier in the article shows our commitment to very clean graphics and obvious gameplay. Everything is deliberately free and open. And we’re keen to get feedback on the way we play, by following the Gameplays we’ve developed.
Pong opposes, MassAffect associates
The fundamental principle of Mass Affect is ultimately as elementary as that of Pong. But Pong is like tennis or ping-pong. Of course, you play in pairs and work together, but it’s primarily through opposition that collaboration is established. It’s much more exciting if the two players are at the same level and challenge each other. But in the end, even with elegance, there’s a winner and a loser. In MassAffect, there are no winners or losers. You can only play with it. Not against. We improvise together at every moment. We discover and learn together. That seems to us to be a fundamental difference. If we follow the results of the Israeli-American study, we might think that playing MassAffect by trying to install a leader and a follower is counterproductive, especially if the gameplay favors speed. In 2011, this study highlighted:
“Despite being at the heart of creative processes and social interaction, joint improvisation remains largely unexplored due to a lack of experimental paradigms.”
MassAffect represents a new paradigm for exploring the modalities and conditions for the emergence of joint improvisation. As with Pong, the basic rule is obvious, but how to turn it into a game that is both engaging and in which both players progress by collaborating and practicing is, in our view, an open question. And terribly exciting.
MassAffect and learning maths
Another open question: when playing MassAffect, is it more interesting to go back to geometric constructions of the barycenter to study the best game strategies, or is it better to rely on intuition? In our experience, when both players have to move points on the same circle, it’s useful to have in mind the geometric construction associated with the barycenter of two points on the circle. Quickly in particular. This gameplay is included in version 1 of MassAffect. But there are other situations that are just as simple to state, and which leave us perplexed in this respect. If the point on the right and the point on the left have to move around a square, how do players move them so that the middle point draws a circle? We don’t know whether it’s an advantage to have “theoretical” thinking associated with the game, so that two players can better control the movement of the midpoint. It must also depend on the players. In general, at this point in time, we don’t have any information that would enable us to anticipate players’ collaboration and learning strategies as a function of the gameplay. We believe that the best thing for everyone – players, developers, designers and even teachers – is to try it out while playing. A double game will develop… The players are in the game proposed by the Gameplay and try to improvise together. The Gameplay designers observe the players, scrutinizing the way they learn and collaborate. They play with the players. We could even imagine giving players the freedom to completely change the game between games. MassAffect could be just a game, or first and foremost a tool for exploring and learning mathematical notions, or both together. It all depends on the design, the gameplay objectives and, of course, the players…
Is it possible to improvise with the center of mass “for real”, and not on a screen? Yoann Bourgeois has done it.
How do you create the material conditions for this joint improvisation of two dancers’ movements? Choreographer and circus artist Yoann Bourgeois answers this question by playing with the position of the dancers’ center of mass. Yoann Bourgeois and his company invented this device many years ago. Like the center-point rule of right-point-left, the principle is quite simple. The device consists of a vast wooden stage set at its center on a single articulated point, as shown in the video. Left alone, the tray comes to rest with one of its sides on the floor. The dancers, once installed on it, must constantly seek this precarious balance in all their evolutions: the center of mass of the whole set and dancers, must be at all times confused with the single point of support. Otherwise, they’ll fall! Deviations are possible, but you have to compensate very quickly with coordinated movements. It’s a joint improvisation by the dancers. The system is so sensitive, so precise, that it’s impossible for the dancers to coordinate their movements in advance. The aim of the couple on stage is to find themselves in balance at the end, facing each other. Then they can finally look at each other.
This show by Yoann Bourgeois is a major source of inspiration for us. We could even see MassAffect as a highly simplified simulator of this device.
Precarious balance and interaction in joint improvisation
Yoann Bourgeois’ set-up underlines the fact that this study of joint improvisation combines with an exploration of the modalities of real-time interaction between partners, in this case the dancers on stage. In this show, the choice is radical. It’s the rule of the game: neither partner can look at, talk to or touch the other. The stage is the only real-time communication channel between them. Proprioception is used here. It detects the slightest imbalances on the stage, which must be constantly corrected by body movements. The two dancers build their couple and their evolutions from this single piece of information, this single constraint. Yoann Bourgeois brings a new experimental paradigm of interaction to the questioning of the Israeli-American study, thanks to this mobile stage. It enables proprioception alone to be used. It conditions joint improvisation and is at the heart of the creative processes and social interaction thus created. In his documentary “Notre véritable 6e sens“, Vincent Amouroux puts into perspective the importance of proprioception in Yoann Bourgeois’ work, and its resonance with ongoing research on the subject in university laboratories.
MassAffect beyond the screen
Our first version of MassAffect constructs on screens a tangible representation of a totally abstract mathematical notion, the barycenter. Mathematics makes it possible to construct these images, but in fact, by definition, it does very well without this type of representation. Mathematical tools such as symbols, definitions, functions and logical operations are absolutely rigorous vehicles for sharing notions and manipulating them together.
For fun and illustration, if G is the barycenter of {(A,a),(B,b)} then for any point M in the plane, we have the relation :
You can illustrate this equation by drawing with arrows and points in the plane. Or not. Drawings, or any form of graphic representation, certainly facilitate exchange and understanding, and sometimes greatly so, but they are not essential.
Games like MassAffect let you play in screen space using mathematical rules. From these representations, it’s human perception in real time that takes over to play. On the one hand, MassAffect can be a way of exploring joint improvisation based on channels of perception and information sharing selected by the gameplay (we could also consider sound and vibrations, for example), but it can also be a first step towards exploring abstract mathematical notions in the real world, based on their representation on screen. It’s also another exploration to see whether this approach enables new pedagogical approaches.
Body in motion and MassAffect
Yoann Bourgeois puts the moving body at the heart of joint improvisation. Not the beta version of MassAffect presented here, since it’s on screen, which doesn’t put the body in motion. With Renaud Bastien, now a CNRS researcher in Toulouse, and Adrien Husson, a designer-engineer, we came up with two other versions of MassAffect. One in Virtual Reality and the other in a Connected Gymnasium. In both cases, joint improvisation becomes body movement. To the point of running out of breath.
A final version, which we’re also keeping in reserve, will use Smartphone GPS: whatever the players’ movements, the barycentre must remain on the forecourt of Notre Dame de Paris, or under the Ombrière du Vieux Port in Marseille, or on a village’s Mairie. Depending on the terrain, good luck! In Marseille, if one has climbed up to Notre Dame de la Garde, the other is somewhere on the Saint Charles Campus of Aix-Marseille University…
And last but not least: playing MassAffect with an AI as a partner?
Immediate benefit: the question isn’t who’s going to win? There is no winner in MassAffect, but rather the pleasure of exchanging and collaborating to do things together. Paul Peter Arslan has proposed pairing an AI with a human player. The AI and the human player learn together and try to settle into joint improvisation. So the AI must also learn improvisation with this human player if we take up the paradigm stated above: “Joint improvisation is the creative action of two or more people without a script or designated leader.” with the associated discovery: “We have found that actors can jointly create novel complex movements without a designated leader, synchronized at less than 40 ms.“
In this possible real-time collaboration between an AI and a human, who together learn each other’s game and way of playing so as to always progress, there’s a very exciting field of exploration!
Credits
MassAffect is based on an original idea by designer Maxime Bouton in Valence in 2015.
The game developments presented here are the result of work by Paul Peter Arslan, a project at DVIC during one of the Covid confinements.
Between spring 2015 and the Covid confinements, we built various versions of MassAffect, including some very sophisticated ones like this one the result of a project at the Centre de Recherches Interdisciplinaires (CRI Paris) at Université Paris Cité.
Teaching, exploring, learning with students... 
