Mediating Complexity

The collective goal of our class was to create an interactive system in the SMALLab environment that enables users to explore the concepts of sacristy and sustainability while performing in roles that encourage collaboration and evolving levels of engagement.  The sound element plays a large role in the system, as it serves as an indicator of user health, and provides broad-level feedback to all participants.  The challenge in designing the sonic equivalent to the feedback provided by visual cues emerges when trying to create audio cues that are relevant to both the individual and the participants as a whole.

The generation of sound is dependent on data sources provided by the interaction model.  Using this data, we define the global health of the system as the balance between the water supply status and the combined statues of the regulator, urban user, and agricultural user.  The interpreted “health” of these actors drives the generation of sound, which can be heard by all users and observers.  In order to utilize the distributed nature of the SMALLab environment, we project sound through two speakers mounted on the Small Lab frame, as well as through two omnidirectional speakers (created by Curtis Bahn and Stephen Moore) located on either side of the agricultural and urban user space. User activity in each user space shapes the audio feedback coming from the speaker pair closest to that user. This broadens the distribution of sounds, which reinforces the roles played by each participant and provides each user with localized, individualized feedback. 

Using the SCREM architecture, SMALLab’s protocol for integrating streaming data and network communication between different programs, we constantly monitor the flow of data from the interaction and computational models.  For the audio component, we followed the SCREM render engine protocol to create three Max/MSP render engines, one for each of two users (urban and agriculture), and one to dynamically alter/filter/play back an audio file. These render engines are used to trigger and adjust filtering and playback options for our audio samples and stream this output to the various speaker locations.  

For the first incarnation of this system, we use the PeRColate library by Dan Trueman and Luke DuBois to apply and manipulate a clarinet sound to the urban user’s actions, and an altered and filtered mandolin sound to the agricultural user’s actions. These sounds help establish the amount of activity and identity in the group choreography as a user requests more water. As a complement to user activity, the amount of water supplied to the user by  the regulator is summarized sonically by a filtered sample of flowing water. If the water supply channel for a user is ample, the sound is full and the filter is wide. As the regulator decreases the supply, the center frequency of the sample increases and the band pass filter narrows, resulting in a thinner sound and a thinning water supply.

Since chaotic movement characterizes the imbalance of efficiency in a user’s space, this chaos is reflected in the global mix.  We seek to demonstrate this through sound by translating the effort VS demand curve produced by negotiations between users.  Equilibrium, the ideal state each user strives to achieve, is a relative point on this curve that has been translated to a visual (through the use of graphics) and sonic representation.  As the distance from the point of equilibrium grows, effects are applied to the audio samples.  An imbalance in the system becomes a perceivable sensation as hi-pass and resonant filters alter the sound.  

Due to the fact that the system as a whole is still under development, we encounter several problems in continuing the sound interaction component.  For example, because the data from the interaction model is sporadically reported, creating a consistent “flow” of water sound samples proved difficult.  When the activity data of each user is reported in a smoother gradation, the evolution of sound will become smoother as well.  Future developments in the sound generation component will likely involve a refinement of the audio samples used for each role, as well as improvements in the audio feedback provided to users.

There’s one more video on blip, too.

Please also see my set of photos on Flickr.

In Mediating Complexity this semester I worked closely with the team to develop the Arizona Water System for Complexity elements having to do with gestures and tangible interfaces. I helped shape the interaction space and helped determine the actions in the game and their effects. I worked closely with Chris Martinez and Todd Ingalls to make the new Native American Wedding Vase SMALLab interaction object and to integrate it into the game’s functionality. In addition, I created a large portion of the floor graphics for the game.

In working on the Wedding Vase Controller, we modified one of Byron’s smart object/UDP multicasting Max/MSP files to work with the vase, constructed body of vase with paper mache and stiffened fabric, assembled embedded electronics in vase including bluetooth module, mini arduino board, 2-axis accelerometer, and LEDs for tracking, and calibrated tipping/pouring gesture for game control using the vase’s accelerometer.

I Worked closely with Ryan and Shawn to create pipes, valves, water sources, and end user graphics, mainly in Adobe Illustrator, for the Water System floor graphics. Ryan and Shawn then animated the color changing water and end user environments as well as made the valves spin.

I also worked to maintain the wiki with up-to-date information regarding the state of the (ever-changing) system. Please also see my video documentation of the working prototype on the blog.

This semester I learned to be a better collaborator while working on this system. I also took my interest in semantic tangible interfaces to a new level, thinking about interactive choreography and its semantic meanings when directed through a controller object.

Christopher Martinez AME598 April 30, 2008

Critique, questions, and lessons learned.  Systems for Media Complexity Spring 2008.

 I spent this past semester working with Becky Stern and Todd Ingalls on tangible interface design and construction, interface implementation, gesture interface design, and distributed choreography as an interactive display for our Arizona Water System for Complexity.  This work includes the design and construction of a water vessel, and Max implementation of this vessel and the accelerometer ball into the system framework.  As part of our object design, we considered the semiotic nature that the objects themselves represent within the context of Arizona culture and water usage, as well as appropriate gestures that provide semantic meanings that are relevant to the nature of different activities defined by different administrative roles and consumer identities.  Our gestural signs are highly linked to the tangible interfaces themselves, as well as the tasks they are designed to represent.  As part of this work, we provide a framework on the wiki that defines how different choreography distributions in a quartet define focal points, semantics, and interpersonal relationships.  These distributions account for choreographic intentions that are tied to the locomotion of movers in a space, as well as directional awareness of the dancers and their proximity to each other.  As part of this framework, we discuss strategies that can be used to create meaningful choreographic displays that are easy to understand and interact with.  (Please refer to our Gesture and Interface Design document on the wiki for detailed information.)  Much of the information provided is theoretical, and will require a robust gesture recognition system to test.  However, I believe that our current choreography and tangible interfaces provide a good foundation from which future improvements can be derived.

 Now that AME has a working prototype for Arizona water usage, many new questions emerge about its potential for social and political awareness, as well as its relevance within the context of culture.  These issues will ultimately determine the evolution of our new system’s form and function.  With this in mind, I wonder if a single system can be designed to capture an audience curve that represents a broad range of professional backgrounds, political motives, and cultural perspectives.  As a student of the arts, I want to use this summary as an opportunity to critique and question aspects of our work.  This is for the purposes of providing constructive thoughts about our project for future growth.

 It appears we have achieved success in building a prototype that can be used for sustainability awareness and education.  This makes art historical sense if we examine ourselves as a collaborative of artists working to apply our craft to identify social and cultural behaviors that emerge within our state’s water supply and usage.  By adjusting and modifying our time scalars, and using live data in such a way that the levels of complexity are intuitive, we will have done what artists often do best when a work of art is successful: we will have provided and framed a picture for an audience to reflect upon a particular issue in the context of culture and the time unique to it’s existence. 

If this perspective is less than naïve and fair to assume, I am excited about the outcome, and think that our model will be well suited for venues such as SMALLab for school field trips, the Arizona Science Center, art/dance/music festivals, conferences, etc.  However, I believe that our SOS partners continue to challenge our abilities to frame our portrait in a way that provides policy makers with new scientific knowledge that they can’t currently attain through graphs provided by Water Sim at Decision Theater.  How does our idea for emerging real-time data visualization and sonification in the form of interactive media and distributed choreography translate into meaningful information that water managers and politicians can apply towards decisions?  I believe that this will be a challenge because we may be asking people from the culture of policy to rethink water through the lens and cultural perspective of art. How do we continue to work towards the challenges set by our SOS partners, and still maintain a form that can continue to serve the other cultural needs mentioned above?  Do we need to be thinking about a system that can intelligently assess the nature of a given gesture field?  Perhaps something about this information curve can tell us about user intent; meaning, do water managers approach the field differently than the public at the Arizona Science Center?  If so, does it mean that we have different systems for different contexts?  Or does it mean that we have a single system with states that adapt to different user intentions? 

 I believe that our water module represents an excellent opportunity for us to explore these research questions.  This is because, regardless of the aspect of culture from which we come, climate change and sustainability represent issues that concern us all.  This makes our challenge to design a system that can effectively communicate to the broadest spectrum of cultural contingencies important.  We have yet to nail down a gesture interface that can evolve from abstract gesture to gestures that represent specific signs.  Once this interface has been worked into the interaction, and is coupled with live historical data, we will then have an opportunity to fairly assess what types of new knowledge emerge through a given interaction, as well as how that knowledge varies within different contexts that are determined by setting, motive, and types of people who are playing the game.  We may find that, under the current conditions, our system is effectively informative to policy makers and water managers, as well as to school children.  It will be interesting to see where our system lies within a cultural target. What steps will we need to take in an effort to model the widest spectrum of the cultural curve?

 A couple of side notes:  (a) Working on a module for water in Arizona is interesting because we not only have to think about the factors implicit in a philosophy for sustainability, but also about how to create an interaction that is sustainable itself.  The system of sustainability that I am referring to is our interaction’s ability to maintain creativity, negotiation, and choreography of its participants for a period of time long enough for new knowledge to emerge. (b) I understand that we can increase the number of players to our game.  I think it will be important to increase our number of consumers because one person cannot realistically model the interests and needs of given contingencies of water usage in Arizona.  No one person has this one-on-one access to regulators and suppliers for water.  What makes the system complex and challenging for sustainability is that it is difficult for consumers of different identities and needs to mobilize and unify.

I have enjoyed watching this process unfold this semester.  It has been a great education for me to work with a team to create an interactive design that accounts for so many different factors.  I feel that what I have learned will provide me with greater access to the problem solving and collaborative conversations that are necessary to put systems like this together.  I also feel that, after working in this context, the education and experience that I got will enable me to be a much more effective collaborator in future works.   

A number of topics seemed to dominate this class, so I’ll touch on each one:

• Understanding of stochasticity
• Choreography and embodied interaction
• Behavioural change and potential uses

Understanding of stochasticity

It’s hard to see where we’ve come with this, as a lot of what drives the system has yet be put in place. Primarily, it might be necessary to actually start the data feed to give the system some variability. There are also certain difficulties in the modeling stage, such as immediate resource transfer rather than flow. Immediate halts in the system probably frustrate the user more than they do reveal stochastic uncertainty in the system. For example, if you’re in SMALLab and the ball tracking suddenly stops working, that’s frustrating. If you’re in SMALLab and are controlling a cloud of particles with stochastic movement rather than a mirror of the ball, that’s introducing stochasticity.

This mostly has to do with the choreography, but for the immediate resource transfer, we don’t allow enough leeway in how people behave for them to learn, really. For example, the supplier will pump some water into a net supply, but as soon as the user has consumed it, the system comes to a halt. Rather than it being harder for the consumer to take water, it’s just impossible. This has to do with a particular way the model was done, and I think future iterations should touch on this. It was planned in the original model, but wasn’t implemented. Particularly, the concept of net supply is erroneous because there’s not much of a real-world analogue. In the next iteration of the model that we’ve been discussing, there would actually be a constant flow of water from the water sources to the consumers that would be affected by all players’ actions. I think this will correct a lot of these issues.

Additionally, although it creates an interesting decision-making process for the user, I’m not sure that our concept of banking is entirely correct. Specifically, the idea of “groundwater banking” doesn’t have much of a real-world analogue, as I understand it. The closest concept our banking interaction represents is groundwater replenishment. I suggest we stop referring to “banking” when mentioning the middle supply source. As I understand it, preparation for drought is primarily a function of SRP’s system of reservoirs, which we have yet to model, as the data we plan to use to drive the system is based off spillage (before storage).

Choreography and embodied interaction

A number of things need to fall into place before we can really see the type of choreography that takes people out of the context of the media and begins to allow them to get a holistic sense of the space.

First, I think there are certain biases in the metaphor that we’re presenting to the players that prevent them from exploring the entire state space of the model. The concept of user vitals is very similar to points. I feel that vitals may have been a concession that we made to simplify the model on the first iteration. The difficulty with points is that when a user is low on points (when my grass is not so green), a player is likely to consider that as a “bad” state and unlikely to explore states beyond that any further. In cases where it is not possible to maintain all vitals at once, this leads to certain frustrations that I am not sure whether or not the interaction should be fueling. As I understand it, we are primarily intending to create an exploratory tool. I’m all for making games, but if we make this a game, we are also making it an extremely difficult game, which can be frustrating. Several traditional computer and console games have had success with leaving the intended goals up to the user and still providing the same, if not a better, sense of enjoyment without limiting exploration.

Additionally, the switch to a resource model based on water flow should provide a state space that is much more rich. When a user can visually see and sonically hear the results of his or her actions beyond the choreography and resultant media ascribed to a specific user, the interaction is much more interesting. As it is right now, the supplier supplies some water, the regulator regulates, magic happens, and then the state is communicated on the other end of the room. There’s nothing in the media connecting users besides their own gestures, so there’s no sense of what’s going on in between (in the current model, this could simply be a matter of visualising or sonifying the net supply).

Behavioural change and potential uses

I think this is probably the most important topic, as it is the topic that most directly relates to the potential uses of the scenario. As the faculty from SoS have discussed, one of the key challenges of sustainability is behavioural change. We can do many things with the scenario, including 1) using it as an academic research tool for studying resource and effort negotiations and 2) using it to promote understanding of complex systems based on use of variable resources.

Both of these uses are important, and I think we are really missing out on the community involvement aspect. One of the most attractive aspects of digital media is that it does not have a physical manifestation that requires any reserved space, but the opposite is true of SMALLab or Decision Theater. To truly be able to do actual research on behavioural change through media, I think it is extremely important that we develop tools that do not have reserved spaces and can be experienced by the public. AME has experience with bringing SMALLab into public places, such as schools and the ASU Art Museum, but I think we are still really missing out on a great opportunity if we limit ourselves to single physical spaces. Personally, I feel this is the more interesting future for the project, because it’s not particularly useful beyond studying negotations in user studies if the only people who get to use the system are people in suits to whom we demonstrate in a dark, sterile room.

Note:  This class review was also posted on wiki 4-27-08

Brigitte Hill AME598 April 27, 2008 Experiential Media Systems for Complexity and Sustainability

Spring 2008 ~ Lessons Learned Sustainability is a nebulous term, evolving as new dimensions of concern arise.  The three major areas of sustainability, economic, environmental and societal, are inherently integrated with each other.  Tradeoffs are often negotiated to sacrifice the needs of one aspect for the needs of another.  As example, many Native American tribes have recently leased their land to host toxic waste dumps, sacrificing environmental health for the much needed economic incentives to sustain their members.  When tradeoffs occur, whichever need is most pressing arises, rarely as a consensus, but most often as a compromise after passionate discourse.  A commonality of sustainability discourse is the need for systems thinking.  A system to mediate complexity is a fantastic tool to help facilitate systems thinking.  It allows the role player, through interactions with a computer and three (or more) other humans, to realize the consequences of his/her movement and gestures via both visual and auditory feedback.  The magnitude of effort, or the ease thereof, provides indicators of the other players’ actions.  Those actions are a result of choice, be it random or contrived.  This uncertainty of how the role player chooses to participate provides an unpredictable, and interesting, human dynamic.  The realization that we are all affected by one another’s choices is intrinsic once the game begins.The transdisciplinary approach of scientific and humanistic perspectives was well represented in AME598.  As a representative of the latter, I marveled at the futuristic technologies integrated into our system.  The visitors who came into the black box theatre, viewing its arresting metal framed stage with cameras and speakers and bordered by a long line of active computers, were consistently impressed.  Feedback indicated we’d time-traveled to develop a systems thinking approach worthy of a Star Trek episode.   The AME598 scientists and engineers who’ve built this system may be accustomed to its appearance, its progress, its potential…. Not that this technology becomes mundane, but it should be noted that those of us who do not live in this black box theatre (which it certainly must feel like you’ve lived at DAR these past few weeks), have an immediate sense of awe and anticipation with this system.  Much like Xenakis’ philosophy of chance in music, the element of chance in our system is very exciting.Based on real world historic and current water numerics, our ‘stochastic model’ is very much dependent upon the unknown of four (or more) independent players.  A singular goal of any one player depletes the richness of the experience.  Per our Hutchins class, “Individual goals impoverish the experience…The experience exists out of you (the player) even though you lose direct control.”  Distribution is richer in variations, or creativity, once players interact and are aware of what other players are doing.  This interaction, with the visual and auditory feedbacks, as well as the futuristic atmosphere of our complexity system, emotionally engages the players.

Our system, to date, is a good example of a negotiation strategy simulator.  Moving forward, we could implement more data, including, but not limited to, water consumptive crop choices for the agricultural user, clearly defined choices to conserve, or not, for the residential user, detailed vitals of profit and life quality, water resources, price curves, and the certainty of climate change that now influences water management decisions which have planned for the expected desert occurrences of drought since 1903.  Although we could increase its complexity in many ways, we need to be careful of too many variables diffusing the tipping points.  As a student of Sustainability, I believe a model that emphasizes collective conservation to the end-users is a valuable tool. It will be interesting to watch our model played by ‘outsiders’ new to this futuristic system.  As David stated in our blog “Emphasis is on human-to-human interaction such that all actors are encouraged to dialog with one another to enact negotiation.  Can users cooperate through communication?  Is the relationship to the regulatory body antagonistic or cooperative?”   And if antagonistic, what evil lurks in the hearts of our players?…. Only the Wiis, the glowing sphere and the lovely Wedding Vase will tell…

urbanVitals

wSupply

vital.swf

MCres.swf  

I have uploaded some video files that may be used as a starting point demos in using the accelerometer balls. The MCres  and wSupply videos control the water levels using the x-y-z axis of the accelerometer. The vitals video uses the x-axis accelerometer to recharge the satiation and recreation bar graphs and the y-axis gyroscope to replenish the hygiene and aesthetics bar graphs as they all diminish in time. The urbanVitals is similar except it employs the y-axis accelerometer and gyroscope. The flash videos looks much better in the application, somehow they got messed up during the upload, perhaps a problem with the flash converter. The avi files look good but take a while to load.

A document that gives details of the design of SOball2x (RFID+IMU smart ball) internals has been posted. The internals for 2  balls, one orange and one green, are ready. Operation instructions begin on page 15.

In addition to the Accelerometer ball, SOball1, that we talked about using as controller simulating the Regulators, I also have the recently completed internal of the RFID ball that might fit in the bottom of the vase. If we do not have a controller built for the vase yet and there is interest in using the RFID ball internal, let me know. We have been waiting for materials to make the casting for the silicone outer surface. It is here now and we are getting ready to seal it in. You need to let me know before we seal it inside the silicone jacket if you want to use it. The current diameter without the outer layers is about 4.5 inches to give you an idea. Here is a picture.

NBC Nightly News report tonight (March 24th) briefly showed our Mediating Complexity system.   I couldn’t find a weblink of the broadcast, but did record it on my DVR.  Come on over.