cognitive model for program auralization - 9/6/12 1:22 AM / 1

A Model of a Model

AURALIZATION: Can include sonification. Pattern matching occurs here, and ideas of how to exaggerate contours of pattern will be explored. Can audio delineate dimensional space in the brain? Can the structure of algorithms be auralized via audio cues indicating arrows, associations and space. Is mapping data outputs of algorithms enough? Or is it necessary to find ways to build a language of understanding aural logic in order for some systems to be aurally communicated? In this sense developments in aural displays, earcons, and aural navigation systems could be helpful. Time is also a variable in this module. The Auralization process may determine multiple temporal relationships inside of datasets before or after cognitive mapping - and maybe need to work with cognitive mapping in order to decide where the cutoff points are for interpreting temporal frames. COGNITIVE MAPPING: maps time into temporal frames of the brain. The cognitive mapping process could be modeled off the way that the brain maps perceptual streams into temporal frames in the brain, understanding timbre, pitch, rhythm, spatialization and structure as distinct parameters of sound. Treating outputs from the system models as perceptual streams. Essential when the performance is an audio output, may be less important when using audio scores with performers - or maybe the cognitive mapping that performance language is doing should really be inside this module? PERFORMANCE LANGUAGE: The performance language module interprets data from the cognitive mapping module and/or auralization module in order to translate the information into a performable language. When the performance language is generated, different methods are

cognitive model for program auralization - 9/6/12 1:22 AM / 2

necessary to consider. Time and meaning play a big role hear. What do performers need to hear in order to do what is intended. The performance language I hope to develop first is an audio score language that will be transmitted to performers via headphones. In this scenario, semiotics and instructions will need to be timed and coded in order to be interpreted by the human brain. An cognitive understanding of how instructions are processed by the performers brains will need to be explored - in addition to semiotics and icon-ic ideas. Similar to the Auralization module, developments in earcons, audio displays, and audio navigational systems. In this model, one advantage is that it can be combined with the spoken language that the performer already knows, but in some instances, complicated instructions will need to be simplified into audio cues, which may require training. The performance module will also be capable of generating training exercises to prepare the performer for particular cues that will be used in a performance.

=====================================================

The data flow between the above modules and the system model(s) is one of input and output, simple enough. Input may be used to create feedback systems, evolutionary systems, and systems that change due to real-time input.

SYSTEM MODEL(S): can output a dataset generated by an algorithm. This could be done in one giant dump, or could be chunked into iterations or recursions, or exchange data with the rest of the program in real-time. Any system, algorithm, musical piece and its rules, performance piece and its rules. The system could also output the rules themselves as data that could feed

cognitive model for program auralization - 9/6/12 1:22 AM / 3

directly into the AURALIZATION module. Then these rules could be applied to audio processing or another type of processing where it is more useful to know the rules of the system than to have just the data output from a system. This may apply for systems that are bottom-up and require subsystems to interact in order to create emergent patterns, etc. In the process of moving the data from the system model module to the other modules, it may be valuable to keep time related to the data or rules, or it may be useful to freeze the data or rules in a timeless state, and have the later modules interpret time appropriately for a particular performance.