Kategorien-Archiv Uncategorized

Erweiterung der akousmatischen Studie – 3D 5th-order Ambisonics

Dieser Beitrag handelt über die vierte Iteration einer akousmatischen Studie von Zeno Lösch, welche im Rahmen des Seminars „Visuelle Programmierung der Raum/Klangsynthese“ bei Prof. Dr. Marlon Schumacher an der HFM Karlsruhe durchgeführt wurden. Es wird über die grundlegende Konzeption, Ideen, aufbauende Iterationen sowie die technische Umsetzung mit OpenMusic behandelt.

Verantwortliche: Zeno Lösch, Master Student Musikinformatik der HFM Karlsruhe, 2. Semester

 

Pixel

Um Parameter zum Modulieren zu erhalten, wurde ein Python-Script verwendet.

 

 

Dieses Script ermöglicht es ein beliebiges Bild auf 10 x 10 Pixel zu skalieren und die jeweiligen Pixel Werte in eine Text Datei zu speichern.

„99 153 187 166 189 195 189 190 186 88 203 186 198 203 210 107 204 143 192 108 164 177 206 167 189 189 74 183 191 110 211 204 110 203 186 206 32 201 193 78 189 152 209 194 47 107 199 203 195 162 194 202 192 71 71 104 60 192 87 128 205 210 147 73 90 67 81 130 188 143 206 43 124 143 137 79 112 182 26 172 208 39 71 94 72 196 188 29 186 191 209 85 122 205 198 195 199 194 195 204 „

Die Werte in der Textdatei sind zwischen 0 und 255. Die Textdatei wird in Open Music importiert und die Werte werden skaliert.    

 

Diese skalierten Werte werden als pos-env Parameter verwendet.

 

   

Reaper und IEM-Plugin Suite

  Mit verschiedenen Bildern und verschiedenen Skalierungen erhält man verschiedene Ergebnisse die für man als Parameter für Modulation verwenden kann. In Reaper wurden bei der Postproduktion die IEM-Plugin-Suite verwendet. Diese Tools verwendet man für Ambisonics von verschiedenen Ordnungen. In diesem Fall wurde Ambisonics 5 Ordnung angewendet. Ein Effekt der oft verwendet wurde ist der FDNReverb. Dieses Hallgerät bietet die Möglichkeit einen Ambisonics Reverb auf ein Mulikanal-File anzuwenden.      

 

 

Die Stereo und Monofiles wurden zuerst in 5th Order Ambisonics Codiert (36 Kanäle) und schließlich mit dem binauralen Encoder in zwei Kanäle umgewandelt.

 

 

 

 

 

 

 

 

Andere Effekte zur Nachbearbeitung(Detune, Reverb) wurden von mir selbst programmiert und sind auf Github verfügbar.

Der Reverb basiert auf einen Paper von James A. Moorer About this Reverberation Business von 1979 und wurde in C++ geschrieben.

Der Algorythmus vom Detuner wurde von der HTML Version vom Handbuch „The Theory and Technique of Electronic Music“ von Miller Puckette  in C++ geschrieben.

 

 

Das Ergebnis der letzen Iteration ist hier zu hören.  

 

Smart Speaker Evaluation

Fragestellung

Im Rahmen des Designs eines Distributed Smart Speaker Arrays*, ob AoIP (Audio over IP) Lautsprecher der Firma Genelec hierfür geeignet sind und voraussetzende Kriterien erfüllen.

Kriterien zur Examinierung der Lautsprechermodelle 4420A und 4430A.

Die Genelec AoIP Lautsprecher bieten folgende Vorteile:

  • Übertragung von Strom,  Audio und Konfigurationsdaten mit nur einem Kabel.
  • Der Interne DSP der Lautsprecher kann via Netzwerk konfiguriert werden, und bietet 20 Filter an, zudem delay und Gesamtlautstärke
  • Es können sowohl Digital (RJ45 AES67) als auch Analog (Euroblock) Eingänge gleichzeitig mit interner Mischung verwendet werden
  • Es besteht eine offizielle public API.
  • Dimensionen sind klein genug (ca 15cm Breite, s. Spatial Aliasing), um kontinuierliche Arrays zur Holophonie (Wellenfeldsynthese, kompakte Arrays, ein/auslaufende Schallfelder, etc.) einzusetzen

Testziele:

  • DSP Parameter setzen (über API, ohne Herstellereigene Software)
  • Relative Latenz mehrer Lautsprecher (selber Stream)
  • Lautstärke/Schalldruck (aufgrund der kleinen Dimensionen)
  • Verhalten bei hohen Pegeln (Klirrfaktor, etc.)
  • Steuerprotokoll-Dokumentation der internen API
  • Latenz bei Steuerung der internen DSP Einstellungen über API

 

Vorläufige Evaluation

Steuerung

Die Lautsprecher werden per REST-Schnittstelle gesteuert. Der Hersteller bietet eine öffentliche API, die einige Einstellungen erlaubt. Die Software zur Konfiguration nutzt ebenfalls diese REST-API, jedoch mit mehr Endpunkten. Mit Hilfe von Wireshark konnte die Kommunikation direkt mitgelesen werden. Es findet keinerlei Verschlüsselung statt. 

 

Dank an Stephan Westfal für Durchführung der Tests.

 

Zylia 6DOF navigable audio

Zylia’s ZM-1 microphone is designed for three-dimensional recording. It allows the user to record sound scenes and separate single sound sources though a combination of microphone array technology and digital signal processing algorithms.

The spherical microphone has 19 capsules that each have an omni-directional polar pattern and that use Micro-Electro-Mechanical Systems (MEMS) technology. The durable casing features an LED ring status indicator. It is capable of recording at sample rates of 44.1 and 48 kHz and 24- bit sample depth. Gain adjustment controls in steps of 0.5 dB from −12 dB to +32 dB are provided. The microphone is 6.1 in high and 4 in in diameter. It weighs 15.5 oz. The ZM-1 has an integrated desktop stand and it can be mounted on microphone stands that use 1⁄4-in and 5⁄8- inthreads. It connects to computers using a micro-USB port in the base.

Zylia Studio is a software application for recording and processing from the Zylia microphone. It can be used to balance and pan the multichannel recording and to separate it into individual audio tracks for use in a digital audio workstation (DAW), if required. Two separation modes are available, a virtual microphone mode and a high separation mode. The first of these mimics a directional microphone and allows any part to be isolated from the three-dimensional space. The high separation mode uses an additional process to mimic the behavior of a close microphone, reducing ambient sounds and background noise and giving even better sound source separation. The audio can be stored as PCM WAV files or using a lossless compression that reduces the size by half. The software is compatible with Macintosh, Windows, and Linux operating systems.

A PRO version of the software is available as a VST/AU plug-in. This provides spatial filtering and signal separation tools within a DAW and gives the user access to virtual microphones with different polar patterns and characteristics in post-production. A number of different group and surround sound presets are provided.

An Ambisonics converter can process convert multi-channel audio into b-format higher-order Ambisonics. It also supports FuMA or ACN channel ordering and SN3D or N3D normalization. It is also compatible with YouTube’s ambiX and Facebook’s TBE formats. The converter is available as a standalone application or as a VST plug-in.

Machine Learning in Music: One Application with Voice and Live Electronics

Klicken Sie hier für eine deutsche Version dieses Textes.

Up until this past August, my impressions of what machine learning could be used for was mostly functional, detached from any aesthetic reference point within my artistic practice. Cars recognizing stop signs, radiologists detecting malignant legions in tissue; these are the first things to come to my mind. There is definitely an art behind programming these tasks. However, it wasn’t clear to me yet how machine learning could relate to my world of contemporary concert music. Therefore, when I participated in Artemi-Maria Gioti’s machine learning workshop at impuls Academy 2021, my primary interest was to make personal artistic connections to this body of research, and to see what ways I could interrogate my underlying aesthetic assumptions in artistic applications of machine learning. The purpose of this text is to share with you the connections I made. I will walk through the composition process of my piece Shepherd for voice and live electronics, using it as a frame to touch upon basic machine learning theories and methods, as well as outline how I aesthetically reacted to them. I will not go deep into the technicalities of machine learning – there are far more qualified people than I for that specific task. However, I will say that the technical content of this blogpost is inspired heavily from Artemi-Maria Gioti, who led this workshop and whose research covers the creative applications of machine learning in a much deeper way. A further dive into the already rich world of machine learning and music can be begun at her website.

A fundamental definition of machine learning can be framed around the idea of improvement through experience. As computer scientist Tom M. Mitchell describes it, “The field of machine learning is concerned with the question of how to construct computer programs that automatically improve with experience.” (Mitchell, T. (1998). Machine Learning. McGraw-Hill.). This premise of ‘improvement’ already confronted me with non-trivial questions. For example, if machine learning is utilized to create an improvising duo partner, what exactly does the computer understand as ‘good’ or ‘bad’ improvisation, as it gains experience? Before even beginning to build a robust machine learning algorithm, answering this preliminary question is an entire undertaking in and of itself. In my piece Shepherd, the electronics were trained to recognize the sound of my voice, specifically whether I was whispering, talking, yelling, or being silent. However, my goal was not to create a perfectly accurate recognition algorithm. Rather, I wanted the effectiveness and the ineffectiveness of the algorithm to both play equal roles in achieving the piece’s concept. Shepherd is a performance piece takes after a metaphor from Jesus in the christian bible – sheep recognize a shepherd by the sound of their voice (John 10). The electronics reacts to my voice in a way that is simultaneously certain and uncertain. It is a reflection, through performance, on the nuances of spiritual faith, the way uncertainty necessarily partakes in the formation of conviction and belief. Here the electronics were not functional instrument (something designed to be controlled by my voice), but rather were functioning more as a second player (a duo partner, reacting to my voice with a level of unpredictability).

Concretely in the program, the electronics returned two separate answers for every input it is given (see figure 1). It gives a decisive, classification answer (“this is ‘silence’, this is ‘whispering’, this is ‘talking’, etc.), and it gives an indecisive, erratic answer via regression (‘silence: 0.833; whispering: 0.126; talking: 0.201; yelling: 0.044’). And important for this concept of conceiving belief through doubt, the classification answer is derived from the regression answer. The decisive answer (classification) was generally stable in its changes over time, while the indecisive answer (regression) moved more quickly and erratically. Overall, this provided a useful material for creating dynamic control of the actual digital sounds that the electronics produced. But before touching on the DSP, I want to outline how exactly these machine learning algorithms operate, how the electronics learn and evaluate the sound of my voice.

Figure 1: Max MSP and Wekinator (off-screen) analyze an audio’s MFCCs to give two outputs on the nature of the input audio. The first output is from a regression algorithm, the second is from a classification algorithm. 

In order for the electronics to evaluate my input voice, it first needs a training set, a collection of data extracted from audio of my voice, with which it could use to ‘learn’ my voice. An important technical point is that the machine learning algorithm never observes actual audio data. With training and testing data, the algorithm is always looking at numerical data (here called ‘descriptors’) extracted from the audio. This is one reason machine learning algorithms can work in realtime, even with audio. As I alluded to, my voice recognition program is underpinned by two machine learning concepts: classification and regression. A classification algorithm will return a discrete value from its input data. In my case, those values are ‘silence’, ‘whispering’, ’talking’, and ‘yelling’. To make a training set then, I recorded audio of each of these classes (4 audio files in total), and extracted MFCCs (Mel-Frequency Cepstrum Coefficients) from it. MFCC’s are a representation a sound’s spectral energy calibrated to the range of typical human auditory perception, and are already commonly used in speech recognition programs, music-information retrieval applications, and other applications based around timbre-recognition.

I used the Max MSP library Zsa.descriptors to calculate my MFCCs. I also experimented with other audio descriptors such as spectral centroid, spectral flatness, amplitude peaks, as well as varying numbers of MFCC’s. Eventually I discovered that my algorithm was most accurate when 13 MFCCs were the only descriptor, and that description data was taken only about five  times a second. I realized that, on a micro-level timescale, my four classes had a lot similarity. For example, the word ‘synthesizer,’ carried lots of ’s’ noise, which is virtually the same when whispered as when talked. Because of this, extracting data at an intentionally slower rate gave the algorithm a more general picture of each of my voice-classes, allowing these micro-moments of similarity to be smoothed out.

The standard algorithm used for my voice recognition concept was classification. However, my classification algorithm was actually built using a second common machine learning algorithm: regression. As I mentioned before, I wanted to build into my electronics a level of ‘indecision’, something erratic that would contrast the stable nature of a standard classification algorithm. Rather than returning discrete values, a regression algorithm gives a new ‘predictive’ value, based on a function derived from the training set data. In the context of my piece, the regression algorithm does not return a specific voice-class. Rather, it gives four percentage values, each corresponding to how close or far my input is to each of the four voice-classes. Therefore, though I may be whispering, the algorithm does not say whether I am whispering or not. It merely tells me how close or far away I am from the ‘whispering’ data that it has been trained on.

I used a regression algorithm in Wekinator, a simple and powerful machine learning tool, to build my model (see figure 2). Input audio was analyzed in Max MSP, and the descriptor data was sent via OSC to Wekinator. Wekinator built the predictive regression model from this data and then sent output back to Max MSP to be used for DSP control. In Max, I made my own version of a classification algorithm based on this regression data.

Figure 2: Wekinator is evaluating MFCC data from Max MSP and returning 4 values from 0.0-1.0, indicating the input’s similarity to the four voice classes (silence, whispering, talking, yelling). The evaluation is a regression model trained on 752 data samples. 

All this algorithm-building once again returns me to my original concern. How can I make an aesthetic connection with these concepts? As I mentioned, this piece, Shepherd is for my solo voice and live electronics. In the piece I stand alone on a stage, switching through different fictional personas (a speaker at a farming convention, a disgruntled restaurant chef, a compilation video of Danny Wolfers saying the word ‘synthesizer,’ and a preacher), and the electronics reacts to these different characters by switching through its own set of personas (sheep; a whispering, whimpering sous chef; a literal synthesizer; and a compilation of christian music). Both the electronics and I change our personas in reaction to each other. I exercise some level control over the electronics, but not total. As I said earlier, the performance of the piece is a reflection on the intertwinement of conviction and doubt, decision and indecision, within spiritual faith. Within this concept, the idea of a machine ‘improving’ towards ‘perfection’ is no longer an effective framework. In the concept, and consequently in the music I attempted to make, stable belief (classification) and unstable indecision (regression) were equal contributors towards the musical relationship between myself and the electronics.

Based on how my voice was classified, the electronics operated one of four DSP modules. The individual parameters of a given module were controlled by the erratic output data of the regression algorithm (see figure 3). For example, when my voice was classified as silent, a granular synthesizer would create textures of sheep-like noises. Within that synthesizer, the percentage levels of whispering and talking ‘detected’ within the silence would manipulate the pitch shifting in the synthesizer (see figure 4). In this way, the music was not just four distinct sound modules. The regression algorithm allowed for each module to bend and flex in certain directions, as my voice subtly suggested hints of one voice class from within another. For example, in one section I alternate rapidly between the persona of a farmer talking at a farming convention, and a chef frustratingly whispering at his sous chef. The electronics moved consequently between my whispering and talking DSP modules. But also, as my whispering became more frustrated and exasperated, the electronics would output higher levels of talking in its regression algorithm. Thus, the internal drama of my theatrical  performances is reacted to by the electronics.

Figure 3: The classification data would trigger one of four DSP modules. A given DSP module would receive the regression values for all four vocal classes. These four values would control the parameters of the DSP module.

Figure 4: Parameter window for granular synth triggered when the electronics classifies my voice as ‘silent’. The amount of whispering and talking detected in the silence would control the pitch of the grain. The amount of silence detected in the silence controlled the grain’s duration. Because this value is relatively static during actual silence from my voice, a level of artificial duration manipulation (seen a the top of the window) was programmed. 

I want to return to Tom Mitchell’s thesis that machine learning involves computer improving  automatically through experience. If Shepherd is a voice recognition tool, then it is inefficient at improvement. However, Shepherd was not conceived as a tool. Rather, creating Shepherd was more so a cultivation of a relationship between my voice and the electronics. The electronics were more of a duo partner, and less of an instrument. To put this more concretely, I was never looking for ‘accurate’ results from the machine. As I programmed, I was searching for results that illustrated Shepherd’s artistic concept of belief intertwined with doubt. In this way, ‘improving’ the piece did not mean improving the algorithm’s accuracy. It meant ‘improving‘ the relationship between myself and the electronics. One positive from this approach is that the compositional process was never separated from the programming of the electronics. Both developed in tandem. The composing this piece brought me to the realization that creative applications of machine learning can be applied at every level of its discourse. If you ware interested in hearing a recording of this performance, a bootleg recording of the premiere can be found here.

References:

  • Artemi-Maria Gioti – composer and artistic researcher working in the field of artificial intelligence. 
  • Wekinator – free, open-source software created by Rebecca Fiebrink that uses machine learning to create musical instruments, game interfaces, computervision, and other tools in sound and animation. 
  • Zsa.descriptors –  library for real-time sound descriptors analysis for Max MSP developed by Mikhail Malt and Emmanuel Jourdan.
  • NYU Music and Audio Research Laboratory – Free online resources and datasets.
  • AIMC – conference on artificial intelligence and musical creativity.
  • ml.star – machine learning library for Max MSP.
  • OM-Pursuit – Dictionary-based sound modelling for computer-aided composition in Open Music.