EEL Scripting for audio

I did my first test getting an EEL DSP script up and running under Genome. Only a few minor syntax changes were necessary to get the script to compile under Visual Studio (though it did take the better part of a day to get it working). Overall I am happy with the features of the language and the fact that it has so few dependencies (and is generally written in ansii C, which hopefully means it's fairly fast). Initial speed tests were a bit disappointing. A simple FM synth (one operator, one modulator) ended up taking about 15-20% of my cpu time. Ideally I'd like a simple EEL module to take a percent or two. A C++ version of the same synth would take 0.1% or less of the cpu. So, I am currently looking into optimizing eel to tailor it to my needs a bit more. The eel core is fairly easy to understand (despite it's overly aggressive use of macros). These days I have been using AMD Code Analyst, which is a nice, free version of VTune, for my optimization needs. The latest version runs right inside visual studio and doesn't require any special instrumentation. You just run the app and it starts collecting results. For those unfamiliar with these kinds of applications, basically they can tell you which functions are consuming cpu resources and then allow you to go line by line and see which lines are taking up the most time. This is essential for helping decide how to go about optimizing a large and complicated program. Obviously it doesn't make sense to optimize everything. Your code will become a mess. You want to target your optimizations to the inner loops - the things that are done the most often.

Most of the EEL execution time is spent looping through instructions, computing arithmetic operations and copying variables. I spotted some areas that could see huge gains in the arithmetic function. Basically there is a gigantic case statement which checks every possible combination of operation and variable type. Instead of one monolithic case statement I could see this being broken down into several nested comparisons and have at most 5 or 6 comparisons per operation, instead of the dozens that an average operation would take now. I also plan on moving up common operations and moving down uncommon ones. I am debating whether the instruction set can be simplified. The smaller the instruction set, the less time I'll be spending figuring out which instruction to execute. For my audio/dsp needs I am going to be operating on floating point numbers most of the time. Some variable types or operations might be removable. So, it should be interesting work to try to speed this up. Already I am learning a lot about VM's and the internals of scripting languages just by going through the code.

UPDATE:
In my searches I found that some compilers will convert a case statement into a what is known as a 'jump table' or 'branch table', which is basically an table of jump locations keyed by the variable you are evaluating in your switch statement. This is much faster than evaluating a huge case statement. Not sure if my compiler is doing that - if it is, then my optimization hopes are dimmed.

Right now I am toying around with making my own ASM-style scripting language with a very reduced opcode and ability set which. The language would be very specifically focused on processing streams of floating point audio data. I am hoping that these two things will make it a lot faster than EEL, which is fairly high level and provides a lot of functionality that isn't important for processing a stream of audio. It might be possible to combine EEL and this ASM language to cover both the high-level and low level bases. The language follows what I posted previously. I coded the 'compiler' last night and I'll work on the VM tonight. I'll post when I get some results.

Genome Work Log

• More event refactoring
• Container control inputs now possible
• Bug fixes
• friendlier names for container inputs / outputs..

Genome Work Log

• Working on modifying some low level stuff with Genome's Control Events. When the dust settles, Genome will have hooks for automation.
• Part of this change will also allow for 'meta patterns'.. Ie, a pattern that may be composed of several different sub patterns patterns.
  
• As per the comments in my last post, Ben from Curverider was kind enough to inform me that Elgg is in fact close to it's 1.0 release (though no specifics yet on when that will be, and they haven't set me up as a beta tester.. *cries* :) I am very closely watching developments with both Elgg (which is hitting 1.0) and Social Engine (which is hitting 3.0). Based on Screenshots of Elgg 1.0, they have definately made some great UI improvements. Also, the underlying code changes promise to make writing plugins even easier and allow for easier sharing and displaying of user generated artifacts. The new social engine is also looking extremely sexy over at it's public beta site even though no plugins have made available yet (elgg people: put up a public beta!).
  

Genome Work Log

• Select / drag, Select / resize controls on control surfaces now working
• Began implementing OSC control for parameters.
  
• Created OSC classes using Juce's networking classes. Had to add some stuff to Juce's DatagramSocket. Will post on the juce forum for Jules to consider adding.. My classes are a simple wrapper for Oscpack
  
• Evaluating SocialEngine for use as my website software for the upcoming Genome community site. It's light years ahead of Elgg in terms of usability, but the discussion forums and overall comment posting system is pretty weak (Elgg's is weak too). Development on Elgg is also going way too slow. Social Engine also offers a great admin with a good amount of control, and a decent amount of quality plugins. Best solution might be SocialEngine + Phpbb (there are some tutorials for integrating the two). In my view a lot of social network sites fall short in the forum area. They focus too much on profiles and not enough on discussions. Some of the best community sites I've seen are little more than a forum. Social engine V3 is coming soon, so I will wait for that before purchasing it. It's a little on the expensive side, as far as website software goes, but I've seen companies charge orders of magnitude more money for basically the same thing (by doing customization work).
  

Genome Work Log

• Continuing to refine ability to select and move controls on the control surface. Still needs ability to resize selected controls..
  
• Mac build is working! Created Mac project for XCode, got my development environment set up. Currently using svnX for subversion.
• Numerous fixes necessary to get mac version copiled
• Removed Fluidsynth stuff for now since I was having some trouble compiling it on the Mac. May just use Juce's sampler in the future.. Will work on beefing it up though.
  

Now officially a cross platform developer - update

I got Genome up and running on the Mac. It's a pretty sweet feeling to see it looking and functioning 100% the same as the PC version (all with very little work needed to get it there). I even had the Mac and the PC versions of Genome connected up over the network, working on a song together. Awww. ;) Anyway, it is pretty cool that there were no issues with networking either. Everything worked on the first try. Woo! So yeah, I am very glad that I decided to switch to Juce. :)

Now Officially a Cross-platform developer

I recently procured a used G5 Mac and have been using it to port Genome to MacOS. Since I've used Juce for pretty much everything, getting it running has been pretty easy. Mostly I just had to clean up some dubious syntax that worked in Visual Studio but not in XCode. I now have genome opening on the Mac and just need to fix one issue before I can test it out. It's only taken 5-6 hours to get it ported.

Juce is awesome. I'm had been using some std:: classes (like vector) for a few things, but I am probably going to replace those with Juce equivalents too. The juce versions usually are easier to work with, offer more functionality, and have less porting issues. Plus they offer the ability to run them thread safe - something you don't get from the regular std:: classes. Also, verison 1.46 of Juce just got released that adds some new goodies. Nice to see that Jules is still working on it and making improvements.

So far my Mac experiences have been pretty positive. I like all the little extras you get with a mac - built in bluetooth, webcam, etc.. Plus I love the hardware - it's small, quiet and sleek, unlike the massive, loud, monstrosity of a PC sitting next to it. If had one criticism of the OS it would be that it always feels like am juggling 10,000 windows. I'm used to looking on the taskbar to see what's running (wheras the mac replaces it with the app bar). Anyway, I'm sure I'll get used to that once I figure out all the keyboard shortcuts. Overall, thumbs up.

Genome Work Log

• Created a 'user settings' .XML file for general, user specific settings.
  
• Did more on control surfaces. They can be opened in their own window now and state information is saved in the user settings .xml file.
  

Genome Work Log

• Worked on Control Surfaces. Mostly working now. Also put them in a resizable layout with the module view. Can be either horizontal or vertical. Working on saving current state to a user preferences file. After that I'll work on getting it to open in a new window if the user clicks something.
  
• Added keyboard shortcuts for almost all the menu commands
  

Genome Work Log

• Control surfaces now work again. Still need to work on adding a few things like the ability to open it in it's own window, the ability to move / delete controls, etc..
• Control surfs are one of the last big ticket items on my list for the moment. OSC support is next.

Genome Work Log

• Added clipboard window
• Made Chat window only come up with you are network connected
• improved selection (for copy and paste)
• added 'show clipboard' menu option

Genome Work Log

• Resumed work after hiatus (always a challenge)
• Selection of controls and copy/cut/paste now work again
• Todo: some improvements to selection, then Control surfaces.. After that OSC control.

I'm really looking forward to OSC (Open Sound Control) since I think it will really open up the app to a high degree of external scripting and control at a very low cost for me to implement. There are OSC libraries in Python, stuff for Flash, Midi -> OSC bridges, etc.. People will be able to create custom instrument interfaces, dj mixers, algorithmic composition tools and more with OSC (if they are up for a little programming).

Genome Work Log

• Finished re-coding the network module in Juce. Basic tests indicate that it is working ok, but more in depth testing will have to wait until I finish re-adding the gui elements to support it (doing that next). Unit tests for this module would be really useful.
• Coded a 'ChatWindow' component. Basic IRC-style chatting is working. In the future I will expand it to allow for different text colors and graphics.
• misc fixes
• overall the network aspect I think is going to be the 'killer app' for genome. I am excited to see it up and running again.

Genome Work Log

• added track delete
• started converting the last external dependencies to Juce classes. Also working on converting all std::string's to Juce Strings. Should take a few days but when I'm done I should have an application that compiles on Mac, Linux and PC with no major changes (since I'm using Juce for everything..).

Genome Work Log

Refactored some stuff with the main song view to separate the track info area (so it scrolls properly now). I thought I would have to do some tricky component juggling to get the pattern selector to appear over top of the song tracks but it turns out that Juce already has a really easy solution - I just used 'addToDesktop' which basically moves the component into a top level window. This is what is used for the popup menus and tooltips. That's the nice thing about Juce - the base Component class already has almost every feature you could possibly dream of built in and ready to use in an easy manner.

Genome Work Log

• Added mute / solo buttons
• made track names renameable
• added ability to delete patterns

Genome Work Log

• added pattern selector.
• added colors for patterns
• section 'cue' working minimally now
• clicking a marker sets the playhead to the start of it

(still haven't put any work into prettying things up yet ;)

Genome Work Log

• Added numerous things to the tracker module. It now resizes to fill the viewable area. I also added some more keyboard shortcut keys and the ability to edit pattern properties (length, #tracks, name, etc..). Still a few things left to add to the tracker then I'll move on to adding the missing functionality to the main Song view.
• The FileParameter is now implemented in juce so the Fluid .SF2 module is working again.

Stuff..

Been a long time since I posted anything so I'll summarize a bit.

I'm still horribly addicted to TF2 - I'm hoping that I will start to get sick of it soon, but so far it hasn't happened yet. The last time I was this into an online game was Quake2 and that was nearly 10 years ago. Stay away from it!

We had our open house at my workplace - my multitouch table was on display and it was a pretty big hit. I've since added a second 144 LED IR illuminator to the table to help get a more even IR distribution. The table works and is very sensitive even in daylight (there's a window next to the table). Plus I can leave it on all day without having to recalibrate or otherwise mess with it. I'll try to post some pics..

I picked up a usb-bluetooth PC adaptor yesterday. I'll probably try messing with some Wii-mote stuff at some point. A Wiimote -> OSC application should be pretty easy to make.

I also managed to get 1 or 2 things done on Genome though I am still trying to get back into the swing of it. It's a large project with a long list of todo's which can make getting started discouraging. Before, I was able to meet my goal of doing one thing on it every morning (working for like 1/2 an hour).

Genome Work Log

It's been exactly one month since I started porting my code to Juce and Genome is finally making noise again. Today I got some basic things working for the Tracker module which enabled me to check to make sure all the underlying sound processing stuff was still working, and it is. Still a huge number of things left to do to get back to where I was, but I'm making steady progress.

Genome Work Log

I'm still working on porting everything over to Juce. I've made some pretty good progress though. I've got the SongView and ModuleViews in there and doing stuff. Next I need to get some kind of note generating machine working before I can actually hear any output. All the low level GUI stuff is done and it's just a matter of recoding the old stuff as necessary. I've already seen a bunch of benefits to using Juce. Writing new GUI components is very easy and Juce provides a lot of helper classes (for instance, dragging and dropping components is very easy to accomplish). Also the fact that Juce's Component class is very flexible and already has tons of functionality built in makes it very easy to get the behavior you are looking for. In most cases you just need to override a few methods to make a new control. There's still lots of little things left to add but a lot of the groundwork is laid now.

Genome Work Log

• Fixed an issue where module controllers weren't getting deleted properly and some other bugs that were uncovered by that
• Got bank / preset selection working for the SF2 player

To do:

• Wav loading for the SF2 player

Genome Work Log

• Added ability to record your song output to .WAV (and other formats) using libsndfile. Now I start posting some Genome mp3's. I still need to work out some details for how this process will work.
• Created a 'Task' class to make it easier to do multi-threaded operations. The .Wav recorder is an example of a Task.

To Do:

• Continue adding more modules

Genome Work Log

• Added SF2 playing support via the great (and open source) Fluid Synth library. Still need to work out some things like giving it a GUI, and deciding whether to store the full .SF2 file in the song or just the path to the file. Also, it would be good to write a .WAV loader for Fluid since then I could use Fluid for all my sampler needs. Fortunately it looks like fluid was written with this in mind so it shouldn't be too hard.

It will be pretty awesome once sample playing is in there.

Genome Work Log

• Added Stereo Reverb
• New screenshot

Genome Work Log

• Added EQ module
• Added Phaser module
• minor additions to the api

Genome Work Log

• Added AudioViewer control. Mainly for debugging, but now any module can have a handy output viewer. Also made a simple plugin using this that you can plug in wherever you like in your signal path.
• Debugged some threading issues. Ended up being something simple but atleast I refreshed my memory on how threading was working in Genome.
• Bug fixes

To Do

• Debug pitch modulation on the basic osc module
• Add more modules

Genome Work Log

• Added knob control
• Added Exponential Envelope module
• Debbugged some stuff with the Builder module

To Do:

• Still working on some low-level debugging before I can get back to doing more modules.

Genome Work Log

• Made it possible to rename modules
• added rename to gui for Tracks and Modules

Todo:

• Rename for song markers
• Work on dialog boxes - song overwrite warning, edit warnings..

Genome Work Log

• Improved interface for modifying patterns in the main song view. Now it properly clips/deletes overlapping song patterns when dragging and resizing.
• It's now possible to edit the song tempo.

Next:

• Rename / Overwrite dialogs

Getting closer! There's a few more general interface fixes to do then I want to work on .WAV export and adding some new modules so I can actually make some songs with Genome and I can post some samples.

Genome Work Log

• Made New/Open Song/Save Song work on all menus
• Added default settings. All modules now load a preset called 'default' from their library on creation. This also means you can easily customize your default settings..

Genome Work Log

Got a few more things done on Genome this weekend:

• Added Mute/Solo operations to tracks in main song view
• Converted Library format to XML
• Saved song Looping state
• Displayed Marker names
• Fixed keyboard operations for labels (cut, paste, copy)
• Fixed a crash bug when switching back to the main song view

Next up:

• Need a shortcut to add a section label between two labels
• Need to be able to rename section labels and song tracks
• Songview context menu (for creating tracks)
• Track context menu (for delete / rename)

Genome progress

I've circled back around to working on my music app Genome. Recently I've added in song section labels. These are basically time markers that allow you to easily select and drag around sections of the song. They also allow you to loop a section of the song and cue up other sections of the song to play when the current section completes, so in essence it allows you to mix things up live like Ableton. I also started incorporating XML into some of the modules as a way to store data. This will allow me to extend things without breaking the file format and also allow the main parts of the song to be exported (could be useful somehow. ;) Still tons left to do, but I'm making good progress.

Long time..

It's been a long time since I've been able to post anything, but I really haven't had too much to blog about. I'm working a bit on strategy game in Flash 9 - think of it as Master of Orion meets Weird Worlds meets Monopoly. I'll post more when I have something worth showing. My goal is to keep it pretty simple so that I actually finish it. I also have been thinking about getting back to working on my music app Genome since both Buzz and Buze leave something to be desired under Windows Vista (and that something is 'not crashing constantly').

I am still working on building a multitouch screen for my workplace - I should have the last bit of hardware this week (a first surface mirror for projection), so I am hoping I will be able to start putting it all together. I am interested in trying out some different things like seeing how well the CityWall approach works and maybe trying to elevate the LED's slightly like the Tangent table. I ordered a big piece of silicone rubber but it sticks to my acrylic too much to be useful. I wonder if a thinner piece attached to a more rigid piece of plastic would stick less. I'd also like to try some of the plasti-dip on a thicker piece of the mylar drafting paper. I think that might work well.

Stuff, Zelda:TP Thoughts

Things have been crazy hectic at my new job. As a result, I've been too stressed out to work on Genome or Touchlib after work. Hopefully things will calm down soon and I'll be able get some stuff done again.

I have been playing a little Zelda: Twilight Princess to calm down a bit. So far it has been a really enjoyable game. I was a little hesitant to buy it after FFXII since I didn't want another game that forces you to play for hours or requires you to cover the same territory over and over again. The last Zelda game (Windwaker) suffered from this problem in that the sailing sections ended up being very time consuming and were pretty boring to me. I'm happy to report that Zelda: TP doesn't suffer from any of those flaws - the level design is very tight and there is little or no need to re-cover old terrain so far. Also, you can save your game wherever you want. The puzzles are clever but not so tough that you can't figure them out on the first or second try. It's a great example of how games should be designed. Your goal is always clear, you receive constant feedback when you do something right, actions that your character can perform are always highlighted on screen, dying is only a temporary set-back (you can restart at the beginning of that room), hints are available if you need them, and it's usually obvious when you need to use a particular tool in your arsenal. Also, the levels themselves act as tutorials for learning how to use your tools when you acquire them - they gradually show you all the different techniques you can execute. It's a far cry from the original Zelda when you had to bomb the fourth rock from the right on a specific map screen (a rock that looks exactly like all the other rocks...). Also, all the items provide different abilities rather than just being stronger versions of the same weapon as most FPS games are. This way the game is always throwing at you fresh variations on the gameplay. Overall, I highly recommend it.

Today's Work

I got a bunch of stuff done on Genome's tracker module. You now scroll patterns larger than the screen, and you can box select with the mouse. I'm planning on putting my own spin on the tracker in an attempt to bring it into the 21st century. For one, I'm making it horizontal instead of vertical (mirroring pretty much every other music program out there). Mouse interaction will be much better than in the past (read: Buzz). You will still be able to do everything with the keyboard, but there's no reason why you shouldn't be able to do everything with the mouse too. Gone is the fact that deleting a cell moves all the cells beneath it up. I always felt that this was annoying behavior. Sometimes it does come in handy for moving blocks of notes around, however a lot of the time you just want to delete something without messing up your timing for everything else. Instead, I will leverage box selection and allow users to move around blocks of notes with the keyboard or mouse, or I will add another keyboard short cut (like ctrl-delete) to do the old behavior.

On the touchlib front, on suggestion from some other #ftir'ers I checked out Oscpack and FLOSC. I was able to get both up and running fairly easily so I think I may take a shot at adding OSC support to Touchlib. I will use the TUIO protocol which is already used by the reactivision software. A handful of programs already support this protocol - Audiomulch for one - so it could expand the available applications that can work with touchlib. Also, it will alow for development in Flash (with FLOSC) which is exciting to me as a Flash Developer. This could be a good option for people to start making apps without having to mess with C++.

Genome Progress

Today I got the Library working. There are still a few things left to do (like allow for sorting), but the basic functionality is there. Now I think I will focus on improving some of the major modules and getting things to the point where you can start making simple songs, before I loop back around and refine things. The first will be the Tracker module which is just one of several input devices you can use to sequence your music. I'm also planning on doing a piano roll, 808-style drum sequencer, and maybe one or two experimental interfaces.

Genome Progress

I fixed up some more GUI related items today. I still need to do a few more GUI related things and then I will work on the Library. The Library will be used for everything from songs to presets and it will work similar to iTunes' song list, allowing you to sort and filter your presets and songs by category, genre, author, rating and more. I will also allow the option of preset 'Variants' which are variations of a preset. This way you can have several versions of the same pad sound (different effects, filter settings, etc). It will also be useful for the Builder module where you may want to save different settings with the same basic construction.

The Library will also abstract the file system - the user just says 'save preset' and provides some metadata. There will be no need to pick what directory you want to save it in and no directories to organize or misplace. Genome handles all that behind the scenes. The library will also give the option to publish your presets or songs online and to download other people's work from within the app.

Genome, Collaborative music applications

Did a bit more work on Genome yesterday which hopefully means I've broken my spell of 'Programmers Block'. The problem is that genome is just a big application. Even though it's fairly cleanly written I still have trouble remembering where to look for things in it's 200+ source files. Some diagrams may be in order.. My next goal will be improving the GUI and making it a bit easier to use. Once I pretty things up, I'll post some pics.

I listened in on some interesting online discussions with Paniq, who is also working on a sequencer in the same vein as Buzz. Both of us would like to make a music program which encourages community via the sharing of song files. In the early days of computer based music, people could share songs rather easily, since each song contained all the patterns and audio samples necessary for the song. Sharing songs allows people to learn from each other, borrow things they like, remix other artists' work and also contribute to the community. These days we have plugin (.DLL) based instruments, some of which are commercial (not free). So we can no longer count on everyone having what it takes to play the song on their computer. Almost no one exchanges songs written in Cubase or Cakewalk. With Buzz, all the plugins are Free, however keeping up to date with the hundreds of plugins becomes a chore in itself. There are a couple solutions to this problem.

1. Plugins could be free or commercial. We would automate the process of purchasing / downloading the necessary components by having the program interface with a central website. This would allow a la carte purchasing/downloading of instruments, presets, etc all within the application. Think of it as i-tunes (or Steam) for music sequencers. This could also help with the software piracy situation since the server tracks your purchases. Free plugins could be automatically downloaded from the server if the song requires them or you could select them from a catalog. The application could check daily for new user-authored components and download them in the background.

2. Do away with traditional .DLL plugins. Plugins could be authored with a scripting language and included in the song. Paniq plans to use a system based on Faust which is a script language that is compiled into C++ code (which is then compiled again..). The method I plan to use is to allow users to build instruments similar to how the Nord Modular and Synthedit work - by visually connecting modules. It's true that this approach doesn't allow the fine control that you would get from scripting or coding an instrument - you can't write your own filters or oscillators, but I think it gets you 90% of the way there and it's more accessible to the average user. By offering a variety of modules you can make up for some of it's limitations. I already have source code written for everything from virtual analog oscillators, additive synthesis, FM and just about every type of effect. It will be very easy to turn this code into modules. Of course you won't be required to build your own instruments - Genome will ship with all the pre-requisites. This is just for the people who want to use it.


Scripting still has a place - it would be interesting to allow the creation of custom control surfaces which contain scripts. You could create surfaces that allow for algorithmic note generation, arpeggiation, unique note layouts, etc.. These surfaces could control multiple modules at the same time.