I have this algorithm...

Algorithms can solve two kinds of problems.

1) Problems with verifiable answers and...
2) Those without verifiable answers (or the verification is proved by statistically sampling)

Verifiable answers:

Algorithms in this category address problems like, find me all the numbers in this list, or even, find me the fastest route between point a and b.

The value in the algorithm is not the correctness, it is easy to find the answers to these problems, rather the value in these algorithms are the speed with which they are executed and the amount of data you have to solve your problem.

So for example there are many different kinds of sorting algorithms - they all correctly sort, but some are faster and some use less memory. This is where the ingenuity or value of the sorting algorithm comes in.

For the mapping example, it is not difficult to map the fastest route between two points. But you could imagine an algorithm that used realtime traffic data would provide faster routes than other algorithms. The answer these algorithms may be different, but their actual processing of information - the algorithm itself - may be the same. The only difference is the input data used by the algorithm.


Now onto the algorithms with no answers -
these are the interesting algorithms that answer questions like - find the top 10 sites on music, or recommend movies I'll like, or find me a date.

In the case of search - the answer can be the top 10 sites on music are the top music sites that people link to (google's algorithm), or the top 10 sites that my friends like (a facebook solution)

In the case of recommendations, we would have an algorithm like that produced by the netflix prize, a statistical sampling of user preferences run through a barrages of matrix filters. We could also have the pandora algorithm, create a genetic code for a piece of music based on frequency, btm, etc and then find other similar genetic codes.

For dating there is a little less innovation out there. Most sites ask a set of questions (perhaps developed by behavioral specialists), and match those with similar or compatible answers, or let users shop for dates by displaying the answers to these behavioral questions.

But that is it go forth and create algorithms for problems with no answers

transducer

i feel like on of the big new media memes is transduction.

For example convert stock market feeds into audio sounds, or convert twitter feeds into graphical visualization.

Data visualization and audio visualization are forms of transduction - translating from one form into another. The issue with transduction is how do we perform the translation. This is where the real art of data visualization or data sonification takes place. There are infinate numbers of ways to map the frequency of #brit on twitter to a visual cue. What signifies frequency (line size?), what signifies subject matter (color?), where do you place the visualization on the screen (existence in different social networks?). All of these are arbitrary and the criterion of success is how transparent the visualization is for understanding the data, or how aesthetically pleasing the result is.

I have been writing an application called doh rae tweet which lets you compose music via twitter. Here again the mappings are arbitrary. an 'A' is a note 'a', but a space is S (it could just be a space). The default time signature is 4/4 unless you start your tweet with something else like 8|8. and you adjust the beat of a note by + or - so A- (in 4/4) would be a half note. This is all arbitrary and it is not necessarily transduction but translation. Not transforming one type of thing into something else, but representing something (musical notation) in something else (140 characters on twitter) . In this way it is much more similar to writing a compiler than writing a data visualization.

However doh rae tweet can also can be used as a transduction engine. I can feed through random twitter feed and use the music compilation engine to hear what that tweet sounds like according to my 'compiler' Although it probably wont sound very nice. Most letters will not have an audio mapping. So for this sort of application I would want to replace my compiler or lexical engine with another. Ideally anyone could create their own mapping. I think it would be interesting to do this via a GUI and feedback loop so you could adjust your mappings visually as you hear the effects of your changes.

One final though - I also want to use this mapping for other notation schemes. One that seems a natural is chess. With the 8x8 chess board mapping to the diatonic scale. However, this too would need its own mapping. ( Also what do you play, to just play the chess piece that is moved - or all the chess pieces. ) I went online to see if anyone has attempted sonficiations of chess notation and found the following:

Halfbakery This has some links relating chess to music
Chess History A list of articles and historical documents connecting chess with music (not very informative but interesting from a historical standpoint).
Music and Chess A great website with all sort of information linking chess and music. I got this from that site: "The Oxford Companion to Chess is a comprehensive encyclopedia of chess. It contains articles on history, terminology, chess players, and the relationship between chess and other subjects such as music, art, theatre, literature and philosophy. Many of the terms listed in this book are also musical terms. For example, in chess, a person who creates puzzles and problems to be solved is called a composer, and two different sequences of moves that lead from one given position to another are said to be related by transposition. Some other terms that are used in chess and music are: play, piece, notation, score, tempo, theme, variation, development, minimal composition, round, major and minor, position, second, retrograde, mirror, attack, anticipation, phase and echo."

Blown away by The Sound and the Fury

Ok - so when I first started the sound and the fury I had no idea what was going on. 'I think they are on a golf course' I told my business partner Paul. 'I have no idea what is going on.' Half way through the book I felt that I was waisting precious moments of my life reading a piece of crap. However, at some point - probably during Quentin's chapter - I reevaluated.

Why did I reevaluate?? I dont know. Perhaps the language... there is a certain lyricism - and i am a sucker for dialogue. But, at the end of the book a few things struck me...

1) Such a wide expanse of time was covered in such a 200 page book. The fragmented storytelling conveyed more information and feeling than a 1000 page tomb of today's magical realism.
2) What is the story?? The book is really about the construction of narrative, or the construction of myth and meaning. At the end of the day - when you read a story or a myth everything seems so tidy - but the construction itself is messy. While the story is being written you have no sense of the whole completed version - the sound of the fury is like a story writing itself
3) Roshimon - Whose point of view is correct? Seeing the compson family through the eyes of the different narrators in a stream of consciousness voice bring home the fact that every story is told from a point of view.
4) It is a mind fuck - the story seems like a puzzle. Not a post modern puzzle, where the whole point of the novel is the puzzle (like paul auster - of whom I am a big fan). Perhaps puzzle is the wrong word... It is coded. I feel like if i diagramed the whole novel out - parsing out the different stories by the different narrators at different times in the life of the characters I would get another story. There is a second story hidden in the story.
5) This is a modern novel. It struggles with modern themes (and it was written in 1930s I think). Sexuality, Economics, Race, Class - the treatment of these issues are not preachy but are integral to the lives of the character. this is great storytelling - bringing up weighty issues as part of the characters struggles

Anyway I am inspired - I may experiment with this sort of writing for some of my short stories. Also reading Sarte's critique of faulkner - I'm into sarte as literary critic (just was gifted book 5 of the family idiot about flaubert).

Local build of ffmpeg

This was going to be a highminded post about fairy tales and morphology but instead i am going to record my trials and trivails of running ffmpeg on amazon's map-reduce.

So - what i am doing is adding a cached directory of ffmpeg to every map-reduce computer i use


#!/usr/bin/bash
./elastic-mapreduce --create --stream --alive --name $1 \
-c credentials.json \
--input s3n://13bit.videoout \
--mapper s3://13bit.scripts/compression.py \
--reduce s3://13bit.scripts/reduce.py \
--output s3n://13bit.videoout.ts \
--log-uri s3n://13bit.log \
--cache-archive s3n://13bit.scripts/ffmpeg.zip#ffmpeg

in order to build ffmpeg i started up a debian sqeeze box via aleatic ami (ami-daf615b3) http://alestic.com/. I then had to update apt-get (apt-get update), install git, download ffmpeg and related libraries and run the ffmpeg configure switch --prefix to set the install directory. I am now zipping it up and testing it on another instance of the squeeze debian - at which point i will start up the map reduce process again

i am also looking at boston terrier art on etsy (thanks to Jane Kim)
http://www.etsy.com/listing/37552859/boston-terrier-dog-art-original-painting
I am doing this while my boston terrier sleeps and farts next to me.

Living the Dream

Last night Paul and I attended a fascinating lecture on the postal history of the Congo Free State at the NY philatelic society. This is research for our latest documentary. There is nothing like listening to a lecture by an expert. Not all fields are capable of producing experts - Literary theory for instance. You can only be an expert is something that exists, that has a counterpoint in reality. You cannot be an expert in unicorns, however you can be an expert on the literature of unicorns, the archaeology of unicorns, and the symbology of unicorns. Likewise you cannot be an expert on literary theory, but you can be an expert on the thoughts of literary theorists. However I think expertise in something like stamps of the Congo is much more worthwhile...

On another note- I ran across the following art project by cory arcangle - it culls the internet for blog posts that begin with i'm sorry. I am a big fan of his video work, especially mario clouds video & the the one where he just projects fields of color from the projector. However, this 'sorry' project is sort of in bad taste. What is it? A project mocking people attempting to maintain blog? I am not quite sure what its purpose is. I doubt it is to be inspired by the heart felt apologies - as cory writes on his blog. So what exactly is the point of this project? to call attention to the dead blogs in the world, to show how apologies have become meaningless, is it supposed to just be funny, it it supposed to show how pathetic people are, how people have no imagination (except for cory)? not sure - here is a blog art project i just blew out of my ass - probably similar to the process of other net artists (although my process does not involve weed) - a dictionary of common words as spelled by bloggers ( or twitters)....

More Adventures with Map Reduce and the Cloud

I am compressing video in the cloud via amazon's EC2/mapreduce/s3/etc... I am using map/reduce for this although it is really not a reduce problem only a map problem and not even that - I just need to spawn a bunch of ffmpeg processes....

The problem is I cant map/reduce on a disk that has ffmpeg. So what I did was compiled map reduce on my machine - zipped it - uploaded it to s3 then added a bootstrap action to unzip and ffmpeg whenever the program starts... lets see how this works - I think I need to recompile on debian though - currently using fedora

As for the arabian nights - i have learned that the arabic oral tradition no longer exists because people sit around in cafes and watch tv instead of telling/listening to stories - ahh technology

Deep thoughts on magical realism

So I am continuing my romp through Irwin's commentary to the Arabian Nights - although I really want to read a copy of The Ocean of the Rivers of Story- which apparently is the source of all stories. But I digress.

In my romp I found that fairy tales were considered part of the genre of the Arabian Nights - and were very much in fashion in 1600s Europe. The author speculates that these were stories were relegated to children's tales as tales of the fantastic were usurped by science fiction.

Today I suppose tales of the fantastic are usurped by magical realism. It is interesting to think though that much of literature is to satisfy this desire for surprise and wonder. We are normally taught in school that fiction imparts some sort of cultural norms - such as homeric heroism. Rather literature engages our desire for something beyond the real. Literature is metaphysics. Perhaps these days video games have replaced this fantabulism because even magical realist books are not quite fantastic enough. We definitely see strands of this in contemporary tv such as Lost and true blood. Sometimes, however, a simple rendering of the real, in all its absurdity, is fantastic (hence my continued obsession with tao lin).