I have this algorithm...

Algorithms can solve two kinds of problems.

1) Problems with verifiable answers and...
2) Those without verifiable answers (or the verification is proved by statistically sampling)

Verifiable answers:

Algorithms in this category address problems like, find me all the numbers in this list, or even, find me the fastest route between point a and b.

The value in the algorithm is not the correctness, it is easy to find the answers to these problems, rather the value in these algorithms are the speed with which they are executed and the amount of data you have to solve your problem.

So for example there are many different kinds of sorting algorithms - they all correctly sort, but some are faster and some use less memory. This is where the ingenuity or value of the sorting algorithm comes in.

For the mapping example, it is not difficult to map the fastest route between two points. But you could imagine an algorithm that used realtime traffic data would provide faster routes than other algorithms. The answer these algorithms may be different, but their actual processing of information - the algorithm itself - may be the same. The only difference is the input data used by the algorithm.


Now onto the algorithms with no answers -
these are the interesting algorithms that answer questions like - find the top 10 sites on music, or recommend movies I'll like, or find me a date.

In the case of search - the answer can be the top 10 sites on music are the top music sites that people link to (google's algorithm), or the top 10 sites that my friends like (a facebook solution)

In the case of recommendations, we would have an algorithm like that produced by the netflix prize, a statistical sampling of user preferences run through a barrages of matrix filters. We could also have the pandora algorithm, create a genetic code for a piece of music based on frequency, btm, etc and then find other similar genetic codes.

For dating there is a little less innovation out there. Most sites ask a set of questions (perhaps developed by behavioral specialists), and match those with similar or compatible answers, or let users shop for dates by displaying the answers to these behavioral questions.

But that is it go forth and create algorithms for problems with no answers

transducer

i feel like on of the big new media memes is transduction.

For example convert stock market feeds into audio sounds, or convert twitter feeds into graphical visualization.

Data visualization and audio visualization are forms of transduction - translating from one form into another. The issue with transduction is how do we perform the translation. This is where the real art of data visualization or data sonification takes place. There are infinate numbers of ways to map the frequency of #brit on twitter to a visual cue. What signifies frequency (line size?), what signifies subject matter (color?), where do you place the visualization on the screen (existence in different social networks?). All of these are arbitrary and the criterion of success is how transparent the visualization is for understanding the data, or how aesthetically pleasing the result is.

I have been writing an application called doh rae tweet which lets you compose music via twitter. Here again the mappings are arbitrary. an 'A' is a note 'a', but a space is S (it could just be a space). The default time signature is 4/4 unless you start your tweet with something else like 8|8. and you adjust the beat of a note by + or - so A- (in 4/4) would be a half note. This is all arbitrary and it is not necessarily transduction but translation. Not transforming one type of thing into something else, but representing something (musical notation) in something else (140 characters on twitter) . In this way it is much more similar to writing a compiler than writing a data visualization.

However doh rae tweet can also can be used as a transduction engine. I can feed through random twitter feed and use the music compilation engine to hear what that tweet sounds like according to my 'compiler' Although it probably wont sound very nice. Most letters will not have an audio mapping. So for this sort of application I would want to replace my compiler or lexical engine with another. Ideally anyone could create their own mapping. I think it would be interesting to do this via a GUI and feedback loop so you could adjust your mappings visually as you hear the effects of your changes.

One final though - I also want to use this mapping for other notation schemes. One that seems a natural is chess. With the 8x8 chess board mapping to the diatonic scale. However, this too would need its own mapping. ( Also what do you play, to just play the chess piece that is moved - or all the chess pieces. ) I went online to see if anyone has attempted sonficiations of chess notation and found the following:

Halfbakery This has some links relating chess to music
Chess History A list of articles and historical documents connecting chess with music (not very informative but interesting from a historical standpoint).
Music and Chess A great website with all sort of information linking chess and music. I got this from that site: "The Oxford Companion to Chess is a comprehensive encyclopedia of chess. It contains articles on history, terminology, chess players, and the relationship between chess and other subjects such as music, art, theatre, literature and philosophy. Many of the terms listed in this book are also musical terms. For example, in chess, a person who creates puzzles and problems to be solved is called a composer, and two different sequences of moves that lead from one given position to another are said to be related by transposition. Some other terms that are used in chess and music are: play, piece, notation, score, tempo, theme, variation, development, minimal composition, round, major and minor, position, second, retrograde, mirror, attack, anticipation, phase and echo."

Blown away by The Sound and the Fury

Ok - so when I first started the sound and the fury I had no idea what was going on. 'I think they are on a golf course' I told my business partner Paul. 'I have no idea what is going on.' Half way through the book I felt that I was waisting precious moments of my life reading a piece of crap. However, at some point - probably during Quentin's chapter - I reevaluated.

Why did I reevaluate?? I dont know. Perhaps the language... there is a certain lyricism - and i am a sucker for dialogue. But, at the end of the book a few things struck me...

1) Such a wide expanse of time was covered in such a 200 page book. The fragmented storytelling conveyed more information and feeling than a 1000 page tomb of today's magical realism.
2) What is the story?? The book is really about the construction of narrative, or the construction of myth and meaning. At the end of the day - when you read a story or a myth everything seems so tidy - but the construction itself is messy. While the story is being written you have no sense of the whole completed version - the sound of the fury is like a story writing itself
3) Roshimon - Whose point of view is correct? Seeing the compson family through the eyes of the different narrators in a stream of consciousness voice bring home the fact that every story is told from a point of view.
4) It is a mind fuck - the story seems like a puzzle. Not a post modern puzzle, where the whole point of the novel is the puzzle (like paul auster - of whom I am a big fan). Perhaps puzzle is the wrong word... It is coded. I feel like if i diagramed the whole novel out - parsing out the different stories by the different narrators at different times in the life of the characters I would get another story. There is a second story hidden in the story.
5) This is a modern novel. It struggles with modern themes (and it was written in 1930s I think). Sexuality, Economics, Race, Class - the treatment of these issues are not preachy but are integral to the lives of the character. this is great storytelling - bringing up weighty issues as part of the characters struggles

Anyway I am inspired - I may experiment with this sort of writing for some of my short stories. Also reading Sarte's critique of faulkner - I'm into sarte as literary critic (just was gifted book 5 of the family idiot about flaubert).

Local build of ffmpeg

This was going to be a highminded post about fairy tales and morphology but instead i am going to record my trials and trivails of running ffmpeg on amazon's map-reduce.

So - what i am doing is adding a cached directory of ffmpeg to every map-reduce computer i use


#!/usr/bin/bash
./elastic-mapreduce --create --stream --alive --name $1 \
-c credentials.json \
--input s3n://13bit.videoout \
--mapper s3://13bit.scripts/compression.py \
--reduce s3://13bit.scripts/reduce.py \
--output s3n://13bit.videoout.ts \
--log-uri s3n://13bit.log \
--cache-archive s3n://13bit.scripts/ffmpeg.zip#ffmpeg

in order to build ffmpeg i started up a debian sqeeze box via aleatic ami (ami-daf615b3) http://alestic.com/. I then had to update apt-get (apt-get update), install git, download ffmpeg and related libraries and run the ffmpeg configure switch --prefix to set the install directory. I am now zipping it up and testing it on another instance of the squeeze debian - at which point i will start up the map reduce process again

i am also looking at boston terrier art on etsy (thanks to Jane Kim)
http://www.etsy.com/listing/37552859/boston-terrier-dog-art-original-painting
I am doing this while my boston terrier sleeps and farts next to me.

Living the Dream

Last night Paul and I attended a fascinating lecture on the postal history of the Congo Free State at the NY philatelic society. This is research for our latest documentary. There is nothing like listening to a lecture by an expert. Not all fields are capable of producing experts - Literary theory for instance. You can only be an expert is something that exists, that has a counterpoint in reality. You cannot be an expert in unicorns, however you can be an expert on the literature of unicorns, the archaeology of unicorns, and the symbology of unicorns. Likewise you cannot be an expert on literary theory, but you can be an expert on the thoughts of literary theorists. However I think expertise in something like stamps of the Congo is much more worthwhile...

On another note- I ran across the following art project by cory arcangle - it culls the internet for blog posts that begin with i'm sorry. I am a big fan of his video work, especially mario clouds video & the the one where he just projects fields of color from the projector. However, this 'sorry' project is sort of in bad taste. What is it? A project mocking people attempting to maintain blog? I am not quite sure what its purpose is. I doubt it is to be inspired by the heart felt apologies - as cory writes on his blog. So what exactly is the point of this project? to call attention to the dead blogs in the world, to show how apologies have become meaningless, is it supposed to just be funny, it it supposed to show how pathetic people are, how people have no imagination (except for cory)? not sure - here is a blog art project i just blew out of my ass - probably similar to the process of other net artists (although my process does not involve weed) - a dictionary of common words as spelled by bloggers ( or twitters)....

More Adventures with Map Reduce and the Cloud

I am compressing video in the cloud via amazon's EC2/mapreduce/s3/etc... I am using map/reduce for this although it is really not a reduce problem only a map problem and not even that - I just need to spawn a bunch of ffmpeg processes....

The problem is I cant map/reduce on a disk that has ffmpeg. So what I did was compiled map reduce on my machine - zipped it - uploaded it to s3 then added a bootstrap action to unzip and ffmpeg whenever the program starts... lets see how this works - I think I need to recompile on debian though - currently using fedora

As for the arabian nights - i have learned that the arabic oral tradition no longer exists because people sit around in cafes and watch tv instead of telling/listening to stories - ahh technology

Deep thoughts on magical realism

So I am continuing my romp through Irwin's commentary to the Arabian Nights - although I really want to read a copy of The Ocean of the Rivers of Story- which apparently is the source of all stories. But I digress.

In my romp I found that fairy tales were considered part of the genre of the Arabian Nights - and were very much in fashion in 1600s Europe. The author speculates that these were stories were relegated to children's tales as tales of the fantastic were usurped by science fiction.

Today I suppose tales of the fantastic are usurped by magical realism. It is interesting to think though that much of literature is to satisfy this desire for surprise and wonder. We are normally taught in school that fiction imparts some sort of cultural norms - such as homeric heroism. Rather literature engages our desire for something beyond the real. Literature is metaphysics. Perhaps these days video games have replaced this fantabulism because even magical realist books are not quite fantastic enough. We definitely see strands of this in contemporary tv such as Lost and true blood. Sometimes, however, a simple rendering of the real, in all its absurdity, is fantastic (hence my continued obsession with tao lin).

Genealogy of manuscripts

I am reading Robert Irwin's "The Arabian Nights" A companion. It was recommended to me as 'the best commentary ever.' I dont really agree. I think the best work of commentary is Herbert Dreyfus's commentary on Heidegger's "Being and Time", "Being-in-the-world." The best work of literary criticism or commentary is Mimesis by Auerbach. I keep a copy of this above my desk - 'being-in-the-world' is relegated to my stacks.

What is interesting about the Arabian nights commentary is its discussion of literary scholarship. This all but disappeared with the advent of ideologies and literary criticism. Rather than the try piecing together of texts from various sources, trying to find the 'true text' in some sort of pseudo-scientific exploration, people now just interpret a text in light of some agenda - in some meta-pseudo-scientific pontification. We are really moving beyond the scientific method - it seems - as a mode of exploration and investigation of the world. String theory is not a result of empirical evidence and nor is the Lacanian interpretation of Pär Lagerkvist.

But back to the Arabian nights commentary. So one of the ideas that caught my eye was that scholars where attempting to create a sort of evolutionary tree of different editions of the Arabian nights. This is sort of an interesting thought to me that works of literature can have a history. Further along, we see that many of the stories in the Nights, have parallels in Chaucer, Boccaccio, various sanskrit works, and other. Here is yet another genealogy - the genealogy of an idea rather than the genealogy of a work. I feel like these sorts of ideas are used in the generation of books themselves, the questionable source of a work is used as a device in Don Quixote and in the "Dictionary of the Khazars" - and probably in 59% of Bevery Borges novels.

The sort of scholarship that went into this sort of detective work seems more apt for the computer age (or the Big Data Age) than the experimental age, however I dont think literature scholars are doing this sort of work anymore -

I leave you with some of my favorite lines from Tao Lin's "Shoplifting from American Apparel"
"On Christmas Eve Sam work around 7 p.m. in his brother's studio apartment in Manhattan. Sam had moved in November into a four-person apartment in Brooklyn but was staying at his brother's studio while his brother was on vacation with his girlfriend. Sam put on music very loud and showered in the dark with the bathroom door open. He put in earphones and walked ten blocks to an organic raw vegan restaurant. He ate a seaweed salad. He drank a smoothie. He walked back to the apartment. He drank an energy drink. He worked on writing for two and a half hours. He lay on his brother's queen-size bed and listened to music. He read most of the newest Stephen Dixon novel and fell asleep around 3 a.m.

The Greatest Work of Our Time Is Lost

I was just watching the imaginarium of dr parnassus which begins full of promise and then sort of craps out. The most interesting part of the film is the story of parnassus not the imaginary wonderlands of random people who go through parnassus' mirror.

I was totally blown away by the production design of Parnassus' Monastery. After watching the DVD extras about the Monastery, it seemed that one of the great achievements of the film were the props. I often think this is the case with other films, and with theater in particular.

It seems after thousands of years of architectural masterpieces, 500 years of scupltural masterpieces, and about 200 years of painted canvas masterpieces - we are in an age where masterpieces are really transitory and in the service of other even more transitory masterpieces. I guess this is what happens when production is in service of entertainment or the triumph of bread on circuses.

A completely different blog post

i have been blogging lately to keep track of what I am doing with my streaming video project. The montagenik blog is probably not the best place for this. I should probably keep this info in evernote or perhaps another blog, or perhaps a del.icio.us account, or perhaps a git gist. However, the fact is, that I actually go back in my blog and look for these tidbits. My blog is like my personal search engine repository. When I spread out this information over a bunch of different application I generally dont use them. I find it more efficient to keep everything in one location.

This week has been a good week in the 13Bit getting things done world, but a bad week in the Meredith getting things done world. The jet lag has put me on a really messed up schedule and I am completely useless after 4pm. Tomorrow Paul and I have a short shooting trip to NJ where we will interview two collectors, then sunday Cindy and I will bot in preparation for the upcoming robot parade. Paul and I also have something special up our sleeves for the Ecopocolypse website. It is very cool....

I started out the day by reading from 'The Book of Disquiet' - This is a fantastic, beautiful, inspiring book. "And at this table in my absurd room, I, a pathetic and anonymous office clerk, write words as if they were the soul'd salvation and I gild myself with the impossible sunset of high and vast hills in the distance." I am considering ripping off Mattins and reading parts of this on a daily podcast.

My new love is the OED on the computer.

More EC2, FFMPEG and video streaming from the cloud

So today I am setting up the compression engine on EC2 to compress video out into 4 formats. I need to do this before I use the segmenter to divide up video out into segments that will then be recompiled on the fly via lexical searches...

Anyway I had a java program downloading my mov files - clips from video out- but it was taking to long. I dont want to have to download the file in order to convert it I want ffmpeg to convert it via http... luckily i found this on stack overload:
http://stackoverflow.com/questions/2677317/how-to-read-remote-video-on-amazon-s3-using-ffmpeg


evermind, I found an easy way to solve my problem.

I set up an amazon cloudfront download distribution pointing to my S3 bucket.
Via cloudfront the files are accessible with ffmpeg over http:

ffmpeg -i "http://subdomain.cloudfront.net/video.mp4" -ss 00:00:10 -vframes 1 -f image2 "image%03d.jpg"

Incoherent Ramblings on EC2 and ICT4D

When you are deploying some sort of web based app with some sort of services (mobile, messaging, whatever) in a far away land, one of the largest time sucks is configuring the server. If the server has already been configured, there is generally no documentation. No one knows you decided to config your ssh port on 666. Granted one 'could' provide better documentation, as well as server install scripts (think a list of apt-get or yum or whatever you are doing). However, after working with EC2 on amazon, I think standardized disk images are the way to go.

1) You are actually deploying the same server everywhere. So, why not make that server once, as an EC2 instance and then actually deploy it.
2) Services that rely on a hardware device such as an SMS modem have to be installed on a local box, but, web access, database storage, and number crunching, can all be stored in the cloud. Then, if you want to edit your web interface or Z-score calculation for child malnutrition, you don't have to worry about power outages in Tanzania. And you will probably have less latency as well. The route that your server takes to ssh into a server in Tanzania is Byzantine.
3) Scaling. This is a dream issue but say all of South Africa starts using your service, all you need to do is deploy some more EC2 instances and load balance. You dont have to send over a bunch of techies to set up a server farm.
4) CAVEAT - as i mentioned above if you have a device attached to your box - such as an SMS modem, or rf transceiver or whatever - then you cannot deploy that aspect of your app to the cloud (the aspect that connects with the device) however you can use the EC2 image to create your local server.

EC2 is great for standardization, especially when you lack people, money, and equipment - i love it.

Superbitter AMI

I am creating an instance for the SuperBitter off the RAILS fedora AMI. I dont use fedora but I decided to use an Amazon recommended AMI and it was fedora or bust.

I added: git, java, csound, ffmpeg, scala, jetty & mavin (for lift)

ffmpeg is still missing some stuff

used this resource for ffmpeg install :http://blog.ryanparman.com/2008/06/28/installing-ffmpeg-php-fedora-amazon-aws/

in the midst of building thrift - (need to update autoconfig), mongodb (sort of a pain), open source segmenter (a pain), http://github.com/mfchang/HTTP-Live-Video-Stream-Segmenter-and-Distributor

For the future I am going to split up the web EC2 (rails) from the backend EC2 (scala, mongodb, csound). But for now I dont want to deal.

Video Server Development

So this is basically to help me keep track of development for a video application I am developing.
Paul sent me the following link that is super useful:
http://www.ioncannon.net/programming/1015/ipad-streaming-video-and-more/
This is also a useful link: http://blog.kyri0s.org/post/271121944/deploying-apples-http-live-streaming-in-a-gnu-linux

The plan:
I am using http live streaming to stream the video. I am streaming 90 minutes of video.

I need to create different bitstreams so I am going to use FFMpeg.

Then I am going to use an open source segmenter instead of the apple segmenter (this is running on unix) http://svn.assembla.com/svn/legend/segmenter/

I am going to set all this up as an instance on EC2.

This means I need to set up an AMI instancehttp://docs.amazonwebservices.com/AmazonEC2/dg/2006-06-26/creating-an-ami.html.

I may even install csound on the AMI

Reading Marx is Making Me Depressed

I feel like I have been reading lots of Marxist inspired thought lately, and so I picked up 'A very short intro to Marx'. Thus started my descent into economic depression. To summarize the summary... history is moving toward the freedom of mankind. Freedom, for Marx, means, freedom from alienation. Human beings are alienated from themselves because their work is not an expression of their nature but a means to survive. Alienation - the key point here. Humans are alienated because of economic conditions. That is what separates us from one another, (according to Marx), is economic conditions. This is opposed to religion (Feuerbach) as the crux of alienation. Or our separation from our true selves (Hegel).

Anyway - I find Marx's critique of alienation spot on - even if his economic theories are a bit dodgy. Everything these days is commodified and monetized. Here, I am blogging for the pleasure of it. However, blogger & google are commodifying this blog by integrating it into their ad/knowledge network. This is even more profound than the realized alienation of the wage slave who must sell his time in order to support himself. In this case I am not even aware of my alienation. I view myself as human, while google views me as a commodity.

Also, as a former wage slave to a corporation, but now an independent wage slave to myself, I doubt there is any way to escape the transformation of time into money. In my thought though the greatest achievements of technology are medical and, in a round about way, social. By social, I mean, technology and technological accomplishment is the great leveler. It is not subject to discrimination. The ability and the actual achievement of technological accomplishment by people of all races, gender, sexual orientation, leads to the end of social discrimination.

I started blogging yesterday and then did not finish

I dont even remember what I was blogging about. But let me free associate here. I am currently reading John Crowley's book "Little, Big." This is a magical realism tale about a new england family that is some how connected with fairies. It is not necessarily plot driven - rather the plot is figuring out what the twist is or why we should be interested in this family in the first place. The book has fairy tale logic.

I have a love/hate relationship with John Crowley's books. I read his Aegypt series on the Kindle when I lived in Hong Kong - and I remember sitting in the IFC court yard during my lunch breaks reading this book and feeling connected to home. I like his ideas. The integration of mystical and magical elements with everyday life. The possibility that esoteric knowledge is practical rather than fantastical. This is not like generic magical realism in that Crowley is integrating historical esoteric elements into his book. He creates a world where the esoteric knowledge, or mystery schools of thought, actually are valid. However, the novels are more like an exploration of these topics rather than a structured story. It is almost like when you are playing a video game and the exploration of the levels is more interesting than the boss fight.

Journey to the west is a tale that treats the 'boss fights' with drama and exposition - fantastic stuff. I like the exploration though, why is probably why I am drawn to Crowley's novels.

Has it really been 3 weeks since my last post!

I wish I had something to write. I have blogger block tho. I blame it on the intense heat that is making me lethargic, unable to focus, and physically weak. There is something about the heat that makes reality less real. So, it is hard to get worked up about work. The flip side of this lethargy is that I am no longer suffering from a low level malaise. How can you feel malaise if it is hard to feel anything. I need inspiration. I was inspired by the 13Bit road trip to document collectors, however I was to exhausted to turn this inspiration into anything concrete. All I can do now is execute the little things, doing laundry, logging footage, reading a poem, walking the dog, and perhaps a short blog post.

the scrivener

I use scrivener to write. Paul and I use it to write our screen plays. I use it to write my stories, plays, and currently I'm using it to write a graphic novel. I was once in an airport bookstore and spied Bartleby the Scrivener, and read it on the flight back to NY. How is scrivener the software like Bartleby the scrivener? Perhaps some industrious 11th grade English student somewhere can deconstruct this. Really I like the scrivener software because I have different outline modes, I can write in a corkboard mode, and it is easy to keep track of research and themes related to a particular topic.

So, can I write a game using Scrivener? Good question. I don't know, maybe I should try that.
I'll post here the first paragraph of my latest short story - it is the documentation for a cellular automata program.

"Documentation for a Computer Program"

Out of the IOStream comes theWorld. The stream is blue, not a microsoft blue, but deep sky blue, #00BFFF, and it is not web safe. theWorld is created from a pseudo random number generator. The first iteration of theWorld was created with a Perlin noise filter. But that produced too much geographic diversity. The purpose of this experiment is to see how cells interact, so we need to strip out all extraneous variables - geographic diversity was the first to go. The pseudo random number generator, and that provides much less variation in worlds from one Universe to the next.

The Great American Novel

My husband always says that I should write a novel. I have no idea where he gets the idea that I am capable of such a feat - but it is an interesting idea to entertain. I have about 20 half finished novels and some new novel ideas, but so does everyone.

A novel is more than an idea. It is not a knock knock joke, a one liner. A novel is not just a good idea - it is the unfolding and refolding and crumpling up and tossing on the floor of everything that surrounds that idea. What novel should I write? I ask my husband. Write that novel about Detroit. I had a idea about a novel that follows generations of a detroit family through immigration, the automotive revolution, and finally to current decay. "How should I start?" I ask. "Dont try to write War and Peace." My husband replies. That is actually very good advice. But the Detroit novel really does seem to have a War and Peace scale, so I try and pick up another idea. Ok - I was thinking about something golem related. For some reason I have been thinking about Golem stories. I have a nascent screenplay on a Golem and World War 2. I also love Jewish esoteric stories (the hasidic tales, tales of nachman) - I like weird esoteric stories in general. So I was thinking about combining the two. A group of jews hiding one night from the pogroms in early 19c russia stay up all night telling stories. Shamelessly stealing from the Decameron, The Canterbury Tales, and other stories in a story. I love framing devices almost as much as the stories they frame. I started writing this but all I got was a frame. A fictional story of my grandmother writing down her recollections from this night 50 years later. And my possession of this manuscript after her death, via my aunt. I actually had a beginning and an ending and all i need to do was fill it with stories. This has proven more difficult that I first thought. (I was also inspired by The Manuscript of Saragasso)

I was going to do 10 stories based on the 10 Sephirot (levels of creation) based on Kabalistic ideas. The first story I was writing was based on Daat (knowledge) and was a fictional account of Spinoza - really a glass grinder in Amsterdam. I was playing with ideas of rationality, speculation. But that is all I got - no plot ! I am thinking that there should be a video game based on the 10 sephirot. Maybe I should do that instead.

Being stymied, I decided to get into shape by writing short stories. Short stories are a different beast than novels - and I think most novelist dont get this. At the moment I am enjoying the short stories of George Saunders - he writes some dystopic fiction about amusement parks or modes of entertainment. Really great stuff. Since I was reading about vagueness in philosophy - I thought it would be interesting to write a story that explores vagueness. I sketched out a story about a bunch of adolescent girls who go to an amusement park and the different rides they can take based on the 'You are this high' mark - a sort of exploration of living in vague territory. I was going to end it with a death - but that was really a bummer. I really could not get behind it. It did not seem like a story I would write. And it really did not have a 'turn', a strong idea. A short story is sort of like a knock knock joke. Maybe I should write some science fiction.

This post was written while listening to passion pit.

Know your audience

I was in Los Angeles last week, and stayed with two friend, Arv and Anjali. Arv is an economist working on developmental economics - so we have lots to discuss - especially the fact that after recent experiences I no longer believe non-profit work is a viable solution to address public health and poverty issues. I have become a die hard capitalist - and believe the only solutions that address issues of poverty and health crises must be self-sustaining and not grant dependent. (More about this in another post perhaps). I am also against pilots (not ones that fly planes, or medical trial pilots). Also with the mass of data available I think that we need to revamp statistics - sample sets are less important - more important is noise reduction, or rather discerning salient information (trends) from the data, rather than extrapolating trends from a sample. (Again for another post)

Anyway, in speaking with Arv I remembered the importance of ethnographic research in the implementation of any solution. Many times projects fail, not because the data collection methodology is incorrect, or because the statistical analysis is not sound, but because people collect data incorrectly.

I was reminded of the passage in "American Caesar" a brilliant book by William Manchester about Douglas MacArthur.
When visiting Japan (before WWI), the Japanese generals said that there was a problem with malaria among their men. The men had prophylaxis (pills), but were still developing malaria - obviously something was wrong with the pills. MacArthur laughed and said something like "If I gave my men pills with instructions to take them every 4 hours - they would be dumped into a ditch and forgotten." The Japanese generals were horrified - our soldiers would never do such a thing. The next week a new batch of pills arrived with a new label - the emperor wishes that you to take a pill every 4 hours. After that there was never a problem with malaria.
No need for any fancy solution involving cell phones or alerts or sharks with laser beams - just ethnography (Arv liked this story too)

Vagueness

I am reading a rather dull philosophical tomb on vagueness. It is not really a tomb but it feels like one. I am just at the beginning, reading about the sorites paradox and borderline cases, but I thought I would blog a few thoughts about it.

The sorites paradox - this addresses the question of what constitutes a heap of sand or a bald man or a tall woman. If I have one piece of sand, and I keep adding sand to it, at which point do I have a heap of sand? There is a further 'philosophical' problem with these terms, namely how do you determine the validity of a statement that uses one of these vague terms. If I a am tall is someone .10 inch shorter than me also tall?

There are all sorts of philosophical solutions to these problems - most of which seem like utter mental masturbation and contrary to common sense. But what interests me is the intersection between this issue and other issues - in particular mathematics, and in particular thoughts about infinity (set theory) and thoughts about series (infinite series perhaps).

First let me address infinity and set theory. Josef Cantor came up with the idea that there are infinite number of infinite sets.
What does this mean? For example lets look at the set of natural numbers. It goes 1,2,3.... indefinitely. We would say the size of the set of natural numbers is infinity. Well is there anything larger than the infinity of the natural numbers. Cantor said (this was one of his proofs), that there were a greater number of rational numbers between 0 and 1 than in the natural number set. These are different types of sets. Natural numbers are countable, I can generate another number by applying some application to a lower number (such as adding 1, or multiplying by 2). For the numbers between 0 and 1 all I need to do is add a number to the denominator. 1/10, 1/100, 1/1000. I am not generating an infinite set by a mathematical operation but by constructing a new number. (I could go on but I want to get back to the paradox)

Also remember there are more irrational numbers than rational numbers (more holes in the number line than points) but I digress.

Anyway all this talk of different sizes of infinity. his seems like a bizarre question to ask at first, because infinity is not really a size but a description of a limiting circumstance. And does it really make sense to say the infinity of natural numbers is different than the infinity of the rational numbers, or irrational numbers. True the way we generate the number set is different but infinity is infinity.

This reminds me of the sorites paradox. When does a set become infinite, or infinite of a particular order? In this case each is defined by a different mathematical expression. A countable infinite set can be expressed by a function, and a non countable set can be defined by a transformation. But does this mean that the infiniteness is different. If I become bald by shaving my head or by genetics does that change the nature of baldness. Infinity is a limiting condition - it describes what differentiates the set from other sets that with definite boundaries.

Ok and my second thought about sorites - namely series, or rather limits. Perhaps baldness is that which is approached but never reached, like the limit of an infinite series. Rather than express the distance from baldness, or the way we reach baldness, we just default and call the condition baldness. Baldness is never actually reached, so it really does not make sense to ask if one hair added or subtracted makes a person bald. Rather, every hair lost brings someone closer to the condition of baldness which is never actually reached. I had more coherent thoughts on the airplane to LA when I was thinking about all these things - but you will have to deal with the limits of my memory.

Gaming the System

I have a surefire way to beat the computer at chess - I make very convaluted moves that eats away at the computer's clock and then i win on time. (granted this strategy only works if you play timed games - I play with sigma chess.

This sort of win never feels satisfying. It feels like I cheated, like my win was not really a win. However, it is a win. I am using the computer's strength (reliance on search algorithms) against it. Were I playing a human, these moves would probably signal amaturish play and I would be swiftly beaten as my opponent changes gears and uses a different strategy (the strategy that defeats idiosyncratic chess players).

Winning is not a matter of degree it is a binary attribute. Either you win or you dont. You can have a better win. We may talk about perhaps a better win, a more elegant win, or a win with more points or something like that - but really these are just language games. You either win or you lose - everything else is superfluous to winning-ness. Winning is achieving something over another person according to certain rules.

So are there winners in life? In this I always think of Alan Watts - life is not a game where to goal is to get from a to b, but a dance or a symphony - and a dance or symphony really has no goal other to experience it. Winning only makes sense in a situation where there are rules - does life have rules? Life is just about existence.

I have been thinking about this idea of winningness in regards to one of the options strategies I trade. I have a few strategies that I use in different market conditions, but let me talk about one strategy where I trade various mean reverting stocks and try and anticipate pin action (pin action describes where the stock will end on options expriy day so that the most option holders to lose money). My algorithms are generally good at anticipating the pin, however they are bad at market timing (which means i have to deal with market to market loses sometimes until options expiry day which reduces my buying power)

In any case - I often feel like 'I win' when I successfully guess the pin and make money on my options (and dont have to deal with mark to market losses anymore) But is this really a win? For example, let me use aapl (which I dont use for my options strategy anymore but I used to) if I sell an aapl put (aapl is at 250) - 250 for $7 and my algorithm thinks the pin will be 250 than that $7 will go to $0 theoretically or probably $1.50. I will make $5.50. However in that time aapl may go down to 240 - that put will go up 10 bucks and I will carry a mark to market loss of $10. Now do I buy back the put as it goes down (for a loss) and then sell it again when it bottoms out (for more money)? Do I average in more puts when the cost goes up? Sometimes it pays to close out for a small loss and then get back in at a better price. Does this mean you lose?

Or for example - say if aapl 260 calls are $6. You can sell them and if aapl hangs out at 250 for a few days or even goes down, those may go down to $4. Should you buy them back then, as aapl will probably shoot back up above 250. and those calls may go to $7 (mark to market loses again). My pin algorithm says aapl will go to 250 so they will end up being probably 1.00, but you will have to deal with the mark to market losses

In any case - a win is not necessarily just picking the correct pin. The correct pin is like the perfect chess game. A noble ambition -but you dont need it to win, and it may end up hurting your game.

Focus

It has been a while since my last blog post. I have written a few blog post drafts that I have not published. The problem with writing blog post drafts is that they quickly become irrelevant to the writer. Blogging for my is a spontaneous activity. I suppose for professional bloggers/writers/journalists this is not true. But for me, blogging is more akin to journaling. And once the inspiration is lost for an entry it is difficult to go back and find that initial spark of enthusiasm.

In the past few weeks I have found it difficult to focus - and this makes it difficult to have cogent thoughts to blog about - or interesting ways to relate my experiences through blogging.

I have attempted different things to get myself out of this rut.
First I attempted to read different types of books: magical realism (Little Big), difficult great fiction (Ulysses), Conspiracy fiction (the secret history of the world), philosophical aphorisms (life & flowers), science (the outer reaches of life), music (one of the grimoire books), books on option pricing, inspirational books (pema) . I could not get involved in any of them - I just could not lock in to the other person's description of reality or non reality.

Then I attempted to play games: nintendo DS (the attorney game), scribblenaughts, zelda. Nothing can focus. I did however become obsessed with playing chess. Sadly, I found myself holding my breath while playing chess - that cant be relaxing.

Even while doing bikram yoga - exercise so exhausting i cant think of anything else - I cant focus. Even while writing this post - I lost focus and started a chess game which reduced the oxygen flow to my brain.

Perhaps I just need more chocolate

Shakespeare & Film

I am attempting to go read all of shakespeare's plays in chronological order. I am at the beginning of the beginning - Henry VI Pt 1. I think I may have read this in my freshman humanities class - (is this one of the ones with falstaff). When writing films, I have started telling paul that our scripts have too many characters for a low budget film. (I think we had 23 in our last film) This probably betrays my lack of vision, but luckily Paul ignores me and I lack conviction and so we continue to write for too many characters. So does Shakespeare!

Granted it is really not fare to compare plays with films, but in many ways i am not quiet sure why. The most obvious reason is first, in theater there has to be more obvious action to direct the viewer's attention. How do we direct action? Dialogue. Our attention is directed towards the character talking and attends to what that character is doing. Here the playwright holds sway (which is probably the playwright rather than the director is the prestige position in theater.

In film we can manipulate the visual field. We can use a close up or a zoom or a dolly to direct the view towards a particular action - AND we dont have to use dialogue as a signal. We can just use visual cues. So here we have the director as opposed to the writer holding the prestige position, because it is the nuance of the acting and the location of the shots that tell the audience what to pay attention to - dialogue is not less important - but it is not the only tool in the tool chest.

In film we dont need as many characters. Props become characters, environments become characters - I wonder what it would be to film Henry VI pt 1 and turn some characters into 'filmic characters.'

It seems this blog post ended up at a completely different place than I had anticipated when I started, I was going to talk about how shakespeare differentiates his characters with dialogue - but i like this subject matter better.

Ecstatic Literature

I did not know which blog post to write - so I am writing one about an idea I was thinking about in Greece.
What is ecstatic literature?

I am convinced I stole this expression from some place, but a quick google only revealed some links to ecstatic poetry.
By ecstatic literature I mean a mode of storytelling where the goal is not necessarily to tell a story but to arouse a certain emotional & visceral response in the reader. This does seem more akin to poetry - poetry is an attempt to transcend language with language - to express some unexpressible sentiment or feeling (because language is limited and in every day speech cannot cause us to have such emotional responses.

These sorts of reactions are perhaps more common in film and perhaps the performing arts since they use a wider range of media and in many instances media that is not open to interpretation. When I watch a moving film, the director has selected the particular angle or close up I should view, the attitude of the actor, the music - the audience is there to receive the film not to engage or interpret the film. Reading is primarily an act of interpretation - it does not engage with one of the 5 senses. Watching a film is an act of sense-perception - we dont need interpretation to understand what we saw (unless it is symbolic performance art)

I am no thinking that it is perhaps the element of time that makes these forms more emotionally stirring than a novel which is not bound by time but can be read/re read/pondered upon by the reader. There is one book that is read, and another book that is the dialogue between the reader and the writer in the reader's mind.

Back to ecstatic literature. I was reading the colossus of maroussi by Henry Miller (an author I dont like at all). But this novel is unlike anything I have read in recent memory (except The Fan Man). The florid language, which I normally deplore, transported me to a state beyond that of communicating a story. It made me see the power of literature - which sounds a bit banal - but it is true. Perhaps contemporary literature can do better than roman a clef and magical realism - perhaps we can inject some poetry into the prose and awaken ourselves from dogmatic slumber.

Back to the Past

Yesterday I spent most of the day traveling back through the classical world.
Landing in Rome and then hopping a lovely Aegean Air flight to Athens.

In Athens, I am attempting to savor the last bits of urban life before heading into the hinterlands to ride horses like the ancient sychthians.

As a one time student of ancient greece, you would think a trip to Athens would be the fulfilment of a lifetime ambition. However, there is a quality of Athens, different than that of Rome, which makes the Agora and the Acropolis not part of some remote past enhabited by Socrates and Pericles, but part of some eternal present - integrated into 'Athens.' Perhaps Athens is the eternal city.

In Rome, you get the sense of living among ruins. That there were different stages of rome and they are all presevered in different strata - all on view in the Museum that is the city of Rome.

Rome is stratified Rock, Athens is metamorphic rock or something (I cant google it now i am being rushed off the computer by german tourists).

The Market and Quantum Physics

These days many physics (and other math/science PhDs) go into finance - in to trading in particular. Why? Well from the point of view of the scientist there is the remunerative aspect (you will probably 3x an academic salary and possibly much more). Also, the professionalization of science has made much of science an exercise in grant writing, front running hot topics, lab management and other tasks not normally associated with the unfettered search for TRUTH.

I would even argue that the professionalization of modern science has changed the methodology of science so that it is not properly following the 'scientific method' ie -
propose hypothesis
design and run experiments to test hypothesis
data analysis of experimental results to
support hypothesis and create a theory, or to reject hypothesis and start again.

With the professionalization of science and a professional scientific class to crunch data the emphasis on the scientific method has shifted to skew the data analysis section - however with our myriad analytical tools and with our surplus of data one can prove almost anything - so we perhaps need a return to first principals.

But I digress - quantum physicists may now work in finance - designing dynamic models to figure out the correlation of asset prices or what now. But what is the real truth is the market is like quantum physics. Namely you can either figure out the price (or location) of a stock/particle or the momentum of a stock/particle but not both. If I take the price of the stock (and buy or sell a share) - i affect the momentum of the system, if i look at the momentum of the system - i may not be able to buy/sell the stock at the current price (because it is moving).

This is why stock market money is not real until you take it out (or get a dividend)

Freud Vs Meditation

I am entering my 4th month of my bikram yoga practice. I practice at the Bikram yoga studio on the lower east side of Manhattan. Bikram (or beginner bikram) is a series of 26 yoga postures practiced for 90min in a hot yoga studio with an instructor yelling out commands. It is one of the most intense workouts I have ever had, and also the most meditative. It is so exhausting that my mind cannot wander and attain a certain amount of mental stillness or being in the moment.

This has started me thinking more about meditation. I used to meditate every night as part of my kung fu practice. This meditation was very much focused on the break and on different types of breathes and cultivating your chi. But like yoga meditation (kung fu has roots in yoga), the focus is on emptying your mind.

I think of this tradition contrasted with the freudian tradition, where everything must be put into context, explained away - or given a story. Pathologies are caused by incorrect or faulty personal histories and can be fixed by rewriting these personal histories via talking therapies. With meditation (inspired by Buddhism) there is not wrong personal history. Rather all personal histories are wrong. The self is an illusion and the practice is to become comfortable with ungroundedness (anxiety). Rather than finding the cause of the anxiety and explaining it away - gaining control over the anxiety, with meditation we must sit with our anxiety. We must notice the anxiety and not label it, not give it a story, just accept it and then let it slip away so we can be present to our sense perceptions rather than our mental processes in our brain.

Stay tuned for more meditation musings in a future post - also I have been reading some Pema Chodron that has been inspiring my thinking on meditation.

Future of Books

Last weekend I went to my local library and got a library card. The awesomeness of this experience was compounded by the fact that I was able to bring my dog otto into the Library. Why the library? My apartment is overflowing with books. This is generally a wonderful thing. Last week after finishing 'Betraying Spinoza' - I went to my personal library and pulled out all my Spinoza books - and some random books by jewish philosophers like 'The Star of Redemption' by Rosenzweig. However there are a few books that might have best been checked out and then RETURNED to the library - such as "The Ultimate Journey", an NYTimes book reviewer's personal memoir on travelling the same path as the Monk in the Chinese classic 'Journey to the West'. Journey to the West is fantastic - The Ultimate Journey not so much. There are also the impulse hardcover fiction purchases that I almost always regret - and regret paying too much for. Also, I hear you can check out DVDs and CDs from the library - and that is very exciting.

The day I went to the Library was also the day that I spent some time playing with Paul's (see 13Bit) iPad - a device that will probably replace the book and I started meditating on what this will mean to a place like my neighborhood library.

I live in chinatown. A few generations ago, when my grandfather was a boy, it was a jewish neighborhood, and during my childhood it was mostly dominican. The thread that runs through all these changes, is that it is an immigrant neighborhood and the little kids probably know english better than their parents.

At the library, there were a bunch of little kids running around, checking out books, checking out DVDs, learning - all for free. What will happen in 10 years when all this 'media' is consumed on a $500 iPad?






This got me

The man behind the power behind the man behind the ...

One of my guilty pleasures is reading conspiracy literature. It is sort of like reading science fiction and I think I enjoy it as a sort of thought experiment for how society might be. But what we call conspiracy literature has a long and possibly legitimate history as the continuation of hermetic or wisdom literature. Namely the idea that there is a hidden truth or knowledge that only some people can access. In fact, much conspiracy literature eventually returns back to secret organizations (ie wisdom traditions) 'really' running human affairs.

I would write more - but I want to finish reading about how the civil war was a conspiracy by european bankers to create more countries in america leading to a greater market for lending debt. (I guess they never heard of its ok we have municipal bonds - we can still trade credit default swaps on California)

As a final note the argument against conspiracies can also be a similar argument against 'intelligent design' namely it is more awesome and incredible that these events have taken place without central planning - to assume otherwise diminishes the wonder and power of these events.

Back From Blog Haitus - Passions

I recently read "On Stories" by Richard Kearney that began my mind working in all sorts of directions. I was captivated by his discussion of the limits of storytelling. This refers to events so awesome and singular that they resist a narrative framework that contextualizes them - to narrate them somehow diminishes their importance or their factuality. The example Kearney uses is the Holocost. One of the problems for these events is how do you integrate them into larger histories if you cannot transform them into stories that place them within a historical continuum. How do you represent unspeakable horror in storytelling?

And from here I started thinking about passion paintings - paintings representing the pain and suffering of Jesus. Depending on your belief system this may or may not represent unspeakable horror - but it does share an element with Holocost retellings - the idea that we must remember this event. In the Holocost we must remember so that it never happens again, in the case of the Passion we must remember the foundations of Christianity (the belief that Christ suffered for the sins of man kind) .

Kearney sort of reaches an impasse however in that there are certain atrocities - in the case of civil war it seems - where we want to forget. To paraphrase Alan Watts, just as we need a memory we also need a forgettory. That to hold on to certain events prevent the building of a healing narrative and dooms a people to live in constant turmoil - or to live in their turbulent past. Narratives, historical narratives, allows the past to become history rather than persist as current events. I am not quiet sure what Kearney's criterion for forgetting versus remembering, really where is the limit of storytelling?

So I have been thinking about what modern passion representations would look like. Representing the singular suffering of an individual in an attempt to both memorialize and historicize and communicate.

Finally this sort of singularity of human suffering and the horizon of storytelling reminds me of another horizon of representation - Kant's notion of the sublime (In the Critique of Judgement). The sublime is the pleasure that comes from 1) something we understand with our mind but cannot experience with our senses (what does it really mean that the earth is 238857 from the moon - how can we comprehend that - how can we experience that) 2) something that we experience with our senses but not with our mind - i thing of meditation or spirtual experiences, or extreme physical exertion (bikram yoga).
How do we represent these things?

Apologies for bad Kantian interpretation you Kant scholars out there.

India Venture & Tech Resource

My new venture in India is spurring me to keep abreast of new tech developments- this post, dear reader, is to remind me what i need to keep in mind

After making this list I have learned this - use BigTable - what whatever else google uses (GFS, Map/Reduce)



Data storage
Mongo db
Big Data from OReilly
Hibernate - Java again
Big Table - google
Languages/Framworks:
Cappuccino/Objective-J Build desktop apps in AJAX -I am not into web AJAX - it does not work well in low bandwidth situations.
Gittr - good ruby blog
Monk
F# - Nice F# Blog (I love F# and started using it 2 years at MS - if only it was interpreted)

good programming blog

Jquery
Jquery
Phone Gap
Fluid db
Sprout Core
Merb
Will I really use erlang?
Sinatra
Camping
Lambda the ultimate - it has been too long

ORMs
Sequel (ruby)
DataMapper (ruby)

The grid:
Hadoop
EC2

Objectified

Last night Lian and I watched Objectified, a film on industrial design, and the second film by the filmmaker of Helvetica - Gary Hustwit.

Some quick thoughts -
* The production values were much higher than those of Helvetica - very impressive for a second film.
* Loved the historical tidbits - how the ancient chinese tried to 'industrial design' arrows- so any arrow could be used in any bow. Why? Any archer could then take the arrows of any fallen soldier. Before this move each soldier made his own arrows - which only fit his bow. (Looking forward to the movie version of war in the age of intelligent machines)
* Loved the muji designer - I really liked his discussion of haiku's. He made a point that art and literature - in japan at least - is not about revealing all your feelings- this will make the audience uncomfortable. Instead you tell a story, or use a metaphor to draw the audience into your world in order to have her feel what you feel.
* Apple and fixtures - one of the most interesting comments was by an apple engineer. Most of the design is an attempt to build the tools (or fixtures) to design something a certain way. So for example, in addition to designing the apple keyboard - you also need to design the tools that hold the apple keyboard in place so that it is machined properly. 'Building the tools with which to make your art' (a quote from video out). This is nothing new - but I am surprised how much it is left out of the development process. It also makes me think of physics experiments: For example how Rutherford build a custom thin glass container so alpha particles could pass through.
*Industrial Design is about mass production. I never really thought about this, but the movie started out by saying we need I.D. because we mass produce things via machinese and we need to standardize the way that items are made. So I think of my brother who runs a clothing design company JF&Son. His pieces are not mass produced, rather they are hand made by a team of sowers and designers in India. His items are not 'industrial designed' they are short runs of less than 100 pieces. However where the 'industrial design' comes in is in his process. Namely of iterative feedback between himself, his customers who often collaborate on pieces, his co-workers in new york, and his co-workers in India. The supply chain is industrial designed not necessarily the pieces. It is 'industrial' rather than 'artisinal' because the intention is to sell these items on a large scale - not as one off works of art. 'Industrial' refers to 'industrial strength' or robust, rather than 'industrial' as pertaining to an assembly line.