pinkham often said, and now i often say, complexity moves up the stack. the amazon primitive web services are pure expressions of this idea. defined simply, complexity in the form of opacity, moving parts, cost, and so on, is transferred further up the stack of infrastructure where it can be either eliminated or reduced through application software. this implies low-level infrastructure that is regular, simple, and static with increasingly complex and dynamic layers built on top of it.

we see a similar concept in an unexpected and far more wondrous place: the evolution and expression of dna. biologically-inspired technologies have been a research topic for years, autonomic computing and epidemic protocols as two examples. these techniques often have very attractive properties to go with their fun names: scaling, self-healing, resilience, blah, blah, blah. i think about infrastructure and have taken some different things from biology.

the not-quite-accurately named universal genetic code (ugc) and the other, extremely similar variants found in mitochondrial DNA and other small organisms, evolved. we’re used to dna being the stuff of evolution (rna fans save your flame mails), so the code itself evolving is a bit of a brain bender. think of it as meta-evolution.

the end of evolution for the ugc happened billions of years ago. experts call this “early fixation”. the code itself is a marvel of efficiency. it resists exactly the sort of errors to which the transcription machinery is prone. when a transcription error gets through, chances are the erroneous amino acid will have properties similar enough to the correct one that there is little functional difference in the geometry of the final protein. we can appreciate just how spectacular it is because we can almost completely quantify its environment, something simply out of the question with complete organisms. physics, chemistry, geometry.

the high school biology textbook version of how genes evolve goes something like this: mutation and crossover. mostly mutation. read chapters 5 through 7 for monday.

reality is far richer. enter systems biology and evolutionary developmental biology (evodevo). i can’t do justice to the topics in a blog post, but the important idea is this: genes change very slowly, while the gene regulation networks that control their expression are incredibly complex and evolve rapidly.

the bottom of the stack is ridiculously efficient, defined almost entirely by physical constraints, and completely static. up a level we see simple, modular structures that can be assembled in a variety of ways, and that change very slowly. finally, at the top, we see the incredible complexity and the roiling expression of evolutionary change.

ugc -> genes -> regulation networks

hardware infrastructure -> software infrastructure -> applications

this is complexity moving up the stack in the most complex, distributed system to which we have access. that’s probably a lesson worth learning.

in 2007 i had the fortune of speaking with john dunagan about designing distributed systems and he introduced me the powerful idea of a “complexity budget”. i now see complexity moving up the stack as merely an effect of complexity budgets. like anything worth knowing, complexity budgets are simple: complexity has a cost, like any other resource, and we can’t expect an infinite budget.

spending our complexity budget wisely means investing it in the areas where it brings the most benefit (the most leverage, if you must), sometimes immediately, sometimes only once a system grows, and not spending it on things unessential to our goals. returning to aws, ask yourself whether buying, racking, and operating your own servers, storage, and network, that is, building your own infrastructure, gives you significant, quantifiable advantage in your space. if it doesn’t, the complexity budget associated with that work can now be applied to your applications. it has moved up the stack. the money and people you would’ve put towards infrastructure are now available for creating more code, supporting more customers, closing more deals.

ultimately, you want to spend 100% of your complexity budget on the things central to your business and none of it on ancillaries. if there is currently no offering out there for something ancillary to your business, meaning you have to spend complexity budget on it, perhaps it’s a new space into which you can expand. looking at the world through the lens of a complexity budget changes the conversation away from tiny details about technology, putting us square to questions of strategy.

these are things i have learned from others. if you like the ideas, thank the folks who taught me. if you don’t, it’s my fault.

among the many crimes of which the “cloud” meme is guilty, the return to prominence of the ignorant, fearmongering, security “expert” ranks high. in the past few weeks we’ve had infoq proclaim the new aws console insecure because it uses passwords, based on this non-post. let that sink in, i’ll come back to it. we’ve also had the geniuses at computerworld turn this verbose and uninteresting blog post into the piracy apocalypse because, horror of horrors, ec2 lets you pay to run arbitrary code and move arbitrary bits.

passwords

aws console using passwords is a security flaw? seriously? can you social engineer password/password resets out of amazon support folks? harder than you might think, and at least as hard as getting some help desk shmo at your enterprise here to give you access to the corporate network. heck, don’t even call in, just email an appropriate chunk of tasty code to anyone in the joint and have the computers harvest the credentials for you.

the notion that the aws console in any way changes this equation is absurd. the console provides no new functionality. got it? it is a gui. if they could steal your credentials now, they could steal them before. aws gives you an interface to your infrastructure that makes a whole new world of automated and secure management of it possible. the baseline is at least as good as typical enterprise infrastructure (i’d argue it is better). the ease of exceeding that baseline is dramatic. where they see weakness i see strength (and their ignorance).

pirates

the scourge of bittorrent cries “avast, bezos!” and slashes their way aboard the good ship ec2. aaarrrgh! ok, not really. in reality, s3 has had a bittorrent interface since it launched (even mentioned in the article), but for some reason there has not been a sudden surge in torrent-based piracy. as for ec2, well, it’s a service to run arbitrary code. people have been running torrent clients on it almost since it launched. again, no massive upswing in the pirated bits. why? here are 2 big reasons: 1) the services are pay per use, not free and 2) the aws terms of service preclude it and amazon actually pays attention to ToS adherence.

perhaps i am giving computerworld too much credit. i had to stifle a snorting laugh when i read this:

Amazon already supports the BitTorrent protocol through its Simple Storage Service (S3), though a heavy user would likely find this service much more expensive than EC2.

ec2 data transfer pricing. s3 data transfer pricing.

even the casual observer at the back of the room is certain to note the prices are identical. i’m surprised computerworld managed to expand the s3 name given the quality of their “research”. i wonder if they even know enough to be embarrassed.

to all those considering aws: use it! with great control comes great responsibility, and aws gives you, in many ways, more control, more agility, than almost any other infrastructure you can get. the guidelines for building and operating securely on aws are no different from those you’d use running applications on your own infrastructure. don’t let the fearmongering scare you off. this is the future and it is good.

apple has a long history of useless UI and feature changes, often changing defaults for no apparent reason and then removing the UI to change things back. it’s maddening when you configure your environment perfectly for getting work done and they pull the rug out. here are some of the settings, mostly undocumented, i use to keep sane in the face of the apple insanity onslaught.

floating glass shelf for the dock? if i wanted gratuitous, distracting, photorealistic elements i’d be running linux and have neon glowing out the side of my fire-breathing gamer PC.

defaults write com.apple.dock no-glass -boolean YES ; killall Dock

dashboard is one of the many osx doodads best suited to the iphone (see also: my multitouch trackpad with lots of scotch tape on it so it behaves more like a single touch).

defaults write com.apple.dashboard mcx-disabled -boolean YES ; killall Dock

you can click to turn off genre in the browser for shared libraries, but not your own. wtf?

defaults write com.apple.itunes show-genre-when-browsing -bool FALSE

time machine is the best backup system i’ve ever used. for reasons i won’t investigate, apple requires that you buy a time capsule to use the feature, rather than allowing you to just attach a hard drive to your airport extreme (or similar). get sneaky.

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

i was trying to avoid writing this post and had succeeded at that goal for almost 2 years. after some recent exchanges, i see the wisest move is the opposite. so, here goes.

in 2003 i was working at amazon for the best manager i’ve ever had, chris pinkham. chris had hired me the previous year as a network engineer, quickly promoting me to manager for the (ridiculously awesome) team. chris was always pushing me to change the infrastructure, especially driving better abstraction and uniformity, essential for efficiently scaling. he wanted an all IP network instead of the mess of VLANs amazon had at the time, so we designed it, built it, and worked with developers so their applications would work with it. he wanted anycast dns, so we hacked up some routing software and put it out there (great idea at the time, but in hindsight we probably should’ve taken a different approach). chris asked for something, we figured out how, and did it.

sorry for the digression, back to what i was saying about 2003: chris and i wrote a short paper describing a vision for amazon infrastructure that was completely standardized, completely automated, and relied extensively on web services for things like storage. we drew on the work of a number of other folks internally who had been thinking and writing (and sometimes even coding) in the storage services space, and we combined it with our own thinking and experience in infrastructure. near the end of it, we mentioned the possibility of selling virtual servers as a service.

we presented the paper to bezos (he doesn’t do slides), he liked a lot of it, and we went back to work.

a few months later, in early 2004, i was told jeff was interested in the virtual server as a service idea and asked for a more detailed write up of it. this i did, also incorporating a couple of requests jeff had, like the idea of a “universe” of virtuals, which i translated into network-speak as a distributed firewall to isolate groups of servers. this first cut at it looked almost nothing like the production EC2 service, and, in my view, every change made by the team who built EC2 was for the better. as just one example, that first paper called for a system manifest from which a server would be built. this is similar to how much systems automation works, but is actually terrible for the sort of dynamism desired for EC2.

after presenting the “executive brief” paper to jeff, the realities of turning this hare-brained scheme into a real service meant involving the smartest folks around (i.e., not me). in the amazon style of “starting from the customer and working backwards”, we produced a “press release” and a FAQ to further detail the how and why of what would become EC2. at this point attention turned from these paper pushing exercises to specifics of getting it built. most importantly, who would lead the effort?

everyone seemed to leap at once to the same conclusion: pinkham. and so it was that pinkham returned to south africa, taking a stellar lead developer with him, and they built the EC2 team, then built EC2. that last part seems awfully compressed, doesn’t it? well, that’s because i had almost no interaction with the EC2 team. they went off and kicked a lot of ass and the rest is history.

the end.

want more data? here’s jeff in a 2008 interview with om malik…

meatclouds of the world, pt. 6

meatclouds of the world, pt. 5

meatclouds of the world, pt. 4

meatclouds of the world, pt. 3

meatclouds of the world, pt. 2

meatclouds of the world, pt. 1