193 symposiums and 30,000 attendees since 2001

Michael Nygard's complete blog can be found at: http://www.michaelnygard.com/blog/index.html

Items:   1 to 5 of 145   Next »

Sunday, January 3, 2010

I'm working on a syllabus for an extensive course on web architecture. This will be for experienced programmers looking to become architects.

Like all of my work about architecture, this covers technology, business, and strategic aspects, so there's an emphasis on creating high-velocity, competitive organizations.

In general, I'm aiming for a mark that's just behind the bleeding edge. So, I'm including several of the NoSQL persistence technologies, for example, but not including Erjang because it's too early. (Or is that "erl-y"? )

(What I'd really love to do is make a screencast series out of all of these. I'm daunted, though. There's a lot of ground to cover here!)

EDIT: Added function and OO styles of programming. (Thanks @deanwampler.) Added JRuby/Java under languages. (Thanks @glv.)

I'm interested in hearing your feedback. What would you add? Remove?

  • Methods and Processes

    • Systems Thinking/Learning Organization
    • High Velocity Organizations
    • Safety Culture
    • Error-Inducing Systems ("Normal Accidents")
    • Points of Leverage
    • Fundamental Dynamics: Iteration, Variation, Selection, Feedback, Constraint
    • 5D architecture
    • Failures of Intuition
    • ToC
    • Critical Chain
    • Lean Software Development
    • Real Options
    • Strategic Navigation
    • OODA
    • Tempo, Adaptation
    • XP
    • Scrum
    • Lean
    • Kanban
    • TDD
  • Architecture Styles

    • REST / ROA
    • SOA
    • Pipes & Filters
    • Actors
    • App-server centric
    • Event-Driven Architecture
  • Web Foundations

    • The "architecture" of the web
    • HTTP 1.0 & 1.1
    • Browser fetch behaviors
    • HTTP Intermediaries
  • The Nature of the Web

    • Crowdsourcing
    • Folksonomy
    • Mashups/APIs/Linked Open Data
  • Testing

    • TDD
    • Unit testing
    • BDD/Spec testing
    • ScalaCheck
    • Selenium
  • Persistence

    • Redis
    • CouchDB
    • Neo4J
    • eXist
    • "Web-shaped" persistence
  • Technical architecture

    • 8 Fallacies of Distributed Computing
    • CAP Theorem
    • Scalability
    • Reliability
    • Performance
    • Latency
    • Capacity
    • Decoupling
    • Safety
  • Languages and Frameworks

    • Spring
    • Groovy/Grails
    • Scala
    • Lift
    • Clojure
    • Compojure
    • JRuby
    • Rails
    • OSGi
  • Design

    • Code Smells
    • Object Thinking
    • Object Design
    • Functional Thinking
    • API Design
    • Design for Operations
    • Information Hiding
    • Recognizing Coupling
  • Deployment

    • Physical
    • Virtual
    • Multisite
    • Cloud (AWS)
    • Chef
    • Puppet
    • Capistrano
  • Build and Version Control

    • Git
    • Ant
    • Maven
    • Leiningen
    • Private repos
    • Collaboration across projects

Thursday, December 3, 2009

The mighty Mississippi River starts in Minnesota, at Lake Itasca. Every kid in Minnesota has to make the ritual pilgrimage to Itasca State Park at some point, where wading across North America's longest river is a rite of passage.

Mississippi River Starts Here

One of the very interesting things in Itasca State Park is a section of forest that is fenced off so that deer cannot enter it. It's part of a decades-long experiment to see how forests are affected by browsing herbivores. What's really interesting is that not only are the quantity of plants different inside the protected area, but the types of plants and trees are different, too. Because deer prefer to nibble on younger trees, fewer saplings survive in the main body of the forest than in the fenced-off portion. Outside the fence, the distribution of tree size and age is biased toward older trees. The population of trees is weighted more toward resinous species like pines, which deer prefer not to eat. Inside the fence, more saplings survive into young maturity, so you see a more even distribution of tree ages and a wider diversity of species represented in the mature trees. The changes in the canopy affect the ground cover which, in turn, change how deer could (if allowed) reach the trees and browse them.

So, here's a feedback loop that involves deer, trees, leaves and brush. The net result is a different ecosystem (albeit a slightly artificial one.)

Most physical and biological systems are like this in several ways, particularly relating to feedback. In our artificial systems (electrical, mechanical, symbolic, or semantic) we build in feedback mechanisms as a deliberate control. These are often one dimensional, proportional, and negative.

In natural systems, feedback arises everywhere. Sometimes, it proves to be helpful for the long-term stability of the system. In which case, the feedback itself gets reinforced by the existence and perpetuation of the system it exists within. In a sense, the system adapts to reinforce beneficial feedback. Conversely, feedback webs that cause too much instability will, like an overly aggressive virus, lead to destruction of their host system and disappear. So, we can see the constituents of a system co-evolving with each other and the system itself.

The old "microphone-amplifier-speaker-squealing" example of feedback really fails here. We lack both language and metaphor to really grasp this kind of interaction over time. In part, I think that's because we like to separate the world into isolated components and only talk about components at a single level of abstraction. The trouble is that abstractions like "level of abstraction" only exist in our minds.

Here's another example of coevolution, courtesy of Jared Diamond in "Guns, Germs, and Steel". I'll apologize in advance for oversimplifying; I'm devoting a paragraph to an argument he develops across entire chapters.

At some point, a group of nomads decided that the seeds of these particular grasses were tasty. In collecting the grasses, they spread it around. Some kinds of seeds survived the winter better and responded well to being sown by humans. Now, nobody sat down and systematically picked out which seeds grew better or worse. They didn't have to, because the seeds that grew better produced more seeds for the next generation. Over time, a tiny difference (fractions of a percent) in productivity would lead some strains to supplant the others. Meanwhile, inextricably linked, some humans figured out how to plants, harvest, and eat these early grains. These humans had an advantage over their neighbors, so they were able to feed more babies. That turns out to be a benefit, because farming is hard work and requires more offspring to help produce food. (Another feedback loop.) Oh, and this kind of labor makes it advantageous to keep livestock, too. Over time, these farmers would breed and feed more children than the nomads, so farmers would come to be a larger and larger percentage of the population. Just as an added wrinkle, keeping livestock and fertilizing fields both lead to diseases that simultaneously harm the individuals and occasionally decimate the population, but also provide some long-term benefits such as better disease resistance and inadvertent biological warfare when encountering other civilizations.

Try to diagram the feedback loops here: nomads, farmers, livestock, grains, birthrates, and so on. Everything is connected to everything else. It's really hard to avoid slipping into teleological language here. We've got feedback and feedforward at several different levels and timescales here, from the scale of microbes to livestock to civilizations, and across centuries. This dynamic altered the course of many species evolution: cattle, wheat, maize, and yes, good old H. Sapiens.

This complexity of interaction extends to planetary and stellar levels as well. At some sufficiently long time scale, the intergalactic medium is coupled to our planetary ecosystem.

The human intellectual penchant for decomposition, isolation, and leveled abstraction is purely an artifact of the size of our bodies and the duration of our lives.


Wednesday, September 2, 2009

Google has published an explanation of the widespread GMail outage from September 1st. In this explanation, they trace the root cause to a layer of "request routers":

...a few of the request routers became overloaded and in effect told the rest of the system "stop sending us traffic, we're too slow!". This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

This perfectly describes the "Chain Reaction" stability antipattern from Release It!


Friday, July 31, 2009

I've been doing some work with Hadoop lately, and I just ran into an interesting problem with networking. This isn't a bug, per se, but a conflict in my configuration.

I'm running on a laptop, using a pseudo-distributed cluster. That means all the different processes are running, but they're all running on one box. That makes it possible to test jobs with full network communication, but without deploying to a production cluster.

I'm also working remotely, connecting to the corporate network by VPN. As is commonly done, our VPN is configured to completely separate the client machine from its local network. (If it didn't, you could use the VPN machine to bridge the secure corporate network to your home ISP, coffeeshop, airport, etc.)

Here's the problem: when on the VPN, my machine can't talk to its own IP address. Right now, ifconfig reports the laptops IP address as 192.168.1.105. That's the address associated with the physical NIC on the machine.

The odd part is that Hadoop mostly works this way. I've configured the name node, job tracker, task tracker, datanodes, etc. to all use "localhost". I can use HDFS, I can submit jobs, and all the map tasks work fine. The only problem is that when the map tasks finish, the task tracker cannot send data from the map tasks to the reduce tasks. The job appears to hang.

In the task tracker's log file, I see reports every 20 seconds or so that say

2009-07-31 11:01:33,992 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200907310946_003_r_000000_0 0.0% reduce > copy >

The instant I disconnected from the VPN, the copy proceeded and the reduce job ran.

I'm sure there's a configuration property somewhere within Hadoop that I can change. When (if) I find it, I'll update this post.


Thursday, July 16, 2009

Spiros Tzavellas pointed me to his implementation of Circuit Breaker. His approach uses AspectJ and can be applied using a bytecode weaver or AspectJ compiler. He's also got unit tests with 85% coverage.

Spiros' project page is here, and the code is (where else?) on GitHub. He appears to be quite actively developing the project.


Items:   1 to 5 of 145   Next »