I've checked in what looks like a pretty good Directory implementation for now. The current version is configurable in terms of whether to use block compression and block size itself.
I've changed the way WXS maps are used to use dynamic Maps. I defined two Map templates, one for file meta data and another for chunks (the file data). The plugin now makes a new pair of Maps for each directory. All the file meta data maps are named "FileMetaData."+dirName and chunk maps are "ChunkMap."+dirName. The reason for this change is to enable better metrics. I had all the chunks and file meta data in one Map before. This made it hard to see per directory metrics on map sizes and the like. Separating the chunks and meta data to by directory maps make it easier to analyze this.
Compression has substantial memory and network bandwidth savings, I've seen as high as 90% in indexes that I have available to me but it does burn more CPU. It's the classic trade off.
