Evil Monkey Labs

Immutability and Clever Variable Usage in the Land of Blocks and Branches

2011-08-05T00:00:00-07:00

Last night, I found myself unconciously refactoring some Scala code (I don't recall if it was something I wrote or someone else did at this point). As I looked at what I was doing I realized that many Scala developers don't seem entirely aware of one of my favorite features. What I'm talking about is effectively capturing values from multibranch block statements in Scala. Used correctly they can greatly decruft complicated code as well as helping us use mutability in places we might not expect an easy way to do so.

In typical C-like languages (such as C, C++, and Java) we are restricted in our syntax should we wish to capture a value when running many branching blocks such as if-else statements, switch statements and even for/foreach constructs. When we find ourselves wanting to set the value of a variable within each possible condition or iteration, we need to declare a mutable variable before the block. We then mutate this variable within each condition or iteration. Take this example from Java:

boolean valid = false;

String status = null;
if (valid) {
    status = "VALIDATED";
} 
else {
    status = "INVALID";
}

Not only have we had to declare a mutable value, but adding insult to injury the typical usage pattern includes declaring it as null (FACT: Everytime you explicitly use null, someone drowns a basket of adorable, whimpering puppies. Even if your language paradigm practically requires it.). Within our if/else block we have established a new value for status based on the value of valid. Granted, this is a somewhat weak example of my point, as you could just as easily have expressed this in a more concise ternary statement:

String status = valid ? "VALIDATED" : "INVALID";

This simplification breaks down quickly once our block becomes more complex. Even the addition of a single elseif would eliminate the usefulness of the ternary syntax:

// In this case, valid is now an int that can hold several states
int valid = -1; 

/* valid = SomeMethodCallThatReturnsValid() */

String status = null;

if (valid == 1) {
    status = "VALIDATED";
} 
else if (valid == 0) {
    status = "INVALID"; // A simple not-valid indicator
} 
else if (valid == -1) {
    status = "UNINITIALIZED_ERROR";
}
else {
    status = "UNKNOWN_ERROR";
}

We've already bounced ourselves back to the land of null and forced mutability (can you hear those poor, helpless puppies trying to learn how to swim?). We will encounter similar (or potentially worse) limitations in the case of a for/foreach construct.

List<User> users = new List<User>();

for (DBRow row : databaseQuery(queryArgs)) {
    users.add(new User(row.get("username"), row.get("first_name"), row.get("last_name"));
}

Admittedly, this last block isn't that bad, but it still irks me to have to declare a mutable list to facilitate it. We probably could work around with a builder pattern but (a) as far as I know Java lacks a builtin List Builder and (b) We'd still probably be "building" a mutable list.

These paradigms obviously don't restrict or limit everyone, but those of us who try to pay attention to immutability and the like start to grind our teeth awfully fast when encountering these patterns. There has to be a better way, right? How do other languages handle these constructs and related problems?

The Pythonic Way

For those that know Python — a language that I have long loved, and using Scala has only reinforced that as I learn & understand more of the FP goodness in Python — there are some nice tricks to alleviate the previously demonstrated pain (and save those puppies). Unfortunately, some of these tricks also stink a bit too much like syntactic magic; I tend to think at least one of them violates Python's "explicit over implicit" rules too.

# Again, valid is now an int that can hold several states
valid = -1 

/* valid = some_method_to_set_valid() */

if valid == 1:
    status = "VALIDATED"
elif valid == 0:
    status = "INVALID"
elif valid == -1:
    status = "UNINITIALIZED_ERROR"
else:
    status = "UNKNOWN_ERROR"

Here's that magic I'm talking about — despite us not explicitly declaring/initializing it, Python will instantiate a variable status at the outer scope (that is to say, outside of our if-else block). This is incredibly useful for cutting down line noise related to unecessary declaration of variables; unfortunately it is also a common confusion point for beginning Pythonistas.

*"Where the hell did this `status` variable come from? It wasn't ever declared in an outer scope!"*

While we've avoided initializing status to null, we still encounter the pesky mutability issue. We ultimately ended up with a variable status which can be mutated later. This isn't ideal, but as far as I know Python has no way to enable/enforce immutable state.

Python also can take us closer to an ideal state with the for/foreach constructs. My big gripe with the previous Java for loop code was the need to declare a mutable List and incrementally add to it. Python has a cure for what ails us here, in the form of generators — a feature which, as you'll see, I'd give my left pinkie (or maybe just the last millimeter of my left pinkie nail) to have in Scala as well.

Here's how a smart Pythonista might express our database processing loop in a way that limits the declaration of mutable variables:

def process_dbusers(dbResults):
    for entry in dbResults:
        yield User(row.field('username'), row.field('first_name'), 
                   row.field('last_name'))

users = process_dbusers(db_query(query_args))

With this code, we've already quickly wandered into the land of "Things Java Can't Even Pretend To Do" here (which I think highlights my amusement at people arguing Java's power over Python. It's all relative when you know the right syntax ... ). To best handle this expression we've constructed a higher- order function process_dbusers, which takes in a database results iterator, returning a list-like construct with the results. I want to emphasize the term list-like here because what we got is not, in fact, a list (rather it is "Iterable"). A quick peek at the REPL (if you don't already have IPython you are really missing out) evinces this:

users
# <generator object process_db at 0x10ca93690>

type(users)
# <type 'generator'>

Python supports a special type of expression (which again, I'd love to have in Scala and have been playing with ways to support) called a generator. In addition to being immutable, a generator is also lazy; it does not evaluate the entire list at creation, instead evaluating each member as it is first read (notably though, Python generators do not memoize and cannot be iterated repeatedly). Note that Python's yield keyword differs from Scala's; it may be easiest to think of it as a "super-return" or "return on crack" in that it actually suspends execution (via coroutines) after returning each value.

The 'generator' produced is actually an iterable value; requesting the 'next' item resumes execution, runs the next iteration of the internal for loop again, yields the return value and suspends again. Not only is this generator value immutable, but it should be significantly more efficient in many cases; if we don't need to store each User but process them as we read them, we should see a lot better resource usage.

There is one other way to express the previous statement without needing a nested function set or the 'yield' keyword, using "generator expressions". The downside to the yield based generators is that they must be wrapped inside another function, making them messy to inline with other code. With Generator Expressions we can get the same behavior in a compact, inline one-liner. A Generator Expression is syntactically identical to a Python List Comprehension with one key difference: You must enclose the expression in parentheses ( ( and ) ) instead of square brackets ( [ and ]). While a List Comprehension will return a list, enclosing that same statement in parentheses instead of brackets will produce a generator. This Generator Expression works as if we had written a multiline for construct with yield:

# Define a tuple of test values
items = ('Foo', 'bar', 'Baz', 'spam', 'eggs', 'apples', 'oranges')

# Use a List Comprehension, returns a list
itemList = [item for item in items]
# ['Foo', 'bar', 'Baz', 'spam', 'eggs', 'apples', 'oranges']
# Use a Generator Expression which returns a generator
itemGen  = (item for item in items)
# <generator object <genexpr> at 0x10cafa050>

# You can also filter these expressions
# Return a list which filters to return only items which are 'food'
foodList = [item for item in items if item in ('spam', 'eggs', 'apples', 'oranges')]
# ['spam', 'eggs', 'apples', 'oranges']
# The same filter, but as a generator expression
foodGen = (item for item in items if item in ('spam', 'eggs', 'apples', 'oranges'))
# <generator object <genexpr> at 0x10cafa140>
# Generators are iterable ...
foodGen.next()
# 'spam'
foodGen.next()
# 'eggs'

We've certainly, on the Python side, gotten closer to a world which can exist without unecessary variable initialization and sane immutability... but aren't entirely where I'd like to be. To understand things better let's look, finally, at what Scala lets us do.

The Scala Way

A rough take on the code I was optimizing last night can be used to highlight the power of Scala's way of solving the kinds of problems we are discussing. Here's a Scala version of our if-elseif-else construct from Java:

val valid: Int = someMethodCallThatSetsValidity() 

var status: String = null // ack, mutable *and* null inited - A scala programmer's worst nightmare

if (valid == 1) {
  status = "VALIDATED"
} else if (valid == 0) {
  status = "INVALID" 
} else if (valid == -1) {
  status = "UNINITIALIZED_ERROR"
} else { 
  status = "UNKNOWN_ERROR"
}

There are lots of things that are egregious here from the eyes of an experienced Scala programmer, but it is also pretty close to the style I would expect from a fairly new Scala programmer. There is of course a much better way to do this in Scala without any magic tricks.

One of the cool little edge behaviors of Scala's syntax is the ability to capture the return value of most block statements — even if-else statements. This lends itself to quickly shortening our code and enforcing immutability on the final value of status.

val valid: Int = someMethodCallThatSetsValidity() 

val status = if (valid == 1) { 
  "VALIDATED" // returns "VALIDATED"
} else if (valid == 0) {
  "INVALID"  // returns "INVALID"
} else if (valid == -1) {
  "UNINITIALIZED_ERROR" // returns "UNINITIALIZED_ERROR"
} else { 
  "UNKNOWN_ERROR" // returns "UNKNOWN_ERROR"
}

Much better! The way that Scala evaluates this code, you can think of each branch of the block (the body of each "if", "else if" and "else" statements) as an anonymous function. Because it is evaluated this way, Scala allows us to return values, which can be captured from the entire block (The use of the explicit return keyword is typically considered bad form in Scala which is why it is omitted here). If valid has a value of 1, this block will return "VALIDATED".

The type of safe in this code will be inferred to String, because each branch returns a String. Although in Scala it is usually recommended to let type inference do its work wherever possible and avoid explicit type annotations, variables capturing values from a block like this might be a good exception to the rule.

The argument that I make is that if the type inference system finds multiple differing types in each branch, it will search backwards on the type hierarchies for the closest "ancestor" type of each type. This can quickly lead to the type of status being something far too generic (or even unexpected in the case of a bug).

We can see this quickly by changing the return of our else branch to a boolean:

val status = if (valid == 1) { 
  "VALIDATED"
} else if (valid == 0) {
  "INVALID" 
} else if (valid == -1) {
  "UNINITIALIZED_ERROR"
} else { 
  false 
}

/* Turn on "Power Mode" in the repl to dump the type (:power) 
scala> :type status
Any
*/

This is not good, and often will result in a bug. If you were expecting String, you will be sorely disappointed. The easiest fix to this is to explicitly annotate an expected return type, which forces the compiler to help you enforce expectations:

val status: String = if (valid == 1) { 
  "VALIDATED"
} else if (valid == 0) {
  "INVALID" 
} else if (valid == -1) {
  "UNINITIALIZED_ERROR"
} else { 
  false 
}
/* Fails to compile!
<console>:29: error: type mismatch;
 found   : Boolean(false)
 required: String
         false 

*/

We can also easily simplify this whole block into a pattern match, which will read more clearly in Scala. Fortuitously, Scala also lets us capture the return values from match blocks, following the same rules (including the Type Annotation concerns) as in if-else.

import annotation.switch

// Even better, cut out one more variable storage by calling our valid check inline
val status: String = (someMethodCallThatSetsValidity(): @switch) match {
    case 1  => "VALIDATED"
    case 0  => "INVALID"
    case -1 => "UNINITIALIZED_ERROR"
    case default => "UNKNOWN_ERROR"
}

The use of the @switch annotation is a trick I learned from Josh Suereth's absolutely superb book Scala in Depth; when used with certain matches (they typically need to match numeric values) they ensure the compiler generates a much more efficient JVM bytecode for the match execution/evaluation.

I'll leave the evaluation of that last as an exercise for the reader. Fundamentally it is a simple restatement of our previous if-else blocks and should make sense.

Now that we've looked at branching block statements and value capture in Scala, we're left with one last item — for constructs (We aren't going to discuss monad operators such as foreach, map, flatmap, etc here).

for (x <- 1 until 42) x
// Loops 42 times, but returns / prints nothing

Unlike other the previously examined branching blocks (if-else and matches), Scala's for loops do not implicitly return the last value. I would make an educated guess that this is because most developers would not expect or require the default behavior of a for loop to generate return values. I'd also go so far as to posit that this is a very sane default.

Much like Python, to generate a return value from a for loop in Scala we must explicitly declare our intention to return a value. While (as I lamented previously) Scala lacks support for Python style generators, it does support a form of List Comprehensions which allow us to get the behavior we want. For a Python developer, the rules for constructing these in Scala may get quickly confusing. The first is that return is not a valid keyword at all inside of a Scala for statement (Incidentally, Python allows it as a 'valid' keyword, but the function will always return the result of the first loop iteration and never progress).

for (x <- 1 until 42) return x
/* Fails to compile 
<console>:29: error: return outside method definition
       for (x <- 1 until 42) return x 
*/

Instead, we need to use a special keyword, yield, to signify our intention to create a List Comprehension rather than a simple loop. Like in Python, yield changes the behavior of a for loop. However, the behavior of the yield keyword in Scala differs significantly from Python, as it does not invoke generator behavior. The use of yield inside a Scala for loop will produce results along the lines of those of Python's square bracket enclosed List Comprehensions:

for (x <- 1 until 42) yield x
/* scala.collection.immutable.IndexedSeq[Int] = Vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42)
*/

The output of the for loop which uses yield will always be an immutable Seq[_], letting us save that result for later usage much like we did with our Python generators and comprehensions. And of course, Scala's for loops allow for filters just like Python's:

val items = List("foo", "bar", "baz", "spam", "eggs", "apples", "oranges")

val itemList = for (item <- items) yield item
/* itemList: List[java.lang.String] = List(foo, bar, baz, spam, eggs, apples, oranges) */

val foodList = 
    for (item <- items if "spam" :: "eggs" :: "apples" :: "oranges" :: Nil contains item) 
        yield item
/* foodList: List[java.lang.String] = List(spam, eggs, apples, oranges) */

// We can also expand that for comprehension into the alternate syntax for clarity
val expandedFoodList = for {
  item <- items
  if "spam" :: "eggs" :: "apples" :: "oranges" :: Nil contains item
} yield item

// expandedFoodList: List[java.lang.String] = List(spam, eggs, apples, oranges)

The intention here wasn't to "bash" on Java or even champion Python or Scala; rather, to highlight the ways that different syntactic features add power, flexiblity and (in the case of things like mutability) potential safety to our code. Though I'm accused of being a Scala fanboy, I want to clearly reiterate generators as something that I still think Python gets as a major edge over Scala. I work in all three of the highlighted languages on a daily basis and though I probably enjoy Java the least, find gems and strengths in each tool as I switch between them (to be honest though, most of Java's gems these days exist in the JVM and the JDK libraries rather than the language).

It may easily be said that I am biased, but I consider the improved functionality highlighted in Python and Scala to be significantly more powerful (and yet less complex in many ways) than Java's approach. Mutating variables, null initializations and general spaghetti code for the sake of expressing something that should be simpler to express are all things that lead to workplace violence and big bonus checks for employees of straight jacket manufacturers.

With our improved examples in Scala, we didn't need to create any mutable placeholder values, initialize anything to null or worry about ending up with a still mutable list to contend with. What we got was a pure, unadulterated, cleanly constructed immutable Seq, just like grandmom used to bake! And it smells freaking delicious.

-b

User Configurable Type Filtering with Scala Type Classes

2011-07-13T00:00:00-07:00

When I woke up this morning and looked through my twitter mentions, I found this gem sitting there from the middle of the night:

The user in question is complaining that when using Casbah's DSL, it doesn't allow a MongoDB ObjectId as a valid type to the $lt operator. But as @justthor points out, it is entirely possible to use ObjectId with the $lt operator since it contains timestamp information (See the documentation for ObjectId if you want nitty gritty detail). When I wrote the code for $lt however, I needed to decide what types were valid and weren't valid; I can't exactly guarantee type safety wih a DSL like Casbah's, but I can enforce type sanity. Whether I forgot that you can use ObjectId in $lt or just decided that most people wouldn't need to is irrelevant --- I had in this case blocked a user from accomplishing something valid that they needed to.

It is a more than reasonable problem, and my initial reaction was "oh crap, I guess I need to patch that". But what I forgot is that a few releases back, I rearchitected Casbah to obviate this kind of problem. Casbah now allows for a user definable (or, if you prefer, "adjustable") type filter on any of its DSL operators. This is accomplished through a very simple application of Scala Type Classes, a term which gets batted around a lot in the Scala community, but few seem able to understand or articulate its meaning to us lesser mortals. Over the last few months I've come to understand Type Classes much more deeply than I think I ever expected, and applied these lessons to the design of my code. As I failed to document the power and usage of these features at the time, I am going to be writing some additional detailed articles about my understanding of Type Classes in the next few weeks, and this is the first of such explanations.

So the question at hand is, how exactly does Casbah allow us to do this magical type filtering that I just mentioned, without patching the driver or creating a new release? First, let's look at how Casbah used to do things before the introduction of the as-yet unexplained Type Class introduction.

/**
 * Trait to provide the $lt (Less Than) method on appropriate callers.
 *
 * Targets (takes a right-hand value of) String, AnyVal (see Scala docs but basically Int, Long, Char, Byte, etc)
 * DBObject and Map[String, Any].
 *
 *
 * @author Brendan W. McAdams <brendan@10gen.com>
 * @see http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-%3C%2C%3C%3D%2C%3E%2C%3E%3D
 */
trait LessThanOp extends QueryOperator {
  private val oper = "$lt" 

  def $lt(target: String) = op(oper, target)
  def $lt(target: java.util.Date) = op(oper, target)
  def $lt(target: AnyVal) = op(oper, target)
  def $lt(target: DBObject) = op(oper, target)
  def $lt(target: Map[String, Any]) = op(oper, target.asDBObject)
}

The op method is just a helper function which assists the DSL in constructing a valid MongoDB query under the covers; it's not particularly relevant to this discussion so let's leave it aside for now. What is important is to note that this is a very naive approach to the DSL game --- it is neither type safe or type sane. It has some very basic support for handling "special" types like String, java.util.Date, DBObject and Map[String, Any] but blindly allows anything which is AnyVal through. There are, in my opinion, three problems here:

Is blindly passing AnyVal a good idea? Typically it consists of things like Int, Char, Boolean, etc but can we be certain this is a safe assumption?
What if a user defines a custom type which they serialize to MongoDB as something which supports $lt --- How do they use that type with the DSL without patching Casbah itself? BigDecimal is an example of this; the current code does not allow it to be used at all with $lt.
What if a user has a domain specific reason to restrict certain of these types? I shouldn't even need to manufacture an example here: Users should be able to restrict types without patching code, if possible

These questions and the fact that I kept patching bugs, adding types, etc on user request led me to seek a new solution while working on Casbah 2.0. While searching for this solution I started to gain a rudamentary understanding of Type Classes and their power, and embarked upon a quest to give the users of Casbah flexible control over the types the DSL will and will not allow through. Of course, I neglected to adequately document those changes, until now.

After the conversion to Type Classes, here's what the $lt operator's code looks like today::

/**
 * Trait to provide the $lt (Less Than) method on appropriate callers.
 *
 * Targets (takes a right-hand value of) String, Numeric, JDK And Joda Dates, 
 * Array, DBObject (and DBList), Iterable[_] and Tuple1->22.
 *
 *
 * @author Brendan W. McAdams <brendan@10gen.com>
 * @see http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-%3C%2C%3C%3D%2C%3E%2C%3E%3D
 */
trait LessThanOp extends QueryOperator {
  private val oper = "$lt"

  def $lt(target: String) = op(oper, target)
  def $lt(target: DBObject) = op(oper, target)
  def $lt(target: Array[_]) = op(oper, target.toList)
  def $lt(target: Tuple1[_]) = op(oper, target.productIterator.toList)
  def $lt(target: Tuple2[_, _]) = op(oper, target.productIterator.toList)
  /** SNIP a bunch of individual tuple handling */
  def $lt(target: Iterable[_]) = op(oper, target.toList)
  def $lt[T: ValidDateOrNumericType](target: T) = op(oper, target)
}

You should, at this point, notice a fairly stark contrast between the old code and the new. While we still have specific allowances for DBObject, and have included allowances for things that are Iterable or Array-like, the explicit support for Dates, Numbers and Booleans (Primitive Numbers and Booleans being part of AnyVal) is gone. Or are they?

The clever reader may have noticed something new lurking at the bottom of this code::

  def $lt[T: ValidDateOrNumericType](target: T) = op(oper, target)

This specific method is the pivot point of our new filter system: it is what allows users to independently define what types, beyond the hardcoded ones above, are and are not allowed to pass into the $lt operator.

But what does it all mean?

Of Type Classes and Context Bounds

The method above defines a type parameter of [T: ValidDateOrNumericType]; you're probably used to seeing covariance (T <: ValidDateOrNumericType) or contravariance (T >: ValidDateOrNumericType), but may not have encountered this notation before now. The single colon type boundary is known as a Context Bound It does not, as first guess might tell you, say that T must be exactly a ValidDateOrNumericType.

Context Bounds are a fairly new syntax, recently introduced to the type parameter system in Scala 2.8. What they give us is a shortcut for a type-dependent implicit argument (they also work with Manifests, which I'll explain in a future post). When the Scala Compiler parses our code it will actually produce the following statement, which is how we would have said this in versions prior to 2.8::

    def $lt[T](target: T)(implicit evidence$1: ValidDateOrNumericType[T]) = op(oper, target)

What the Context Boundary syntax says is that we want to accept a generic type T, as long as there is an implicit instance of ValidDateOrNumericType[T] available. With this simple statement, what we say is quite literally as long as an implicit instance of ValidDateOrNumericType is available for type T, it is a valid type for this method. This means that you, as a user, can quickly change what type are and aren't allowed into this $lt method merely by adjusting the implicit scope. Powerful, no?

In this case, ValidDateOrNumericType itself is a completely empty trait (but, as we'll see in my next post can actually become quite powerful with the introduction of some methods!) which is used as a simple filter. The default imports for Casbah's DSL automatically gives you several predefined filter types::

implicit object JDKDateDoNOk extends JDKDateOk with ValidDateOrNumericType[java.util.Date]
implicit object JodaDateTimeDoNOk extends JDKDateOk with ValidDateOrNumericType[org.joda.time.DateTime]
implicit object BigIntDoNOk extends BigIntOk with ValidDateOrNumericType[BigInt]
implicit object IntDoNOk extends IntOk with ValidDateOrNumericType[Int]
implicit object ShortDoNOk extends ShortOk with ValidDateOrNumericType[Short]
implicit object ByteDoNOk extends ByteOk with ValidDateOrNumericType[Byte]
implicit object LongDoNOk extends LongOk with ValidDateOrNumericType[Long]
implicit object FloatDoNOk extends FloatOk with ValidDateOrNumericType[Float]
implicit object BigDecimalDoNOk extends BigDecimalOk with ValidDateOrNumericType[BigDecimal]
implicit object DoubleDoNOk extends DoubleOk with ValidDateOrNumericType[Double]

All one has to do to introduce a new type as valid to $lt is define an implicit instance of ValidDateOrNumeric[<YourType>]. Circling back to our original problem, the solution to "How do I allow ObjectId in $lt" is quite easy:

implicit object ObjectIdOK extends ValidDateOrNumericType[org.bson.types.ObjectId]

As long as that is in scope of your code when you invoke your query statement (such as "_id" $lt new ObjectId(timestamp)) it will construct a valid query to send to MongoDB.

Of course, there is a lot more that we can do with Type Classes than mere filtering. In my next post, I'll show you how to take decoupling and separation of concerns to the ultimate extreme using Type Classes.

MapReduce with MongoDB 1.8 and Java

2011-02-28T00:00:00-08:00

In my last post, I introduced the new MapReduce features introduced in MongoDB 1.8, which is now available as a release candidate. Most importantly the temporary collection system has gone away, now requiring that you specify an output parameter. With that required output comes new options for how to create incremental output using the merge and reduce output modes.

As I write this, we are prepping new releases of our Java Driver (v2.5) and our Scala Driver, Casbah (v2.1) which are intended to support MongoDB 1.8's new features including incremental MapReduce. Since I implemented the APIs for the new MapReduce output in both drivers, I thought I'd demonstrate the application of these new output features to the previous dataset. This post is focused on the Java API, but a Scala one will likely follow.

As a reminder (or a primer for those who skipped my last post), I've been testing the 1.8 MapReduce using a dataset and MapReduce job originally created to test the MongoDB+Hadoop Plugin. It consists of daily U.S. Treasury Yield Data for about 20 years; the MapReduce task calculates an annual average for each year in the collection. You can grab a copy of the entire collection in a handy mongoimport friendly datadump from the MongoDB+Hadoop repo; here's a quick snippet of it:

{ "_id" : ISODate("1990-01-10T00:00:00Z"), "dayOfWeek" : "WEDNESDAY", "bc3Year" : 7.95, "bc5Year" : 7.92, "bc10Year" : 8.03, "bc20Year" : null, "bc1Month" : null, "bc2Year" : 7.91, "bc3Month" : 7.75, "bc30Year" : 8.11, "bc1Year" : 7.77, "bc7Year" : 8, "bc6Month" : 7.78 }
{ "_id" : ISODate("1990-01-11T00:00:00Z"), "dayOfWeek" : "THURSDAY", "bc3Year" : 7.95, "bc5Year" : 7.94, "bc10Year" : 8.04, "bc20Year" : null, "bc1Month" : null, "bc2Year" : 7.91, "bc3Month" : 7.8, "bc30Year" : 8.11, "bc1Year" : 7.77, "bc7Year" : 8.01, "bc6Month" : 7.8 }
{ "_id" : ISODate("1990-01-12T00:00:00Z"), "dayOfWeek" : "FRIDAY", "bc3Year" : 7.98, "bc5Year" : 7.99, "bc10Year" : 8.1, "bc20Year" : null, "bc1Month" : null, "bc2Year" : 7.93, "bc3Month" : 7.74, "bc30Year" : 8.17, "bc1Year" : 7.76, "bc7Year" : 8.07, "bc6Month" : 7.8100000000000005 }
{ "_id" : ISODate("1990-01-16T00:00:00Z"), "dayOfWeek" : "TUESDAY", "bc3Year" : 8.13, "bc5Year" : 8.11, "bc10Year" : 8.2, "bc20Year" : null, "bc1Month" : null, "bc2Year" : 8.1, "bc3Month" : 7.89, "bc30Year" : 8.25, "bc1Year" : 7.92, "bc7Year" : 8.18, "bc6Month" : 7.99 }

The map function I'm using extracts the year from the date, and the 10 year benchmark value:

function m() { 
    key = typeof( this._id ) == "number" ? this._id : this._id.getYear() + 1900; 
    emit( key, { count: 1, sum: this.bc10Year } ) ;
}

... while the reduce function aggregates the data by year, creating a set that can be averaged. Remember that MongoDB reduce tasks have to be able to be called repeatedly, so the output is crafted to match the input: something that becomes even more important when we say, ask MongoDB to re-reduce our output with the old data.

function r( year, values ) { 
  var n = { count: 0, sum: 0 } 
  for ( var i = 0; i < values.length; i++ ){ 
      n.sum += values[i].sum; 
      n.count += values[i].count; 
  } 
   
  return n; 
}

We'll round it all out out with a quick and dirty finalize function which can calculate the current average. Note that I'm keeping all the intermediate data around for demonstrating "reduce" mode.

function f( year, value ){
  value.avg = value.sum / value.count;
  return value;
}

First, we'll need to stick these functions into some Java strings to pass around for our testing:

        String m = "function() { key = typeof( this._id ) == \"number\" ? this._id : this._id.getYear() + 1900;" +
                   "emit( key, { count: 1, sum: this.bc10Year } );";

        String r = "function( year, values ) { var n = { count: 0, sum: 0};" +
                   " for ( var i = 0; i < values.length; i ++ ) { n.sum += values[i].sum; " + 
                   " n.count += values[i].count; } return n; }";

        String f = "function( year, value ) { value.avg = value.sum / value.count; return value; }";

The Java API now allows you to pass an optional MapReduceCommand.OutputType value, which controls the type of output received. If one is not specifed, the output collection is assumed to be REPLACE --- namely, the standard mode in which the named collection is replaced completely with the output of the MapReduce job. Looking at INLINE as our example, we can call the new method in collection. Feel free to set the output collection name to null or a throwaway value; it is ignored by the Java driver in Inline output mode.

MapReduceOutput out = coll.mapReduce(m, r, null, MapReduceCommand.OutputType.INLINE, null);

for ( DBObject obj : out.results() ) {
    System.out.println( obj );
}

Which should output each DBObject in the results to the screen. The new MapReduceOutput code detects the result set type from MongoDB and provides it in results() as a Iterable<DBObject>---whether the results are INLINE or stored in a collection. Note that I did not specify the finalize function here, as the interface on DBCollection doesn't accept it as a parameter. Alternately, we could construct a MapReduceCommand instance, which allows us to set finalize:

MapReduceCommand cmd = new MapReduceCommand( coll, m, r, null, 
                                             MapReduceCommand.OutputType.INLINE, null);

cmd.setFinalize(f);

out = coll.mapReduce(cmd);

for ( DBObject obj : out.results() ) {
    System.out.println( obj );
}

Using the MERGE and REDUCE Output Modes follows much the same pattern. I'll leave figuring out MERGE as an exercise for the reader (it should be easy if you read my last post) but here's how we would handle a REDUCE on two halves of the same year.

cmd = new MapReduceCommand( coll, m, r, "yield_historical.out", 
                            MapReduceCommand.OutputType.REDUCE, 
                            new BasicDBObject("_id", new BasicDBObject(
                                                        "$gte", new Date( 101, 0, 1 )
                                                     ).append(
                                                        "$lte", new Date( 101, 5, 1 )
                                                     )
                                            )
                           );

cmd.setFinalize(f);

/** Ignore output of first half... */
coll.mapReduce(cmd);

/** Reduce in second half */
cmd = new MapReduceCommand( coll, m, r, "yield_historical.out", 
                            MapReduceCommand.OutputType.REDUCE, 
                            new BasicDBObject("_id", new BasicDBObject(
                                                        "$gt", new Date( 101, 5, 1 )
                                                     ).append(
                                                        "$lte", new Date( 101, 11, 31 )
                                                     )
                                             )
                           );

cmd.setFinalize(f);

out = coll.mapReduce(cmd);

for ( DBObject obj : out.results() ) {
    System.out.println( obj );
}

That's it! Go forth, and MapReduce...

A Look At MongoDB 1.8's MapReduce Changes

2011-01-27T00:00:00-08:00

MongoDB 1.7.5 shipped yesterday, and is expected to be the last 'beta' release of what will become MongoDB 1.8. As part of the release, I've been doing testing of the new MapReduce functionality and thought this a good time to highlight those changes for people.

If you aren't new to MongoDB MapReduce, the most important thing to note since MongoDB 1.6.x is that temporary collections are gone; it is now required to specify an output. Previously, if you omitted the out argument MongoDB would create a temporary collection and return its name with the job results; In non-sharded MongoDB setups these temporary collections would go out of scope and be cleaned up when the connection closed. Unfortunately, for sharded setups it wasn't possible to safely clean these up–--they would remain behind and clutter up the database. For this and other reasons the temporary collection feature was removed. There is good news though: they've been replaced with an even better system for saving the results of MapReduce jobs!

While the out argument is now a required parameter in MapReduce jobs, it has a number of options for controlling what MongoDB does with results. If you're running a truly one-off job where you don't need to keep the results later, MongoDB now supports returning results "inline". Be careful here though: your results are being returned in a single document and are subject to the document size limitations of MongoDB (16MB per document in 1.8). To use inline results, set the value of out to a document {inline: 1}. The result object will contain an additional key results which contains the MapReduce output; the result field will be omitted.

As with previous versions of MongoDB, you can specify a collection name (as a string) in the out argument. If the named collection already exists MongoDB will replace it entirely with the MapReduce results. Along with the inline mode, MongoDB 1.8 introduces support for "merge" and "reduce" output modes; instead of replacing the target collection MongoDB can be instructed to reconcile the MapReduce results with the existing data. To use these modes, set the value of out to a document with a key of either "merge" or "reduce" and a value of the collection to save to.

The difference in "merge" and "reduce" has to do with MongoDB does when it encounters duplicate keys in both the existing collection and the MapReduce results. In "merge" mode, MongoDB will simply overwrite the existing key with the new one from the MapReduce output. In "reduce" mode, MongoDB will run the reduce function again with both the new and old data, saving those results to the collection (you remembered to make your reduce function idempotent, right?). UPDATE: If you specified a "finalize" function, MongoDB will re-run this after the "reduce" runs.

Now that I've thoroughly confused you, lets dig into examples of each of these behaviors. I've been testing the 1.8 MapReduce using a dataset and MapReduce job originally created to test the MongoDB+Hadoop Plugin. It consists of daily U.S. Treasury Yield Data for about 20 years; the MapReduce task calculates an annual average for each year in the collection. You can grab a copy of the entire collection in a handy mongoimport friendly datadump from the MongoDB+Hadoop repo; here's a quick snippet of it:

{ "_id" : ISODate("1990-01-10T00:00:00Z"), "dayOfWeek" : "WEDNESDAY", "bc3Year" : 7.95, "bc5Year" : 7.92, "bc10Year" : 8.03, "bc20Year" : null, "bc1Month" : null, "bc2Year" : 7.91, "bc3Month" : 7.75, "bc30Year" : 8.11, "bc1Year" : 7.77, "bc7Year" : 8, "bc6Month" : 7.78 }
{ "_id" : ISODate("1990-01-11T00:00:00Z"), "dayOfWeek" : "THURSDAY", "bc3Year" : 7.95, "bc5Year" : 7.94, "bc10Year" : 8.04, "bc20Year" : null, "bc1Month" : null, "bc2Year" : 7.91, "bc3Month" : 7.8, "bc30Year" : 8.11, "bc1Year" : 7.77, "bc7Year" : 8.01, "bc6Month" : 7.8 }
{ "_id" : ISODate("1990-01-12T00:00:00Z"), "dayOfWeek" : "FRIDAY", "bc3Year" : 7.98, "bc5Year" : 7.99, "bc10Year" : 8.1, "bc20Year" : null, "bc1Month" : null, "bc2Year" : 7.93, "bc3Month" : 7.74, "bc30Year" : 8.17, "bc1Year" : 7.76, "bc7Year" : 8.07, "bc6Month" : 7.8100000000000005 }
{ "_id" : ISODate("1990-01-16T00:00:00Z"), "dayOfWeek" : "TUESDAY", "bc3Year" : 8.13, "bc5Year" : 8.11, "bc10Year" : 8.2, "bc20Year" : null, "bc1Month" : null, "bc2Year" : 8.1, "bc3Month" : 7.89, "bc30Year" : 8.25, "bc1Year" : 7.92, "bc7Year" : 8.18, "bc6Month" : 7.99 }

The map function I'm using extracts the year from the date, and the 10 year benchmark value:

function m() { 
    key = typeof( this._id ) == "number" ? this._id : this._id.getYear() + 1900; 
    emit( key, { count: 1, sum: this.bc10Year } ) ;
}

While the reduce function aggregates the data by year, creating a set that can be averaged. Remember that MongoDB reduce tasks have to be able to be called repeatedly, so the output is crafted to match the input: something that becomes even more important when we say, ask MongoDB to re-reduce our output with the old data.

function r( year, values ) { 
  var n = { count: 0, sum: 0 } 
  for ( var i = 0; i < values.length; i++ ){ 
      n.sum += values[i].sum; 
      n.count += values[i].count; 
  } 
   
  return n; 
}

We'll round it all out out with a quick and dirty finalize function which can calculate the current average. Note that I'm keeping all the intermediate data around for demonstrating "reduce" mode.

function f( year, value ){
  value.avg = value.sum / value.count;
  return value;
}

First, a quick look at "inline" mode (I'll leave plain old name a collection as an exercise to you, my humble reader).

> res = db.runCommand(
...   { 
...     "mapreduce": "yield_historical.in",
...     "map": m,
...     "reduce": r,
...     "finalize": f,
...     "query" : { "_id" : { "$gt" : new Date(2000, 0, 1) } },
...     "verbose" : true , 
...     "out" : { "inline" : 1 }
...   }
... )
{
    "results" : [
        {
            "_id" : 1990,
            "value" : 8.552400000000002
        },
        /* ... */
        {
            "_id" : 2010,
            "value" : 3.3255026455026435
        }
    ],
    "timeMillis" : 218,
    "timing" : {
        "mapTime" : NumberLong(168),
        "emitLoop" : 215,
        "total" : 218
    },
    "counts" : {
        "input" : 2690,
        "emit" : 2690,
        "output" : 11
    },
    "ok" : 1
}

To demonstrate "merge" and "reduce" mode, I'm going to use queries to break out the data a bit. Lets look first at "merge", by first running MapReduce against the first half of the data, and then merge in the second half.

> res = db.runCommand(
...   { 
...     "mapreduce": "yield_historical.in",
...     "map": m,
...     "reduce": r,
...     "finalize": f,
...     "query" : { "_id" : { "$lt" : new Date(2000, 0, 1) } },
...     "verbose" : true , 
...     "out" : "yield_historical.merged",
...   }
... )
{
    "result" : "yield_historical.merged",
    "timeMillis" : 223,
    "timing" : {
        "mapTime" : NumberLong(166),
        "emitLoop" : 217,
        "total" : 223
    },
    "counts" : {
        "input" : 2503,
        "emit" : 2503,
        "output" : 10
    },
    "ok" : 1
}
> db.yield_historical.merged.find({}, {"value.avg": 1})
{ "_id" : 1990, "value" : { "avg" : 8.552400000000002 } }
{ "_id" : 1991, "value" : { "avg" : 7.8623600000000025 } }
{ "_id" : 1992, "value" : { "avg" : 7.008844621513946 } }
{ "_id" : 1993, "value" : { "avg" : 5.866279999999999 } }
{ "_id" : 1994, "value" : { "avg" : 7.085180722891565 } }
{ "_id" : 1995, "value" : { "avg" : 6.573920000000002 } }
{ "_id" : 1996, "value" : { "avg" : 6.443531746031743 } }
{ "_id" : 1997, "value" : { "avg" : 6.353959999999992 } }
{ "_id" : 1998, "value" : { "avg" : 5.262879999999994 } }
{ "_id" : 1999, "value" : { "avg" : 5.646135458167332 } }
>

That gives us our first half of the data; we ran that with a normal named collection output. Lets merge in the second half:

> res = db.runCommand(
...   { 
...     "mapreduce": "yield_historical.in",
...     "map": m,
...     "reduce": r,
...     "finalize": f,
...     "query" : { "_id" : { "$gt" : new Date(2000, 0, 1) } },
...     "verbose" : true , 
...     "out" : { "merge" : "yield_historical.merged" },
...   }
... )
{
    "result" : "yield_historical.merged",
    "timeMillis" : 242,
    "timing" : {
        "mapTime" : NumberLong(173),
        "emitLoop" : 236,
        "total" : 242
    },
    "counts" : {
        "input" : 2690,
        "emit" : 2690,
        "output" : 21
    },
    "ok" : 1
}

> db.yield_historical.merged.find({"_id": {$gt: 1998}}, {"value.avg": 1}) 
{ "_id" : 1999, "value" : { "avg" : 5.646135458167332 } }
{ "_id" : 2000, "value" : { "avg" : 6.030278884462145 } }
{ "_id" : 2001, "value" : { "avg" : 5.020685483870969 } }
{ "_id" : 2002, "value" : { "avg" : 4.61308 } }
{ "_id" : 2003, "value" : { "avg" : 4.013879999999999 } }
{ "_id" : 2004, "value" : { "avg" : 4.271320000000004 } }
{ "_id" : 2005, "value" : { "avg" : 4.288880000000001 } }
{ "_id" : 2006, "value" : { "avg" : 4.7949999999999955 } }
{ "_id" : 2007, "value" : { "avg" : 4.634661354581674 } }
{ "_id" : 2008, "value" : { "avg" : 3.6642629482071714 } }
{ "_id" : 2009, "value" : { "avg" : 3.2641200000000037 } }
{ "_id" : 2010, "value" : { "avg" : 3.3255026455026435 } }

To close out, lets take "reduce" mode for a quick spin. We'll select a half of a year for the first part, and then reduce in the second half.

> res = db.runCommand(
...   { 
...     "mapreduce": "yield_historical.in",
...     "map": m,
...     "reduce": r,
...     "finalize": f,
...     "query" : { "_id" : { 
...         "$gte": new Date(2001, 0, 1),
...         "$lte" : new Date(2001, 5, 1) 
...     } },
...     "verbose" : true , 
...     "out" : "yield_historical.reduced",
...   }
... )
{
    "result" : "yield_historical.reduced",
    "timeMillis" : 21,
    "timing" : {
        "mapTime" : NumberLong(6),
        "emitLoop" : 17,
        "total" : 21
    },
    "counts" : {
        "input" : 105,
        "emit" : 105,
        "output" : 1
    },
    "ok" : 1
}
> db.yield_historical.reduced.find()
{ "_id" : 2001, "value" : { "count" : 105, "sum" : 539.5599999999998, "avg" : 5.138666666666665 } }

That handles the first half... Let's grab the second:

> res = db.runCommand(              
...   { 
...     "mapreduce": "yield_historical.in",
...     "map": m,
...     "reduce": r,
...     "finalize": f,
...     "query" : { "_id" : { 
...         "$gt": new Date(2001, 5, 1),
...         "$lte" : new Date(2001, 11, 31) 
...     } },
...     "verbose" : true , 
...     "out" : { "reduce" : "yield_historical.reduced" },
...   }
... )
{
    "result" : "yield_historical.reduced",
    "timeMillis" : 26,
    "timing" : {
        "mapTime" : NumberLong(9),
        "emitLoop" : 22,
        "total" : 26
    },
    "counts" : {
        "input" : 143,
        "emit" : 143,
        "output" : 1
    },
    "ok" : 1
}
> db.yield_historical.reduced.find()
{ "_id" : 2001, "value" : { "count" : 248, "sum" : 1245.1299999999997, "avg" : 5.020685483870967 } }

Of course, this does us no good if the results don't add up. A quick comparison between the 'merged' output and the 'reduced' output validates our code:

> db.yield_historical.reduced.find({_id: 2001})
{ "_id" : 2001, "value" : { "count" : 248, "sum" : 1245.1299999999997, "avg" : 5.020685483870967 } }
> db.yield_historical.merged.find({_id: 2001}) 
{ "_id" : 2001, "value" : { "count" : 248, "sum" : 1245.1300000000003, "avg" : 5.020685483870969 } }

There are some minor differences at a decimal level since we are working with floating point numbers here, but the results are the same.

These new MapReduce output parameters are available in MongoDB as of version 1.7.4 (which is part of the unstable/development branch) and will ship with MongoDB 1.8. Leave a comment; I'd love to hear what clever tricks you can pull off with these new options.

Exploring Scala with MongoDB

2011-01-10T00:00:00-08:00

2010 proved to be a great year for growth and adoption of many fledgling technologies---not least among them, MongoDB and Scala. Scala is designed as an alternative language for the Java platform, with a focus on scalability. It merges many of the Object Oriented concepts of languages like Java and C++ with the functional tools of Erlang, Haskell and Lisp with a bit of the dynamic natures of modern languages like Ruby and Python. This flexible nature has sped Scala's adoption in the technology stacks of platforms like LinkedIn, Twitter, FourSquare and many more. By running on the JVM Scala has a strong affinity for working alongside existing Java applications, which allows users to build on their existing technology investments.

For 2011, MongoDB has added official support for Scala with the release of Casbah, a Scala driver for MongoDB. Casbah is built around the existing MongoDB Java Driver to give it a strong foundation, but designed to take advantage of many of the idioms of Scala such as a strong collections library, fluid syntax for building DSLs and functional concepts like closures and currying.

Because it is designed to be easy to work with for Scala users, Casbah introduces a more 'friendly' syntax for creating MongoDB Objects, using Scala's Map syntax:

 
import com.mongodb.casbah.Imports._

/** Create an object directly */
val newObj = MongoDBObject("foo" -> "bar",
                           "x" -> "y",
                           "pie" -> 3.14,
                           "spam" -> "eggs")

/** Or, use a builder interface */
val builder = MongoDBObject.newBuilder
builder += "foo" -> "bar"
builder += "x" -> "y"
builder += ("pie" -> 3.14)
builder += ("spam" -> "eggs", "mmm" -> "bacon")
val newObj = builder.result

The goal of this syntax is to be more readable, similar to what one might expect from a dynamic language like Ruby or Python. In contrast, the same statements in Java tend to be more verbose:

import com.mongodb.*;

DBObject newObj = new BasicDBObject();
newObj.put("foo", "bar");
newObj.put("x", "y");
newObj.put("pie", 3.14);
newObj.put("spam", "eggs");

/** or, builder style */

BasicDBObjectBuilder builder = BasicDBObjectBuilder.start();
builder.add("foo", "bar");
builder.add("x", "y");
builder.add("pie", 3.14);
builder.add("spam", "eggs");
builder.add("mmm", "bacon");
DBObject newObj = builder.get();

The semantics of working with Collections and Cursors in Casbah are similar to the Java driver they wrap, with a bit of Scala-friendly syntactic sugar added for things like for comprehensions. Where Casbah really shines is in its use of a DSL syntax for creating MongoDB Queries.

/** This Query Object ... */
val query = new MongoDBObject(
                "foo" -> MongoDObject("$gte" -> 5, "$lte" -> 10),
                "baz" -> 5,
                "x" -> "y",
                "n" -> "r"
            )
/** Can be constructed instead with the Query DSL: */
val queryDSL = ("foo" $gte 5 $lte 10) ++ ("baz" -> 5) ++ ("x" -> "y") ++ ("n" -> "r")

/** Easily create negated statements. 
    Instead of a nested DBObject constructor like this: */

val ltGt = MongoDBObject(
            "foo" -> MongoDBObject(
                "$not" -> MongoDBObject(
                    "$gte" -> 15, 
                    "$lt" -> 35.2, 
                    "$ne" -> 16)
                )
            )

/** Use Casbah's Query DSL to say it much simpler */
val ltGtDSL = "foo" $not { _ $gte 15 $lt 35.2 $ne 16 }

All of MongoDB's $ Operators including Geospatial Queries are supported by Casbah's DSL.

This is just a small taste of what Casbah and Scala offer to the MongoDB user, but we encourage you to explore more. Version 2.01 is now available for download.

Talking to ActiveDirectory from IronPython

2009-10-20T00:00:00-07:00

We're building a new intranet system at work, and I've been toying with a few things that the Windows admin asked for. Namely, since the secretaries here will update the intranet data to add people's Work & Emergency contact numbers, AIM handles, email addresses, etc. that we find a way to keep it all in sync with ActiveDirectory. Thereby keeping all the Outlooks and Blackberries up to date with the latest contact information.

This seemed like a fairly reasonable request, presuming we could figure out how to do it and since I've been using Mono and IronPython a lot more lately, I figured there would be a way to accomplish it. Most of the information I found online was either really old and/or crappy docs for doing it in C#, or more commonly using PowerShell or VBScript. So, I managed to poke around and sort out how to get IronPython on Mono (IronPython 2.6RC + Mono 2.4.2.3) to find and update our users.

The end result is that I can now, from IronPython, find and update valid information on ActiveDirectory entries to reflect the latest and greatest information. One thing to note, the MS .Net ActiveDirectory APIs (System.DirectoryServices, which is mirrored in Mono) do something that confused and annoyed me. There are a limited set of 'valid' attribute keys for a user object in Active Directory (Which is really just LDAP, in case you didn't know). The DirectoryEntry object has a Properties attribute, which contains a hashmap of these values.

The object will not allow you to set an "Invalid" key (see this list for valid keys). But if you call .Properties.Keys you only get back the Properties that have values set. So, it doesn't appear to be possible to actually ask What keys are valid? and do some introspective programming. I have written a wrapper class to make the DirectoryEntry properties look a bit more pythonic (but disabled support for multi-value attributes for now) - at some point in the near future i'll likely add in a "valid value" filter.

The end result is, if I want to find my own user in ActiveDirectory by my name, I can do the following from the IronPython console:

>>> import ad_util
>>> adh = ad_util.ActiveDirectorySearcher('mydomaincontroller.hostname.or.ip', 'my.domain', 'myUsernameAllowedToChangeObjects', 'myPassword')
>>> adh
<ActiveDirectorySearcher object at 0x000000000000002D>
>>> userObj = adh.find_name('McAdams', 'Brendan')
[debug] Searching activedirectory for (&amp;(objectCategory=user)(objectClass=person)(sAMAccountName=*)(sn=McAdams*)
(givenname=Brendan*)).  Allow multiple results? False
>>> userObj
<DirectoryEntryHelper:{sn=McAdams,givenName=Brendan,mail=None,sAMAccountName=brendan.mcadams}>

You'll note the object returned is a "DirectoryEntryHelper" type; this is a crappy little wrapper class I put together to simplify attribute access, etc. You can tell any of the find_ methods to return a raw .Net API object instead of wrapped by passing the kwarg nowrap=True; find_name() takes last_name, first_name. Several other utility methods exist on the class including find by account name. Note that, in order to filter out entries in the Global Address List I'm requiring the search to find objects who have sAMAccountNames. I know the bare minimum about ActiveDirectory, but my Windows admin here tells me that sAMAccountName is a required and unique attribute on any actual domain account object.

You can see the keys that are already defined on the object with the keys helper method I added onto DirectoryEntryHelper:

>>> userObj.keys
['lockouttime', 'primarygroupid', 'msexchuseraccountcontrol', 'distinguishedname', 'cn', 'dscorepropagationdata', 
'whencreated', 'logoncount', 'msexchhomeservername', 'objectclass', 'memberof', 'lastlogontimestamp', 'displayname',    'msexchalobjectversion', 'objectguid', 'whenchanged', 'badpwdcount', 'useraccountcontrol', 'badpasswordtime', 'name', 
'samaccountname', 'mdbusedefaults', 'accountexpires', 'countrycode', 'msds-supportedencryptiontypes', 'homedirectory', 
'userprincipalname', 'lastlogon', 'objectsid', 'givenname', 'homedrive', 'usncreated', 'admincount', 'instancetype', 
'codepage', 'physicaldeliveryofficename', 'samaccounttype', 'sn', 'objectcategory', 'telephonenumber', 'pwdlastset', 
'usnchanged', 'lastlogoff', 'initials']

This returns a comprehended list of the keys (Rather than calling resultObj.Properties.Keys, which returns a Hashtable of HashKeys. It just makes life easier.

If I want to setup my mobile phone # (which isn't already set already) in Active Directory:

>>> userObj.mobile
# Nothing returned as mobile isn't set...
>>> userObj.mobile = '(646) 555-1212'
# It is committed on set to Active Directory, so another find gets it...
>>> newUserObj = test.adh.find_name('McAdams', 'Brendan')   
[debug] Searching activedirectory for (&amp;(objectCategory=user)(objectClass=person)(sAMAccountName=*)(sn=McAdams*)
(givenname=Brendan*)).  Allow multiple results? False 
>>> newUserObj.mobile
'(646)555-1212'

I'll leave the rest as an exercise for the reader, but I'm interested in comments and changes. You can fetch the latest code from my BitBucket toybox. I've also pasted it, for posterity sake, below the fold.

#!IronPython Specific Script!
# 
# Brendan W. McAdams <bwmcadams@gmail.com>
#
#------------------------------------------------- 
# No copyright or licensing, made public
# as example code.  Feel free to use it as you like;
# No warranty, liability or guarantees are implied.  
# In other words, if you use it YOU ARE ON YOUR OWN. 
#-------------------------------------------------
#
# Utility script to interface from IronPython to 
# MS Active Directory, allowing queries of properties 
# such as telephone numbers, and changes to said properties
# if you'd like to update them.
# 
# Note that your authentication user has to have valid permissions.
# If you're uncertain as to what permissions are needed, please see
# your AD Admin.

import clr 

# Add the System.DirectoryServices Mono DLL in.
# not sure on Windows what you need to add,
# but for Mono/Linux you just need to set IRONPYTHONPATH
# to include the location of the dll

clr.AddReference('System.DirectoryServices.dll')

import sys

# Import the LDAP / ActiveDirectory interface
from System.DirectoryServices import *

# A few predefined values to simplify LDAP querying, pilfered from various internet-ey type places.
BINARY_PROPS = ('objectguid', 'objectsid', 'msexchmailboxsecuritydescriptor', 'msexchmailboxguid')
ACCOUNT_QUERY = "(&amp;(objectCategory=user)(objectClass=person)(sAMAccountName=_ACCOUNTNAME_))"
PRINCIPAL_QUERY = "(&amp;(objectCategory=user)(objectClass=person)(userPrincipalName=_PRINCIPALID__))"
GROUP_QUERY = "(&amp;(objectCategory=group)(sAMAccountName=_GROUPNAME_))"
PARTIAL_LAST_QUERY = "(&amp;(objectCategory=user)(sAMAccountName=*)(objectClass=person)(sn=_LASTNAME_*))" 
PARTIAL_NAME_QUERY = "(&amp;(objectCategory=user)(objectClass=person)(sAMAccountName=*)(sn=_LASTNAME_*)(givenname=_FIRSTNAME_*))" 
PARTIAL_FIRST_QUERY = "(&amp;(objectCategory=user)(objectClass=person)(sAMAccountName=*)(givenname=_FIRSTNAME_*))" 



class DirectoryEntryHelper(object):
  """ Helper class for wrapping
  and simplifying AD search results.
  As much fun as dealing with Microsoft style
  object hierarchies can be...
  TODO: Is there an easy way to inject things into __dict__ for dir &amp; ipy tab completion?
  """

  def __init__(self, entry):
      if isinstance(entry, SearchResult):
          self.__dict__['_entry'] = entry.GetDirectoryEntry()
      elif isinstance(entry, DirectoryEntry):
          self.__dict__['_entry'] = entry
      else:
          raise TypeError, "Invalid type '%s', don't know how to proxy it" % type(entry)

  @property
  def properties(self):
      return self._entry.Properties

  @property
  def keys(self):
      """Returns a comprehended list
      of the *defined* Keys on the Properties object.
      Any valid keys w/o values won't get listed.
      """
      return [key for key in self.properties.Keys]

  def __getattr__(self, key):
      """Proxies the Properties objects in the DirectoryEntry
      to provide a simple getter.
      First checks if DirectoryEntry has an attribute (property)
      matching requested key, and passes that if so.
      Otherwise, fetches a matching property.  If you have
      a name collison, use the .properties property on this object
      to get a clean copy of the Properties object
      TODO: Better support for multi-value keys (only returns first idx right now]
      """
      if hasattr(self._entry, key):
          return getattr(self._entry, key)
      elif self._entry.Properties[key].Count:
          return self._entry.Properties[key][0]
      else:
          return None

  def __setattr__(self, key, value):
      """Proxies the Properties objects in the DirectoryEntry 
      to provide a simpler setter.
      First checks if DirectoryEntry has an attribute (property)
      matching requested key, and sets on that if so.
      Otherwise, sets upon a matching property.  
      If the property has an existing count > 0, sets Index 0  
      If not, it does an Add().
      The way the MS Classes work, Properties.Keys only returns keys that
      have a value count.  However, any *valid* Key has a silent value which
      can be initialized via Add().
      Invalid keys throw an error, so you can't be arbitrary.
      I use a list of keyspace found at: 

          http://www.dotnetactivedirectory.com/\
              Understanding_LDAP_Active_Directory_User_Object_Properties.html

      If you have a name collison, use the .properties property on this object
      to get a clean copy of the Properties object
      TODO: Better support for multi-value keys (only sets first idx right now]
      """
      if hasattr(self._entry, key):
          setattr(self._entry, key, value)
      else:
          if self._entry.Properties[key].Count > 0:
              if self._entry.Properties[key].Count > 1:
                  print >> sys.stderr, "[warning] Key '%s' contains multiple values which we don't properly support yet.  Using Index 0"
              self._entry.Properties[key][0] = value
          else:
              self._entry.Properties[key].Add(value)
          # commit early, commit often...
          self._entry.CommitChanges()

  def __repr__(self):
      """Print all pretty like on console.
      My biggest pet peeve working in .Net is that most objects don't have
      any kind of __str__/__repr__ value on them to give you quick information
      on their contents.  ARGH!
      """
      return "<DirectoryEntryHelper:{sn=%s,givenName=%s,mail=%s,sAMAccountName=%s}>" %\
          (self.sn, self.givenName, self.mail, self.sAMAccountName)

class ActiveDirectory(object):
  """ ActiveDirectory interface class.
  Tested on IronPython 2.6RC + Mono 2.4.2.3 on Linux. YMMV.
  """

  # Active Directory Handle
  adh = None

  server = None
  domain = None
  baseDN = None

  def __init__(self, server, domain, username, password, baseDN=""):
      """Constructor to instantiate the LDAP connection.
      server is a resolvable address to your domain controller (e.g. the resolvable hostname or IP)
      domain should be the ActiveDirectory domain as specified by your admin.
      If you don't pass BaseDN, your 'domain' argument will be split on
      . and each piece will be passed as DC=<piece>. This will be used as your BaseDN
      E.G. domain aurigasv.local becomes "DC=aurigasv,DC=local"

      *** Any exception in connecting will be wrapped and rethrown, aborting construction.
      """
      try:
          # Setup the baseDN
          self.server = server
          self.domain = domain
          if baseDN:
              self.baseDN = baseDN
          else:
              for tier in domain.split('.'):
                  baseDN += "DC=%s," % tier
              self.baseDN = baseDN[:-1]
              #print "Established a parsed BaseDN of '%s'" % self.baseDN

          # Establish a directory object, with auth info, pointing at the authenticating user
          self.adh = DirectoryEntry("LDAP://%s/%s" % (server, self.baseDN))
          self.adh.AuthenticationType = AuthenticationTypes.Secure
          self.adh.Username = username
          self.adh.Password = password

          # connect and fetch a copy of the authenticating user
          filter = "sAMAccountName=%s" % username
          dsSystem = DirectorySearcher(self.adh, filter)
          dsSystem.SearchScope = SearchScope.Subtree
          result = dsSystem.FindOne()

          if not result:
              raise Exception, "Found no result searching for specified authentication user %s." % username

      except Exception, e:
          raise e, "Authentication failure.  Did you provide valid connection information / authentication credentials?"

  def get_helper(self, result):
      if hasattr(result, "__iter__"):
          return (DirectoryEntryHelper(item) for item in result)
      else:
          return DirectoryEntryHelper(result)


class ActiveDirectorySearcher(ActiveDirectory):

  def _search(self, filter, multi=True):
      """Semi-private method for searching.
      Note that "filter" needs to be a string which is a 
      *VALID* LDAP query/filter.  You should probably use one of the useful
      utility find methods instead of querying this directly
      unless you understand LDAP queries
      By default searches for one only, pass multi=True for multiple results.
      """
      print "[debug] Searching activedirectory for %s.  Allow multiple results? %s" % (filter, multi)
      searcher = DirectorySearcher(self.adh)

      searcher.ReferralChasing = ReferralChasingOption.All
      searcher.SearchScope = SearchScope.Subtree
      searcher.Filter = filter

      results = searcher.FindAll()

      if results.Count == 0:
          raise Exception, "No results found for search."

      if multi:
          # Generator for performance
          return (entry for entry in results)
      else:
          if results.Count > 1:
              raise Exception, "Found multiple results in single-search mode.  To support multiples, please pass multi=True to the find method."
          return results[0]  


  def find_last_name(self, last_name, multi=False, nowrap=False):
      """Searches by sn (lastname).
      This can be a partial query, as long as the first
      few letters of last_name are defined 
      By default searches for one only, pass multi=True for multiple results.
      If you want  raw .Net SearchResult(s) object(s), pass nowrap=True, otherwise
      you get it back wrapped in a helper class.
      """
      result = self._search(PARTIAL_LAST_QUERY.replace('_LASTNAME_', last_name), multi)
      if not nowrap:
          result = self.get_helper(result)

      return result

  def find_name(self, last_name=None, first_name=None, multi=False, nowrap=False):
      """Searches by sn (lastname) &amp; givenName( firstname) 
      This can be a partial query, as long as the first
      few letters of last_name are defined
      If firstname is None, it quietly passes into find_last_name.
      By default searches for one only, pass multi=True for multiple results.
      If you want  raw .Net SearchResult(s) object(s), pass nowrap=True, otherwise
      you get it back wrapped in a helper class.
      """
      if not last_name:
          raise Exception, "You must, at a minimum, specify last_name for a name search."

      if not first_name:
          result = self.find_last_name(last_name, multi)
      else:
          result = self._search(PARTIAL_NAME_QUERY.replace('_FIRSTNAME_', first_name).replace('_LASTNAME_', last_name), multi)
          if not nowrap:
              result = self.get_helper(result)

      return result

  def find_account(self, account_name, multi=False, nowrap=False):
      """Searches by  sAMAccountName (account_name), which should be unique
      This can be a partial query, as long as the first
      few letters of account_name are defined
      If firstname is None, it quietly passes into find_last_name.
      By default searches for one only, pass multi=True for multiple results.
      If you want  raw .Net SearchResult(s) object(s), pass nowrap=True, otherwise
      you get it back wrapped in a helper class.
      """
      result = self._search(ACCOUNT_QUERY.replace('_ACCOUNTNAME_', account_name), multi)
      if not nowrap:
          result = self.get_helper(result)

      return result

  def find_first_name(self, first_name, multi=False, nowrap=False):
      """Searches by givenName, which should be unique
      This can be a partial query, as long as the first
      few letters of account_name are defined
      If firstname is None, it quietly passes into find_last_name.
      By default searches for one only, pass multi=True for multiple results.
      If you want  raw .Net SearchResult(s) object(s), pass nowrap=True, otherwise
      you get it back wrapped in a helper class.
      """
      result = self._search(PARTIAL_FIRST_QUERY.replace('_FIRSTNAME_', first_name), multi)
      if not nowrap:
          result = self.get_helper(result)

      return result

Double Clicking in IronPython + Silverlight

2009-09-02T00:00:00-07:00

Silverlight rather oddly lacks a double click event. You can detect single click, but for double clicking you're on your own. I found some examples for C#, but none for IronPython and had, a few months ago, ported some code I found here for IronPython.

I've been using it in application development and user testing for about 6 months and it works fairly well in both Silverlight 2 and 3, although YMMV.

 
CLICK_INTERVAL = 500
 
class ClickHandler(object):
    """Attach a Double Click handler to any given UIElement,
    as by default Silverlight lacks support for double clicking.
    """
    _lastClickTime = 0
    _callback = None
 
    @staticmethod
    def attachDoubleClickHandler(target, callback):
        return ClickHandler(target, callback)
 
    def __init__(self, target, callback):
        target.MouseLeftButtonUp += self.target_MouseLeftButtonUp
        target.MouseLeftButtonDown += self.target_MouseLeftButtonDown
        self._callback = callback
 
    def target_MouseLeftButtonUp(self, sender, args):
        self._lastClickTime = DateTime.Now.Ticks / 10000
 
    def target_MouseLeftButtonDown(self, sender, args):
        now = DateTime.Now.Ticks / 10000
        if now - self._lastClickTime < CLICK_INTERVAL and \
          now - self._lastClickTime > 25:
            self._callback(sender, args)

You may want to tweak the click interval based on what works for you. 500 milliseconds worked well with my users, for the application in question. To attach a ClickHandler simply invoke the static method ClickHandler.attachDoubleClickHandler(«uiElementObj», «callbackFunc»)

from System import Uri, UriKind
from System.Windows import Application, UIElement
from System.Windows.Browser import HtmlPage
 
from util import ClickHandler
 
 
class Foo(UIElement):
  """Sample UIElement class demonstrating usage of 
  ClickHandler and it's events.
  """
 
  def __init__(self):
    Application.LoadComponent(self, Uri('xaml/widgets/foo.xaml',
            UriKind.Relative))
    ClickHandler.attachDoubleClickHandler(self, self.evt_dblClicked)
 
  def evt_dblClicked(self, sender, args):
    """Event for double click; this gets passed
    the original "MouseLeftButtonDown" arguments 
    which ClickHandler got, so you'll want to match
    the "MouseButtonEventHandler" delegate arguments.
    Object sender - the object the mouse click handler is attached to,
            in this case, should be the same as 'self'
    System.Windows.Input.MouseButtonEventArgs args - the mouse event arguments
    """
    HtmlPage.Window.Alert('%s got double clicked!' % sender)

It's not the world's greatest solution, but it gets the job done.