Config evaluated values

[bholt]

The whole "hadoop.tmp.dir" situation has exposed something I didn't realize about Config and Hadoop's JobConf. Apparently when you call JobConf.get, it evaluates any variables in it. But the way we extract key/values from JobConf, we don't get this behavior:

scalding> val jc = new org.apache.hadoop.mapred.JobConf
scalding> jc.get("hadoop.tmp.dir")
res0: java.lang.String = /tmp/hadoop-bholt

scalding> val c = Config.fromHadoop(jc)
scalding> c.get("hadoop.tmp.dir")
res1: Option[String] = Some(/tmp/hadoop-${user.name})

It seems to work out alright in some situations, since Hadoop/Cascading must be evaluating it to get the actual path later. But this causes validateSources to fail when "strict" is set on the mode.

Seems like it would be preferable to force Hadoop to evaluate each of these values in Config.fromHadoop(). @johnynek (or anyone else): do you forsee any problems with this?