Format reports on Golang CLI

[twalluxio]

For now, Java is in charge of formatting the output of reports, e.g. file size in bytes to readable values; date formats.

This PR moves the format job to the golang CLI side. When checking reports, data transferred to golang CLI, and CLI deal with the formatting.

Current date format: 2006-01-02T15:04:05Z07:00
Current duration format: 0d 12h34m56s
Current file size format: 1024.00MB (scale up when number > 5120)

Format on dates:

• summary object -> started object
• job_service object -> masterStatus array -> startTime object
• job_service object -> recentModifiedJobs array -> timestamp object
• job_service object -> recentFailedJobs array -> timestamp object
• job_service object -> longestRunningJobs array -> timestamp object

Format on duration:

• summary object -> uptime object

Format on file size:

• summary object -> freeCapacity object
• summary object -> totalCapacityOnTiers map
• summary object -> usedCapacityOnTiers map
• ufs map -> ufsCapacityBytes object
• ufs map -> ufsUsedBytes object

[Xenorith]

this approach is very brute force, but it is hard to determine which fields need to be formatted. i have 2 ideas:

1. represent the output as a struct. set annotations on struct fields to determine if any conversion function should be applied when serializing

• pros: define exactly which fields need to be converted, no chance for mistakenly converting the wrong field
• cons: need to define a struct for the output and the struct needs to be updated if the output is updated

2. define a series of patterns/rules to determine if a field needs to be converted (ex. if field name ends with Bytes, cast value to float64 and run convertBytesToString

• pros: no need to do anything if fields are changed
• cons: need to be careful with field names. also it can be hard to navigate the output if the output is not a flat map (ex. some fields have a map as a value instead of a string or number)

example for 1:

type SummaryOutput struct {
  FreeCapacity float64 `json:"freeCapacity" yaml:"freeCapacity" format:"convertBytesToString"`
  Started float64 `json:"started" yaml:"started" format:"convertMsToDatetime"`
  ...
}

func (o SummaryOutput) MarshalJSON() ([]byte, error) {
  val := reflect.Value(o)
  // the below can be abstracted to a function, reusable for each struct
  obj := orderedmap.New()
  for i := 0; i < val.NumField(); i++ {
    f := val.Field(i)
    convertName, ok := f.Tag.Lookup("format")
    if !ok {
      obj.Set(<fieldName>, <fieldValue>)
      continue
    }
    convertFunc, ok := convertFuncMap[convertName] // convertFuncMap is a mapping of name (ex. "convertBytesToString") to the function definition
    // check ok
    obj.Set(<fieldName>, convertFunc(<fieldValue>)
  } 
  return json.Marshal(obj)
}

var summary SummaryOutput
json.Unmarshal(buf, &summary)
json.Marshal(summary)

example for 2:
use the argument suffix as the way to determine if the value should be formatted.

• if the key ends with Bytes -> cast value to float64 and run convertBytesToString
• if the key ends with Date -> cast value to float64 and run convertMsToDatetime
• if the key ends with Duration -> cast the value to float64 and run convertMsToDuration
• if the key ends with Tiers -> assume the value is a map of key -> bytes

this will require some keys to be renamed to fit the above mapping

• freeCapacity -> freeCapacityBytes
• started -> startDate
• uptime -> uptimeDuration

// after populating ordered map obj
out := orderedMap.New()
for k, v := obj {
  if strings.HasSuffix(k, "Bytes") {
    out.Set(k, convertBytesToString(v.(float64))
  else if ...
}
json.Marshal(out)

[twalluxio]

I prefer example 2, it's right time to review all summary property formats and keep them in similar formats.
But there is one more thing, need to think about a way to handle nested JSON conversion.
There are three scenarios:

1. The key-value pair to convert is in an object node. e.g. "report": {"keyBytes": valueBytes}
2. The key-value pair to convert is in an array node. e.g. "reports": [{"keyBytes": valueBytes}, {...}]
3. The key-value pair to convert is in a map. e.g. "report": {"keyBytes": {"keyMap": valueBytes}, {...}}

I might need to think about all scenarios.

[Kai-Zhang]

@jiacheliu3 can the tiers information be safely removed?

[Kai-Zhang]

@jiacheliu3 also can you help to confirm if this report is still needed?

[Xenorith]

alluxio-bot, merge this please