Skip to content

Commit

Permalink
Rewrite in Python
Browse files Browse the repository at this point in the history
Python's libraries for TOML and YAML solve the problem of
Remarshal not being able to process TOML newer than version 0.2 (#2)
and at least some of the issues with the generated YAML (#1, #3).
Floating point precision (#1) in all formats requires further
investigation, though it seems that what PyYAML and pytoml do
matches the respective specs.

v0.4.0
  • Loading branch information
dbohdan committed Sep 2, 2016
1 parent 37e7cbf commit 7d0a5d8
Show file tree
Hide file tree
Showing 18 changed files with 412 additions and 460 deletions.
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
remarshal
/build
/dist
/venv*
*.egg-info
*.pyc
__pycache__
26 changes: 9 additions & 17 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,18 +1,10 @@
language: go
language: python
sudo: false
go:
- 1.2
- 1.3
- 1.4
- 1.5
- 1.6
- 1.7
- release
install:
- go get github.com/BurntSushi/toml
- go get gopkg.in/yaml.v2
before_script:
- chmod +x tests.sh
script:
- go build remarshal.go
- ./tests.sh
python:
- 2.7
- 3.2
- 3.3
- 3.4
- 3.5
install: python setup.py install
script: python setup.py test
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Copyright (c) 2014 Danyil Bohdan
Copyright (c) 2014, 2015, 2016 dbohdan

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
129 changes: 57 additions & 72 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# remarshal
# Remarshal

[![Build Status](https://travis-ci.org/dbohdan/remarshal.svg?branch=master)](https://travis-ci.org/dbohdan/remarshal)

Expand All @@ -10,97 +10,83 @@ commands `toml2yaml`, `toml2json`, `yaml2toml`, `yaml2json`. `json2toml` and
# Usage

```
remarshal -if inputformat -of outputformat [-indent-json=(true|false)]
[-i inputfile] [-o outputfile] [-wrap wrapper] [-unwrap wrapper]
usage: remarshal.py [-h] [-i INPUT] [-o OUTPUT] -if {json,toml,yaml} -of
{json,toml,yaml} [--indent-json] [--wrap WRAP]
[--unwrap UNWRAP]
[inputfile]
```

where `inputformat` and `outputformat` can each be `toml`, `yaml` or
`json`.
```
usage: {json,toml,yaml}2{toml,yaml} [-h] [-i INPUT] [-o OUTPUT] [--wrap WRAP]
[--unwrap UNWRAP] [inputfile]
```

```
toml2toml [-wrap wrapper] [-unwrap wrapper] [-o outputfile] [[-i] inputfile]
yaml2toml [-wrap wrapper] [-unwrap wrapper] [-o outputfile] [[-i] inputfile]
json2toml [-wrap wrapper] [-unwrap wrapper] [-o outputfile] [[-i] inputfile]
toml2yaml [-wrap wrapper] [-unwrap wrapper] [-o outputfile] [[-i] inputfile]
yaml2yaml [-wrap wrapper] [-unwrap wrapper] [-o outputfile] [[-i] inputfile]
json2yaml [-wrap wrapper] [-unwrap wrapper] [-o outputfile] [[-i] inputfile]
toml2json [-indent-json=(true|false)] [-wrap wrapper] [-unwrap wrapper]
[-o outputfile] [[-i] inputfile]
yaml2json [-indent-json=(true|false)] [-wrap wrapper] [-unwrap wrapper]
[-o outputfile] [[-i] inputfile]
json2json [-indent-json=(true|false)] [-wrap wrapper] [-unwrap wrapper]
[-o outputfile] [[-i] inputfile]
usage: {json,toml,yaml}2json [-h] [-i INPUT] [-o OUTPUT] [--indent-json]
[-wrap WRAP] [-unwrap UNWRAP] [inputfile]
```

All of the commands above exit with status 0 on success and 1 on failure.

If no `inputfile` is given or it is `-` or a blank string the data to convert is
read from standard input. If no `outputfile` is given or it is `-` or a blank
string the result of the conversion is written to standard output.
If no `inputfile` or `-i INPUT` is given or it is `-` or a blank string the data
to convert is read from standard input. If no `-o OUTPUT` is given or it is `-`
or a blank string the result of the conversion is written to standard output.

For the short commands (`x2y`) the flag `-i` before `inputfile` can be omitted
if `inputfile` is the last argument.

## Wrappers

The flags `-wrap` and `-unwrap` are there to solve the problem of converting
The flags `--wrap` and `--unwrap` are there to solve the problem of converting
JSON and YAML data to TOML if the topmost element of that data is not of a map
type (i.e., not an object in JSON or an associative array in YAML) but a list, a
string or a number. Such data can not be represented as TOML directly; it needs
to wrapped in a map type first. Passing the flag `-wrap someKey` to `remarshal`
to wrapped in a map type first. Passing the flag `--wrap someKey` to `remarshal`
or one of its short commands wraps the input data in a "wrapper" map with one
key, "someKey", with the input data as its value. The flag `-unwrap someKey`
key, "someKey", with the input data as its value. The flag `--unwrap someKey`
does the opposite: if it is specified only the value stored under the key
"someKey" in the top-level map element of the input data is converted to the
target format and output; all other data is skipped. If the top-level element is
not a map or does not have the key `someKey` then `-unwrap someKey` returns an
not a map or does not have the key `someKey` then `--unwrap someKey` returns an
error.

The following shell transcript demonstrates the problem and how `-wrap` and
`-unwrap` solve it:
The following shell transcript demonstrates the problem and how `--wrap` and
`--unwrap` solve it:

```
$ echo '[{"a":"b"},{"c":[1,2,3]}]' | ./remarshal -if json -of toml
cannot convert data: top-level values must be a Go map or struct
$ echo '[{"a":"b"},{"c":[1,2,3]}]' | ./remarshal.py -if json -of toml
Error: cannot convert non-dictionary data to TOML; use "wrap" to wrap it in a dictionary
$ echo '[{"a":"b"},{"c":[1,2,3]}]' | \
./remarshal -if json -of toml -wrap main
./remarshal.py -if json -of toml --wrap main
[[main]]
a = "b"
a = "b"
[[main]]
c = [1, 2, 3]
c = [1, 2, 3]
$ echo '[{"a":"b"},{"c":[1,2,3]}]' | \
./remarshal -if json -of toml -wrap main > test.toml
./remarshal.py -if json -of toml --wrap main > test.toml
$ ./remarshal -if toml -of json -indent-json=0 < test.toml
$ ./remarshal.py -if toml -of json < test.toml
{"main":[{"a":"b"},{"c":[1,2,3]}]}
$ ./remarshal -if toml -of json -indent-json=0 -unwrap main < test.toml
$ ./remarshal.py -if toml -of json --unwrap main < test.toml
[{"a":"b"},{"c":[1,2,3]}]
```

# Building and installation
# Installation

Tested with `go version go1.2.2 linux/amd64`. Do the following to install
`remarshal`:
You will need Python 2.7 or Python 3.x.

```sh
go get github.com/BurntSushi/toml
go get gopkg.in/yaml.v2
git clone https://github.com/dbohdan/remarshal.git
cd remarshal
go build remarshal.go
sh tests.sh
sudo sh install.sh # install into /usr/local/bin
sudo python setup.py install
```

# Examples

```
$ ./remarshal -i example.toml -if toml -of yaml
$ ./remarshal.py -i example.toml -if toml -of yaml
clients:
data:
- - gamma
Expand All @@ -119,10 +105,10 @@ database:
- 8002
server: 192.168.1.1
owner:
bio: |-
GitHub Cofounder & CEO
Likes tater tots and beer.
dob: 1979-05-27T07:32:00Z
bio: 'GitHub Cofounder & CEO
Likes tater tots and beer.'
dob: 1979-05-27 07:32:00+00:00
name: Tom Preston-Werner
organization: GitHub
products:
Expand All @@ -142,51 +128,50 @@ servers:
title: TOML Example
$ curl -s http://api.openweathermap.org/data/2.5/weather\?q\=Kiev,ua | \
./remarshal -if json -of toml
./remarshal.py -if json -of toml
base = "cmc stations"
cod = 200
dt = 1412532000
id = 703448
name = "Kiev"
[clouds]
all = 44
all = 44
[coord]
lat = 50.43
lon = 30.52
lat = 50.42999999999999972
lon = 30.51999999999999957
[main]
humidity = 66
pressure = 1026
temp = 283.49
temp_max = 284.15
temp_min = 283.15
humidity = 66
pressure = 1026
temp = 283.49000000000000909
temp_max = 284.14999999999997726
temp_min = 283.14999999999997726
[sys]
country = "UA"
id = 7358
message = 0.2437
sunrise = 1412481902
sunset = 1412522846
type = 1
country = "UA"
id = 7358
message = 0.24370000000000000
sunrise = 1412481902
sunset = 1412522846
type = 1
[[weather]]
description = "scattered clouds"
icon = "03n"
id = 802
main = "Clouds"
description = "scattered clouds"
icon = "03n"
id = 802
main = "Clouds"
[wind]
deg = 80
speed = 2
deg = 80
speed = 2
```

# Known bugs and limitations

* Converting data with floating point values to YAML may cause a loss of
precision.
* `remarshal` only supports TOML [v0.2.0](https://github.com/toml-lang/toml/tree/v0.2.0).

# License

Expand Down
1 change: 0 additions & 1 deletion VERSION

This file was deleted.

1 change: 1 addition & 0 deletions array.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[{"a":"b"},{"c":[1,2,3]}]
6 changes: 6 additions & 0 deletions array.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@

[[data]]
a = "b"

[[data]]
c = [1, 2, 3]
6 changes: 3 additions & 3 deletions example.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
"server": "192.168.1.1"
},
"owner": {
"bio": "GitHub Cofounder \u0026 CEO\nLikes tater tots and beer.",
"dob": "1979-05-27T07:32:00Z",
"bio": "GitHub Cofounder & CEO\nLikes tater tots and beer.",
"dob": "1979-05-27T07:32:00+00:00",
"name": "Tom Preston-Werner",
"organization": "GitHub"
},
Expand All @@ -54,4 +54,4 @@
}
},
"title": "TOML Example"
}
}
8 changes: 4 additions & 4 deletions example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ database:
- 8002
server: 192.168.1.1
owner:
bio: |-
GitHub Cofounder & CEO
Likes tater tots and beer.
dob: 1979-05-27T07:32:00Z
bio: 'GitHub Cofounder & CEO
Likes tater tots and beer.'
dob: 1979-05-27 07:32:00+00:00
name: Tom Preston-Werner
organization: GitHub
products:
Expand Down
8 changes: 0 additions & 8 deletions install.sh

This file was deleted.

Loading

0 comments on commit 7d0a5d8

Please sign in to comment.