Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568

Merged
merged 10 commits into from
Apr 21, 2022
+204 −58
Merged

fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ #8568

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
4448de5
fix(cmds): use correct ipld multicodecs in cmds
schomatis Nov 25, 2021
e348b59
refactor: block put --cid-codec
lidel Apr 13, 2022
70c13cf
chore: go-multicodec v0.4.1 + helptext fixes
lidel Apr 13, 2022
a33ede2
fix: add libp2p-key to supported codecs
lidel Apr 13, 2022
2119931
fix(test): t0110-gateway.sh
lidel Apr 13, 2022
44cc485
test: interface-go-ipfs-core CIDv1 raw and dag-pb
lidel Apr 13, 2022
02aff74
fix(test): t0290-cid.sh
lidel Apr 13, 2022
c34fe9c
test: use of both 'format' and 'cid-codec'
lidel Apr 16, 2022
2f5b8a4
docs(changelog): document breaking changes
lidel Apr 21, 2022
1f4902b
chore: interface-go-ipfs-core v0.7.0
lidel Apr 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 37 additions & 25 deletions core/commands/block.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ var BlockCmd = &cmds.Command{
Tagline: "Interact with raw IPFS blocks.",
ShortDescription: `
'ipfs block' is a plumbing command used to manipulate raw IPFS blocks.
Reads from stdin or writes to stdout, and <key> is a base58 encoded
multihash.
Reads from stdin or writes to stdout. A block is identified by a Multihash
passed with a valid CID.
`,
},

Expand All @@ -51,14 +51,14 @@ var blockStatCmd = &cmds.Command{
'ipfs block stat' is a plumbing command for retrieving information
on raw IPFS blocks. It outputs the following to stdout:

Key - the base58 encoded multihash
Key - the CID of the block
Size - the size of the block in bytes

`,
},

Arguments: []cmds.Argument{
cmds.StringArg("key", true, false, "The base58 multihash of an existing block to stat.").EnableStdin(),
cmds.StringArg("cid", true, false, "The CID of an existing block to stat.").EnableStdin(),
},
Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
api, err := cmdenv.GetApi(env, req)
Expand Down Expand Up @@ -90,12 +90,12 @@ var blockGetCmd = &cmds.Command{
Tagline: "Get a raw IPFS block.",
ShortDescription: `
'ipfs block get' is a plumbing command for retrieving raw IPFS blocks.
It outputs to stdout, and <key> is a base58 encoded multihash.
It takes a <cid>, and outputs the block to stdout.
`,
},

Arguments: []cmds.Argument{
cmds.StringArg("key", true, false, "The base58 multihash of an existing block to get.").EnableStdin(),
cmds.StringArg("cid", true, false, "The CID of an existing block to get.").EnableStdin(),
},
Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
api, err := cmdenv.GetApi(env, req)
Expand All @@ -113,32 +113,41 @@ It outputs to stdout, and <key> is a base58 encoded multihash.
}

const (
blockFormatOptionName = "format"
mhtypeOptionName = "mhtype"
mhlenOptionName = "mhlen"
blockFormatOptionName = "format"
blockCidCodecOptionName = "cid-codec"
mhtypeOptionName = "mhtype"
mhlenOptionName = "mhlen"
)

var blockPutCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "Store input as an IPFS block.",
ShortDescription: `
'ipfs block put' is a plumbing command for storing raw IPFS blocks.
It reads from stdin, and outputs the block's CID to stdout.
It reads data from stdin, and outputs the block's CID to stdout.

Unless specified, this command returns dag-pb CIDv0 CIDs. Setting 'mhtype' to anything
other than 'sha2-256' or format to anything other than 'v0' will result in CIDv1.
Unless cid-codec is specified, this command returns raw (0x55) CIDv1 CIDs.

Passing alternative --cid-codec does not modify imported data, nor run any
validation. It is provided solely for convenience for users who create blocks
in userland.

NOTE:
Do not use --format for any new code. It got superseded by --cid-codec and left
only for backward compatibility when a legacy CIDv0 is required (--format=v0).
`,
},

Arguments: []cmds.Argument{
cmds.FileArg("data", true, true, "The data to be stored as an IPFS block.").EnableStdin(),
},
Options: []cmds.Option{
cmds.StringOption(blockFormatOptionName, "f", "cid format for blocks to be created with."),
cmds.StringOption(mhtypeOptionName, "multihash hash function").WithDefault("sha2-256"),
cmds.IntOption(mhlenOptionName, "multihash hash length").WithDefault(-1),
cmds.BoolOption(pinOptionName, "pin added blocks recursively").WithDefault(false),
cmds.StringOption(blockCidCodecOptionName, "Multicodec to use in returned CID").WithDefault("raw"),
Copy link
Member

@lidel lidel Apr 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aschmahmann I did some tests around this and the good news is that I was wrong, there is a way to have unified handling of default opts that works the same in ipfs block put CLI and raw HTTP POST to /api/v0/block/put.

If we keep the .WithDefault("raw") here, then this implicit default is also applied when people use curl against /api/v0/block/put and forget to pass cid-codec. It will also produce good docs at https://docs.ipfs.io/reference/http/api/#api-v0-block-put.

If we want to be extra strict and return error, I'd have to remove WithDefault and add some glue code that detects cidCodec == "" and throws error, but that feels hacky and annoying because it would make cid-codec a mandatory parameter, making it bad UX for the common use case where people are fine with codec being raw.

My vote is to keep WithDefault("raw").

Copy link
Contributor

@aschmahmann aschmahmann Apr 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, the only breaking change is:

ipfs block put now produces CIDv1 with raw codec by default now

Which only effects the output CID, but ipfs block put <some-dagpb-stuff>; ipfs block get QmDagPBv0Thing will still work fine. This seems like a not too bad silent breakage, although I'm generally averse to silent breakages. WDYT?

Copy link
Member

@lidel lidel Apr 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm. Added 0.13 breaking changes section to CHANGELOG.md in 2f5b8a4

cmds.StringOption(mhtypeOptionName, "Multihash hash function").WithDefault("sha2-256"),
cmds.IntOption(mhlenOptionName, "Multihash hash length").WithDefault(-1),
cmds.BoolOption(pinOptionName, "Pin added blocks recursively").WithDefault(false),
cmdutils.AllowBigBlockOption,
cmds.StringOption(blockFormatOptionName, "f", "Use legacy format for returned CID (DEPRECATED)"),
},
Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
api, err := cmdenv.GetApi(env, req)
Expand All @@ -157,13 +166,15 @@ other than 'sha2-256' or format to anything other than 'v0' will result in CIDv1
return errors.New("missing option \"mhlen\"")
}

format, formatSet := req.Options[blockFormatOptionName].(string)
if !formatSet {
if mhtval != mh.SHA2_256 || (mhlen != -1 && mhlen != 32) {
format = "protobuf"
} else {
format = "v0"
cidCodec, _ := req.Options[blockCidCodecOptionName].(string)
format, _ := req.Options[blockFormatOptionName].(string) // deprecated

// use of legacy 'format' needs to supress 'cid-codec'
if format != "" {
if cidCodec != "" && cidCodec != "raw" {
return fmt.Errorf("unable to use %q (deprecated) and a custom %q at the same time", blockFormatOptionName, blockCidCodecOptionName)
}
cidCodec = "" // makes it no-op
}

pin, _ := req.Options[pinOptionName].(bool)
Expand All @@ -177,6 +188,7 @@ other than 'sha2-256' or format to anything other than 'v0' will result in CIDv1

p, err := api.Block().Put(req.Context, file,
options.Block.Hash(mhtval, mhlen),
options.Block.CidCodec(cidCodec),
options.Block.Format(format),
options.Block.Pin(pin))
if err != nil {
Expand Down Expand Up @@ -219,14 +231,14 @@ type removedBlock struct {

var blockRmCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "Remove IPFS block(s).",
Tagline: "Remove IPFS block(s) from the local datastore.",
ShortDescription: `
'ipfs block rm' is a plumbing command for removing raw ipfs blocks.
It takes a list of base58 encoded multihashes to remove.
It takes a list of CIDs to remove from the local datastore..
`,
},
Arguments: []cmds.Argument{
cmds.StringArg("hash", true, true, "Bash58 encoded multihash of block(s) to remove."),
cmds.StringArg("cid", true, true, "CIDs of block(s) to remove."),
},
Options: []cmds.Option{
cmds.BoolOption(forceOptionName, "f", "Ignore nonexistent blocks."),
Expand Down
60 changes: 48 additions & 12 deletions core/commands/cid.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ import (
cidutil "github.com/ipfs/go-cidutil"
cmds "github.com/ipfs/go-ipfs-cmds"
verifcid "github.com/ipfs/go-verifcid"
"github.com/ipld/go-ipld-prime/multicodec"
mbase "github.com/multiformats/go-multibase"
mc "github.com/multiformats/go-multicodec"
mhash "github.com/multiformats/go-multihash"
)

Expand Down Expand Up @@ -46,7 +48,7 @@ The optional format string is a printf style format string:
` + cidutil.FormatRef,
},
Arguments: []cmds.Argument{
cmds.StringArg("cid", true, true, "Cids to format.").EnableStdin(),
cmds.StringArg("cid", true, true, "CIDs to format.").EnableStdin(),
},
Options: []cmds.Option{
cmds.StringOption(cidFormatOptionName, "Printf style format string.").WithDefault("%s"),
Expand All @@ -63,14 +65,14 @@ The optional format string is a printf style format string:
opts := cidFormatOpts{}

if strings.IndexByte(fmtStr, '%') == -1 {
return fmt.Errorf("invalid format string: %s", fmtStr)
return fmt.Errorf("invalid format string: %q", fmtStr)
}
opts.fmtStr = fmtStr

if codecStr != "" {
codec, ok := cid.Codecs[codecStr]
if !ok {
return fmt.Errorf("unknown IPLD codec: %s", codecStr)
return fmt.Errorf("unknown IPLD codec: %q", codecStr)
}
opts.newCodec = codec
} // otherwise, leave it as 0 (not a valid IPLD codec)
Expand All @@ -80,13 +82,13 @@ The optional format string is a printf style format string:
// noop
case "0":
if opts.newCodec != 0 && opts.newCodec != cid.DagProtobuf {
return fmt.Errorf("cannot convert to CIDv0 with any codec other than DagPB")
return fmt.Errorf("cannot convert to CIDv0 with any codec other than dag-pb")
}
opts.verConv = toCidV0
case "1":
opts.verConv = toCidV1
default:
return fmt.Errorf("invalid cid version: %s", verStr)
return fmt.Errorf("invalid cid version: %q", verStr)
}

if baseStr != "" {
Expand Down Expand Up @@ -123,9 +125,13 @@ type CidFormatRes struct {
var base32Cmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "Convert CIDs to Base32 CID version 1.",
ShortDescription: `
'ipfs cid base32' normalizes passes CIDs to their canonical case-insensitive encoding.
Useful when processing third-party CIDs which could come with arbitrary formats.
`,
},
Arguments: []cmds.Argument{
cmds.StringArg("cid", true, true, "Cids to convert.").EnableStdin(),
cmds.StringArg("cid", true, true, "CIDs to convert.").EnableStdin(),
},
Run: func(req *cmds.Request, resp cmds.ResponseEmitter, env cmds.Environment) error {
opts := cidFormatOpts{
Expand Down Expand Up @@ -232,7 +238,7 @@ func emitCids(req *cmds.Request, resp cmds.ResponseEmitter, opts cidFormatOpts)

func toCidV0(c cid.Cid) (cid.Cid, error) {
if c.Type() != cid.DagProtobuf {
return cid.Cid{}, fmt.Errorf("can't convert non-protobuf nodes to cidv0")
return cid.Cid{}, fmt.Errorf("can't convert non-dag-pb nodes to cidv0")
}
return cid.NewCidV0(c.Hash()), nil
}
Expand All @@ -254,6 +260,9 @@ const (
var basesCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "List available multibase encodings.",
ShortDescription: `
'ipfs cid bases' relies on https://github.com/multiformats/go-multibase
`,
},
Options: []cmds.Option{
cmds.BoolOption(prefixOptionName, "also include the single letter prefixes in addition to the code"),
Expand Down Expand Up @@ -296,21 +305,45 @@ var basesCmd = &cmds.Command{
}

const (
codecsNumericOptionName = "numeric"
codecsNumericOptionName = "numeric"
codecsSupportedOptionName = "supported"
)

var codecsCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "List available CID codecs.",
ShortDescription: `
'ipfs cid codecs' relies on https://github.com/multiformats/go-multicodec
`,
},
Options: []cmds.Option{
cmds.BoolOption(codecsNumericOptionName, "also include numeric codes"),
cmds.BoolOption(codecsNumericOptionName, "n", "also include numeric codes"),
cmds.BoolOption(codecsSupportedOptionName, "s", "list only codecs supported by go-ipfs commands"),
},
Run: func(req *cmds.Request, resp cmds.ResponseEmitter, env cmds.Environment) error {
listSupported, _ := req.Options[codecsSupportedOptionName].(bool)
supportedCodecs := make(map[uint64]struct{})
if listSupported {
for _, code := range multicodec.ListEncoders() {
supportedCodecs[code] = struct{}{}
}
for _, code := range multicodec.ListDecoders() {
supportedCodecs[code] = struct{}{}
}
// add libp2p-key
supportedCodecs[uint64(mc.Libp2pKey)] = struct{}{}
}

var res []CodeAndName
// use CodecToStr as there are multiple names for a given code
for code, name := range cid.CodecToStr {
res = append(res, CodeAndName{int(code), name})
for _, code := range mc.KnownCodes() {
if code.Tag() == "ipld" {
if listSupported {
if _, ok := supportedCodecs[uint64(code)]; !ok {
continue
}
}
res = append(res, CodeAndName{int(code), mc.Code(code).String()})
}
}
return cmds.EmitOnce(resp, res)
},
Expand All @@ -334,6 +367,9 @@ var codecsCmd = &cmds.Command{
var hashesCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "List available multihashes.",
ShortDescription: `
'ipfs cid hashes' relies on https://github.com/multiformats/go-multihash
`,
},
Options: codecsCmd.Options,
Run: func(req *cmds.Request, resp cmds.ResponseEmitter, env cmds.Environment) error {
Expand Down
4 changes: 2 additions & 2 deletions core/coreapi/block.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ func (api *BlockAPI) Put(ctx context.Context, src io.Reader, opts ...caopts.Bloc
ctx, span := tracing.Span(ctx, "CoreAPI.BlockAPI", "Put")
defer span.End()

settings, pref, err := caopts.BlockPutOptions(opts...)
settings, err := caopts.BlockPutOptions(opts...)
if err != nil {
return nil, err
}
Expand All @@ -41,7 +41,7 @@ func (api *BlockAPI) Put(ctx context.Context, src io.Reader, opts ...caopts.Bloc
return nil, err
}

bcid, err := pref.Sum(data)
bcid, err := settings.CidPrefix.Sum(data)
if err != nil {
return nil, err
}
Expand Down
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ require (
github.com/ipfs/go-unixfs v0.3.1
github.com/ipfs/go-unixfsnode v1.1.3
github.com/ipfs/go-verifcid v0.0.1
github.com/ipfs/interface-go-ipfs-core v0.6.2
github.com/ipfs/interface-go-ipfs-core v0.6.3-0.20220413011532-7e14e1e09703
github.com/ipfs/tar-utils v0.0.2
github.com/ipld/go-car v0.3.2
github.com/ipld/go-car/v2 v2.1.1
Expand Down Expand Up @@ -96,7 +96,7 @@ require (
github.com/multiformats/go-multiaddr v0.5.0
github.com/multiformats/go-multiaddr-dns v0.3.1
github.com/multiformats/go-multibase v0.0.3
github.com/multiformats/go-multicodec v0.4.0
github.com/multiformats/go-multicodec v0.4.1
github.com/multiformats/go-multihash v0.1.0
github.com/opentracing/opentracing-go v1.2.0
github.com/pkg/errors v0.9.1
Expand Down
10 changes: 6 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -605,8 +605,10 @@ github.com/ipfs/go-unixfsnode v1.1.3/go.mod h1:ZZxUM5wXBC+G0Co9FjrYTOm+UlhZTjxLf
github.com/ipfs/go-verifcid v0.0.1 h1:m2HI7zIuR5TFyQ1b79Da5N9dnnCP1vcu2QqawmWlK2E=
github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0=
github.com/ipfs/interface-go-ipfs-core v0.4.0/go.mod h1:UJBcU6iNennuI05amq3FQ7g0JHUkibHFAfhfUIy927o=
github.com/ipfs/interface-go-ipfs-core v0.6.2 h1:nnkq9zhb5O8lPzkZeynEymc83RqkTRqfYH4x5JNUkT4=
github.com/ipfs/interface-go-ipfs-core v0.6.2/go.mod h1:h3NuO3wzv2KuKazt0zDF2/i8AFRqiKHusyh5DUQQdPA=
github.com/ipfs/interface-go-ipfs-core v0.6.3-0.20220412233708-a6eb01527953 h1:0jHj/vY8khomu6mCvmRlJT0RNMWM2f108uYoMGsqyX0=
github.com/ipfs/interface-go-ipfs-core v0.6.3-0.20220412233708-a6eb01527953/go.mod h1:lF27E/nnSPbylPqKVXGZghal2hzifs3MmjyiEjnc9FY=
github.com/ipfs/interface-go-ipfs-core v0.6.3-0.20220413011532-7e14e1e09703 h1:HAdB02slFC+/zxf3szFvXqdtRrwvE7xBKxCRxzFrOFc=
github.com/ipfs/interface-go-ipfs-core v0.6.3-0.20220413011532-7e14e1e09703/go.mod h1:lF27E/nnSPbylPqKVXGZghal2hzifs3MmjyiEjnc9FY=
github.com/ipfs/tar-utils v0.0.2 h1:UNgHB4x/PPzbMkmJi+7EqC9LNMPDztOVSnx1HAqSNg4=
github.com/ipfs/tar-utils v0.0.2/go.mod h1:4qlnRWgTVljIMhSG2SqRYn66NT+3wrv/kZt9V+eqxDM=
github.com/ipld/go-car v0.3.2 h1:V9wt/80FNfbMRWSD98W5br6fyjUAyVgI2lDOTZX16Lg=
Expand Down Expand Up @@ -1188,8 +1190,8 @@ github.com/multiformats/go-multibase v0.0.3/go.mod h1:5+1R4eQrT3PkYZ24C3W2Ue2tPw
github.com/multiformats/go-multicodec v0.2.0/go.mod h1:/y4YVwkfMyry5kFbMTbLJKErhycTIftytRV+llXdyS4=
github.com/multiformats/go-multicodec v0.3.0/go.mod h1:qGGaQmioCDh+TeFOnxrbU0DaIPw8yFgAZgFG0V7p1qQ=
github.com/multiformats/go-multicodec v0.3.1-0.20210902112759-1539a079fd61/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ=
github.com/multiformats/go-multicodec v0.4.0 h1:fbqb6ky7erjdD+/zaEBJgZWu1i8D6i/wmPywGK7sdow=
github.com/multiformats/go-multicodec v0.4.0/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ=
github.com/multiformats/go-multicodec v0.4.1 h1:BSJbf+zpghcZMZrwTYBGwy0CPcVZGWiC72Cp8bBd4R4=
github.com/multiformats/go-multicodec v0.4.1/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ=
github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U=
github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po=
github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew=
Expand Down
Loading