None of the other methods in the measure package return this error,
instead they only call RecordValue() when the value is []byte.  This
change makes batch Put consistent with the other methods and allows
non []byte data to be passed though the measure datastore.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

The datastore has an optional "advanced" datastore that handles Put
requests for non []byte values, a "normal" datastore that handles all
other put requests, and then any number of other datastore, some of
them that can be designated read-only.  Delete requests are passed on
to all datastore not designed read-only.

For now, querying will only work on a "normal" datastore.

Note: Only tested in the case of just a "normal" datastore and the
case of an "advanced" and "normal" datastore.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Create a new AdvReader interface that returns an ExtraInfo object that
contains extra information about the the filehandle.  This includes
the current offset and the absolute path of a file.

Also modify chunk.Splitter to return a Bytes struct in the NextBytes()
method that in addition to the raw bytes returns the ExtraInfo object
from the AdvReader that contains the current offset in the file
(amoung other information).

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

This involved:

1) Maintaing a DataPtr in merkledag.Node and blocks.Block.  The
DataPtr points to the location of the data within a file on the file
system.  It the node is a leaf it also contains an alternative
serialization of the Node or Block that does not contain the data.

3) A new datastore "filestore" that stores just the information in
DataPtr.  When retrieving blocks the Merkle-DAG node is reconstructed
from combining AltData with the data from the file in the file system.
Because the datastore needs to reconstruct the node it needs access to
the Protocol Buffers for "merkledag" and "unixfs" and thus, for now,
lives in go-ipfs instead of go-datastore.

The filestore uses another datastore to store the protocol buffer
encoded DataPtr.  By default this is the leveldb datastore, as the size
for the encoded DataPtr is small.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Note that as per issue ipfs#2259 content is not verified for local
retrieval so changing the file content will not always be detected.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

….Put.

If the dataObj.Offset is zero and the WholeFile flag is not set, check
if it should be.  That is check if the file size is the same as
dataObj.Size, and if it is, set dataObj.WholeFile.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Add tests for:
  filestore ls
  filestore verify
  filestore rm-invalid
  filestore rm

Also rename t0046-add-no-copy.sh to t0260-filestore.sh.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

LevelDB Delete() method will not return ErrNotFound so check that the key
exists first by calling Has() and return ds.ErrNotFound if Has() returns
false.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Use the new function reconstructDirect that will reconstruct the block
directly without any intermediate data structures and without
performing any unnecessary copies of the block data from the file.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Move most of the code from core/command/filestore.go into
filestore/util.go.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

This will simplify adding new flags.

Also modify the related Protocol Buffer.  The old format will still be
recognized for now.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

…hanged.

In DataObj add two fields: the file's modification time, and a flag to
mark the content as invalid.  If the modification time of the file is
the same as the one in the DataObj, skip validation and return the
cached value of the invalid flag.  If the modification time differs
than recheck the hash of the block and update the modification time
and invalid flag accordingly.

With this change retrieval of blocks from the filestore is as fast as
retrieving a block from the normal datastore.

Note that when a block is first added the modification time is unset
so the first retrial will be slower since the contents need to be
verified.  After the first retrieval no additional verification needs
to be done until the file changed.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

… read.

When adding a file to the filestore get the modification time of a
file before we read any data from it and use that time to set the
initial timestamp in the DataObj.  This avoids the need for verifying a
file when it is retrieved for the first time.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

In addition to checking if blocks can be read also check for other
problems.  (1) For each block representing a file root, check that all
the blocks in the merkledag tree can be read and the contents of the
file can be reconstructed.  (2) Check for orphan nodes, those are
nodes that are a child of another node that was not found in the
filestore. (3) Check if a file was appended too as the block is still
valid but no longer represents the contents of the file.

The "missing" status, which was used indicate that the file is not
found, is now "no-file" and "missing" is now used to indicate when a
block could not be found in the datastore.

The original verify behavior is still available by using the "--basic"
option.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

…mand.

The new "filestore clean" uses the output of "filestore verify" to
remove invalid or useless blocks.  The new command can now also removes
incomplete or orphan nodes.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

…ns".

The new command will repair broken pins by either removing or
reconstructing the pin so that invalid blocks are not pinned.  If the
pin points to a block that no longer exists it is simply removed.  If
the pin is a recursive pin that points to a valid block but some of
the nodes of the merkle tree are invalid the recursive pin is
converted to a direct pin and new recursive pins are created for the
nodes that are still valid.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

The new command pins any un-pined nodes in a filestore that represent
whole files.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Refactor some of the functionality of Repin into a new walkPins function.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

…dle pines.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Just listing the unpinned objects is more flexible.  For the old
functionally just use something like:
  ipfs filestore unpinned | xargs ipfs pin add

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

If "--force" is used than indirect pins are ignored so save time and
don't check for them in the first place.  This will also produce
cleaner output when using "filestore clean" as it basically calls
"filestore rm --force --direct".

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

In the various components of the "files" package do not store a
separate absolute path, just make the FullPath absolute when possible.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

…he data.

If the option API.ServerSideAdds is enabled than the new command
"add-ss" will add files to ipfs by sending the filename, rather than
the contents of the file, to the API server.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

With the new "add-ss" command it is now possible to add files to the
filestore with the daemon online.  Update README to reflect this
fact and create some new test cases to test the new functionally.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

…ned node.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

The default value of "ifchanged" and blocks are only verified when
the backing file has changed.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Issues now on https://github.com/ipfs-filestore/go-ipfs/issues

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Restore blockstore, write_cache to original unmodified versions.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

This also eliminate the need for AddOpts and ExtraInfo interface.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

This completely elements the need for AdvReader and cleans up a lot of
other code.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

The datastore now only sends Put and Delete requests to the first
datastore.  Adding to other datastores is done directly.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

If neither "-w" or "-r" is specified don't use an mfs-root.  Add and
pin each file separately.

Towards ipfs#2811

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Also change "add-ss --no-copy" to "filestore add-ss"  and remove "add-ss".

The API.ServerSideAdds option is now Filestore.APIServerSidePaths.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Rework "filestore add" command to now always accept string arguments for
better control on how filenames are handled and eliminate the need
for a separate "add-ss" command.

For now absolute paths are always required.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Implements: "// TODO: handle err (which wasn't handled either when
the splitter was channeled".

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Directory adds not supported yet.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Empty files don't have a DataPtr for some reason and thus will cause an
error when added to the filestore.  So, for now just don't add them to
the filestore.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Use the new function to clean the path before adding it to the filestore.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

…options.

Add the '-P' (--physical) and '-l' (--logical) options to filestore
add to create absolute directories from relative ones.  The '-P'
options uses the physical (no symbolic links) location of the current
directory and the '-l' option uses the PWD env. variable if it is
available and if it names the current directory.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Requires some windows specific stubs.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

(Accidental deleted some lines before commit.)

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Update README to reflect the fact that the filestore has now been tested
on Windows.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

… leak.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Eliminate the need for DataPtr by reconstruct the the Node before
sending it to the Filestore.

This eliminates the need for some bookkeeping at the cost of doing
some unnecessary work further down.

Informal test show the performance is about the same.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

Generalize Reconstruct() function so that either the reference or the
optimized version of reconstruct can be used interchangeably.  For now
the reference implementation is used as the optimized version does not
work when blockDataSize == 0.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

The old version could only handle adding the data component to a
protocol buffer encoded node.  The new version adds the ability to
also remove the data component.

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

This also eliminates the need for FilestoreBlock

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT
Signed-off-by: Kevin Atkinson <k@kevina.org>