nsqd: Make sure data directory is writeable on startup

[mreiferson]

We ran into a problem this morning where our nsqd processes filled up their in-memory queues and couldn't persist the messages to disk. The error log is here:

2013/04/25 17:42:50 CHANNEL(**) ERROR: failed to write message to backend - open *__:**.diskqueue.000000.dat: permission denied

It was our fault for not specifying the --data-path on the command line, but this is something that could easily be checked on startup, leaving it until the in-memory queue is full means that if nsqd needs to be reconfigured and restarted, all messages will be dropped. This is a significant problem.

Also, it's impossible to tell from the error message where the directory is that the data is being saved. Does this default to the root directory, or is it a blank string like the command-line documentation shows?

[mreiferson]

@rafikk yep, agreed, thanks for the report.

It does actually default to an empty string, the current working directory.

[rafikk]

Okay, @mreiferson, thanks for the response. Maybe this should be a required parameter? We were using supervisord to manage the process and it was starting it from the root directory.

Perhaps nsqd should create the .dat files on startup and append to them when the memory queue overflows instead of deferring their creation to when it's needed.

[mreiferson]

yea, I've gone back and forth in my head about being able to provide sane defaults for command line arguments so when you're just experimenting with the tools you don't have to configure so much.

I think your suggestion is the right one... it should just confirm that its data directory is writable on startup and exit spectacularly if it isn't 😄

You can't create the topic/channel .dat files at initial startup because they don't exist yet (they're created lazily by a producer or consumer) but it can still create the daemon's .dat file to confirm writability.

[mreiferson]

RFR @jehiah (cc @rafikk)

[jehiah]

LGTM