nsqd: panic when running two instances with same -data-path

[mreiferson]

panic: runtime error: makeslice: len out of range

goroutine 16 [running]:
github.com/bitly/nsq/nsqd.(*diskQueue).readOne(0xc2080d4160, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/cheney/Projects/GoProject/src/github.com/bitly/nsq/nsqd/diskqueue.go:264 +0x56a
github.com/bitly/nsq/nsqd.(*diskQueue).ioLoop(0xc2080d4160)
    /Users/cheney/Projects/GoProject/src/github.com/bitly/nsq/nsqd/diskqueue.go:576 +0x706
created by github.com/bitly/nsq/nsqd.newDiskQueue
    /Users/cheney/Projects/GoProject/src/github.com/bitly/nsq/nsqd/diskqueue.go:92 +0x403

This panic above is cause by

readBuf := make([]byte, msgSize)

I assume that maybe msgSize is negative, and running multiple instances in the same directory is possible?

[mreiferson]

The way diskqueue filenames are currently implemented, it's not going to work.

I don't have strong feelings on this either. It sounds like something we should probably just document? It isn't too hard of a requirement to ask that data paths be unique.

Thoughts @jehiah?

[jehiah]

I'm not sure there is a portable way to solve this, but could we open nsq.%d.dat with an exclusive lock similar to setlock?

[cespare]

Easy enough on platforms with flock(2):

https://github.com/cespare/kvcache/blob/master/db.go#L376-L391

[jehiah]

@cespare cool. Would you be interested in contributing that for the platforms where we can support it?

[mreiferson]

I don't think it's as simple as locking on the metadata file - the nsqd would be competing trying to deliver + cleanup any persisted message backlog.

To me, the question is: do we actually want you to be able to point two nsqd at the same data path?

[jehiah]

Oh, @mreiferson you mean because we write to a temp file and replace the metadata locking is somewhat a fools errand.

We don't want to allow two nsqd to point at the same data path, i was thinking this as a way to ensure that you don't accidentally end up with that happening. (fail fast, fail early.

[mreiferson]

@jehiah got it - maybe we just explicitly add a lock file to the data path that we can rely on to detect and fail fast?

[chzyer]

@mreiferson @jehiah
The idea that adding a lock file to the data path is awesome!
But if nsqd crash or kill by SIGTERM any ways that unable to delete the lock file, this nsqd couldn't run any more before delete the lock file manually?
I think the key point is named each nsqd when it starting, the name which nsqd belongs to should be unique with any other nsqd running in this server, and every time a nsqd started with the same configure should got the same name. (e.g. {TcpPort}:{HttpPort})
So if the configure not changed, nsqd can detect that the lockfile is belong to it.

[cespare]

You create the file if it doesn't exist, and then try to lock it. You don't need to delete it.

When nsqd exits (maybe because it crashes), the flock is released.

[cespare]

(Also if you have a dir to put the lock file, you might as well just lock the dir instead.)

[chzyer]

👍 cool :)

[mreiferson]

👍 to locking dir

[mreiferson]

RFR @jehiah

Windows isn't implemented - if someone who's running on Windows wants to contribute those code paths that would be great!

[jehiah]

I think it would be helpful if this error gave a little bit of a hint that it's in use by another nsqd, or that the resolution is to use a different/unique dirpath.

Thoughts?

[mreiferson]

sure

[mreiferson]

updated

[pacoguzman]

I'm getting:

nsqd --lookupd-tcp-address=localhost:4160 --data-path=/data
[nsqd] 2017/04/21 16:18:59.356983 FATAL: --data-path=/data in use (possibly by another instance of nsqd)

On a Mac OS machine, but that' the only process starting nsqd, could someone help me to setup a directory to store disk-backed messages

[ploxiln]

please ask this kind of question at https://groups.google.com/d/forum/nsq-users