runtime: support for daemonize

[gopherbot]

by azummo-google@towertech.it:

(Take this as LongTerm or Thinking)

It would be fine to have a fork and daemonize calls,
something like this:

func Daemonize()
{
        os.Stdin.Close();
        os.Stdout.Close();
        os.Stderr.Close();

        pid := Fork();

        // In parent
        if pid != 0 {
                os.Exit(0);
        }

        // In child
        syscall.Setsid()

        ....
}

[rsc]

Comment 1:

Unfortunately, this is a bit harder than it looks.
On most systems, threads and daemonize do
not play well with each other.

Owner changed to r...@golang.org.

Status changed to LongTerm.

[gopherbot]

Comment 2 by azummo-google@towertech.it:

I'm sure it is!
Here (linux/386) I'm trying daemonizing early in main()
and launching goroutine safterwards. Seems working nicely.

[rsc]

Comment 3:

Labels changed: added packagechange.

[gopherbot]

Comment 4 by graycardinalster:

Full analog of "daemon(3)"
import (
 "syscall"
 "os"
)
func daemon (nochdir, noclose int) int {
    var ret uintptr
    var err uintptr
    ret,_,err = syscall.Syscall(syscall.SYS_FORK, 0, 0, 0)
    if err != 0 { return -1 }
    switch (ret) {
        case 0:
            break
        default:
            os.Exit (0)
    }
    if syscall.Setsid () == -1 { return -1 }
    if (nochdir == 0) { os.Chdir("/") }
    if noclose == 0 {
        f, e := os.Open ("/dev/null", os.O_RDWR, 0)
        if e == nil {
            fd := f.Fd ()
            syscall.Dup2 (fd, os.Stdin.Fd ())
            syscall.Dup2 (fd, os.Stdout.Fd ())
            syscall.Dup2 (fd, os.Stderr.Fd ())
        }
    }
    return 0
}

[gopherbot]

Comment 5 by graycardinalster:

Is it possible to add this feature in Go?

[rsc]

Comment 6:

It's possible but not simple to add this feature to Go.
The implementation in comment #4 is not correct when
the program is multithreaded, as most interesting Go 
programs are.  (The parent gets all the threads and
the child gets none of them.)

[gopherbot]

Comment 7 by graycardinalster:

Threads can not be copied anyway. Just use "daemon" before any go's

[rsc]

Comment 8:

There's no guarantee that you can arrange for the call to daemon
to happen before any go's.

[gopherbot]

Comment 9 by krmigr:

While it might be useful to have a Fork() function for other purposes, I'd suggest
omitting and discouraging any sort of Daemon()-like feature.
It's usually a mistake for a daemon to fork itself. Self-daemonizing makes life
unnecessarily hard for anyone who wants to monitor the daemon process.

[bsiegert]

Comment 10:

I disagree with the last comment. Unix daemons have always worked this way. While a
well-written daemon should always have an option to disable forking and to stay in the
foreground, not providing a daemon()-like call in the language for this reason is the
wrong approach.

[rsc]

Comment 11:

Labels changed: added priority-later.

[extemporalgenome]

Comment 12:

I've run into a number of cases where, say, a given Linux distribution doesn't provide a
useful/secure/correct daemonization wrapper utility, if any at all. In this case,
self-daemonization, which is the defacto method, is assumed (and a good self-daemonizing
program will have an option to leave a pidfile and logfiles, which will make it easy to
monitor the process).  Since daemonization really only occurs in the wild during
initialization, it really ought to be called from an init function.
In order to make fork safe for daemonization, and seeing that the spec has now changed
(goroutines now *can* run during initialization), all we need to do is any one of three
things:
1) Disallow creation of os threads during initialization, where go's concurrency support
is only for concurrency, not parallelization, as Rob Pike might say -- all goroutine
switching during init would probably need to be cooperative (through blocking or
runtime.Gosched), which may not be possible to accomplish in the current runtime.
2) Create something like a runtime.SuspendAll() function, upon returning, guarantees
that all other threads are muxed off of os threads (which are destroyed) and the calling
goroutine is run alone until a complimentary function, like a runtime.ResumeAll() is
called. This is not the same as GOMAXPROCS, which doesn't restrict the number of threads
used by the runtime (this option would need to account for those as well).
3) Amend the published spec to require that unrelated dependencies initialize in
code-order (this would also mean that gofmt wouldn't be allowed to reorder imports), or
at least that independent import groups occur in source order. So at the minimum, the
following should require that daemon initialize fully before threadspawner:
import "daemon" // daemon imports nothing
import (
  "fmt"
  "os"
  "threadspawner" // threadspawner imports nothing
)
(Binaries produced from 6g and friends appear to do this in either undefined or reverse
order).
Knowing nothing about the toolchain/runtime internals, #3 is probably the simplest to
implement -- if you want to argue that the compiler be allowed to optimize for speed
(putting goroutines-spawning initializers before non-concurrent initializers), keep in
mind that initialization is not a critical place for speed (in a long running program,
initialization time counts for nothing, and in short running programs, heavy lifting is
usually *not* done in initialization).

[rsc]

Comment 13:

It is extremely unlikely that we will make a spec change
for daemonization.  We will just make it work.

[gopherbot]

Comment 14 by cw@f00f.org:

We can make daemonize work 'as expected' with a few restrictions.
These restrictions won't apply to most people and almost never if called in or near
program start-up.
These restrictions aren't unique to Go, but we can detect and error on them rather than
silently ignore them (as is sometimes the case elsewhere).

[rsc]

Comment 15:

The way to do it is to have a separate package
that you have to import to get Daemonize.
If you've imported that package, then the runtime
will know from the very beginning that you are
going to call it and can plan accordingly.
Russ

[extemporalgenome]

Comment 16:

Special casing a standard daemon package actually sounds like the most intuitive thing
-- thanks Russ! In any case, programmers who roll their own almost always do the wrong
thing (I think even the C daemon call doesn't handle it even close to securely), and we
shouldn't make them worry about implementation details.

[extemporalgenome]

Comment 17:

Special casing a standard daemon package actually sounds like the most intuitive thing
-- thanks Russ! In any case, programmers who roll their own almost always do the wrong
thing (I think even the C daemon call doesn't handle it even close to securely), and we
shouldn't make them worry about implementation details.

[alberts]

Comment 19:

I think there is a similar problem with calling unshare inside a Go program.

[rsc]

Comment 20:

Labels changed: added go1.1maybe.

[gopherbot]

Comment 21 by rjmcguire:

Would your changes allow one to have multiple processes accepting socket connections for
single listening socket like C does*?
basically:
proc1: sock.listen
proc1: fork N children
procN: s = sock.accept
procN: do work on s
* on Linux this is possible not sure about other OSes

[rsc]

Comment 22:

That's unrelated to daemonize, although it is something 'daemons' do.
You can do that today by doing a net.Listen and then passing the
l.Fd() to multiple os.StartProcess.
Russ

[gopherbot]

Comment 23 by cw@f00f.org:

Or you could use SCM_RIGHTS after the fact.

[gopherbot]

Comment 24 by rjmcguire:

Fd as args doesn't seem to work, File in Files doesn't seem to work either. UNIX sockets
are ancient (lines of code and problems).
Are there any examples that work and show passing a socket through Files to StartProcess.

[gopherbot]

Comment 25 by rjmcguire:

Thanks, using StartProcess for now.

[gopherbot]

Comment 26 by matthias.benkmann:

How about this
type MainFunc func(...interface{})   
// Forks a child process and executes main(args...) within that process. 
// Returns the process id of the child.
//
// Within the child process, a new Go runtime is initialized from scratch, as
// if the program had started its own binary via os.StartProcess(). However as
// the last step instead of starting main.main(), the child process will start    
// the main function passed to Fork(). 
//
// The parameters args are passed via encoding/gob with the corresponding
// restrictions.
//
// Open file descriptors, os.Args and the environment are inherited from  
// the parent process, NOT reset to the values they had at the start of     
// the program.
//
// Note: Fork() does not actually re-execute the program's binary.
// The program's binary need not be executable for purposes of Fork()
// Fork() will succeed even if the process no longer has the necessary
// privileges to execute the binary or if it has been (re)moved.
func Fork(main MainFunc, args... interface{}) (int, error) {
...
}
This Fork() has well-defined semantics and can be used for daemonizing as well as many
other uses of fork().
Implementing the ability to set up a new runtime environment without accessing the
binary shouldn't be too tough. The compiler might have to be changed to keep an
additional read-only copy of initial values for certain structures around. But that
doesn't require changing the language spec.

[gopherbot]

Comment 27 by rjmcguire:

You can already detach from the controlling terminal by using StartProcess
and setting stdin,stdout, and stderr to nil (Only tested on ubuntu
12.04 bash in a gnome-terminal).
However we would still need a fork() approach if you wanted to
drop privileges before detaching.
Rory McGuire
Email: rjmcguire@gmail.com
Mobile: (+27)(0) 82 856 3646

[robpike]

Comment 28:

Labels changed: removed go1.1maybe.

[gopherbot]

Comment 29 by sergey.yarmonov:

You can try package https://github.com/sevlyar/go-daemon

[rsc]

Comment 30:

Labels changed: added go1.3maybe.

[rsc]

Comment 31:

Labels changed: added release-none, removed go1.3maybe.

[cschwarz-inco]

I found those daemon packages that have emerged too complicated, here is my solution:

package main

import (
    "log"
    "os"
    "os/exec"
    "syscall"
    "time"
)

func main() {
    // check command line flags, configuration etc.

    // short delay to avoid race condition between os.StartProcess and os.Exit
    // can be omitted if the work done above amounts to a sufficient delay
    time.Sleep(1 * time.Second)

    if os.Getppid() != 1 {
        // I am the parent, spawn child to run as daemon
        binary, err := exec.LookPath(os.Args[0])
        if err != nil {
            log.Fatalln("Failed to lookup binary:", err)
        }
        _, err = os.StartProcess(binary, os.Args, &os.ProcAttr{Dir: "", Env: nil,
                Files: []*os.File{os.Stdin, os.Stdout, os.Stderr}, Sys: nil})
        if err != nil {
            log.Fatalln("Failed to start process:", err)
        }
        os.Exit(0)
    } else {
        // I am the child, i.e. the daemon, start new session and detach from terminal
        _, err := syscall.Setsid()
        if err != nil {
            log.Fatalln("Failed to create new session:", err)
        }
        file, err := os.OpenFile("/dev/null", os.O_RDWR, 0)
        if err != nil {
            log.Fatalln("Failed to open /dev/null:", err)
        }
        syscall.Dup2(int(file.Fd()), int(os.Stdin.Fd()))
        syscall.Dup2(int(file.Fd()), int(os.Stdout.Fd()))
        syscall.Dup2(int(file.Fd()), int(os.Stderr.Fd()))
        file.Close()
    }

    // daemon business logic starts here
}

[extemporalgenome]

Your race-avoidance code still forms a race condition

[cschwarz-inco]

This should eliminate the race condition. Instead of the sleep, do:

    err := syscall.FcntlFlock(os.Stdout.Fd(), syscall.F_SETLKW, &syscall.Flock_t{
        Type: syscall.F_WRLCK, Whence: 0, Start: 0, Len: 0 })
    if err != nil {
        log.Fatalln("Failed to lock stdout:", err)
    }

[bradfitz]

Yeah, I think it's time we finally closed this. The answer is init systems like systemd, launchd, daemontools, supervisor, runit, Kubernetes, heroku, Borg, etc etc.

[robpike]

Don't always need systemd, launchd, daemontools, supervisor, runit, Kubernetes, heroku, Borg, etc etc. when you have the shell's & operator.