022 Project 22: Log Shipper CLI

Updated

September 8, 2026

022 Build a Log Shipper CLI

Tail a log file and POST new lines to an HTTP ingestion endpoint. This project combines follow-mode I/O, JSON payloads, HTTP client timeouts, and optional batching for efficiency.

file tail -> line channel -> (optional batch) -> HTTP POST /ingest

Problem statement

Build logship:

  • Seek to end of file (or start with -from-start)
  • Follow new lines (like tail -f)
  • POST each line (or batch) as JSON to -endpoint
  • Survive transient network errors without exiting the process
  • Exit cleanly on SIGINT

Acceptance criteria

  • New lines after start are POSTed
  • HTTP client has a timeout
  • Failed POSTs log to stderr and continue following
  • Ctrl+C stops the loop
  • Module builds with stdlib only

Setup

mkdir logship && cd logship
go mod init example.com/logship
# go 1.27

Full main.go

package main

import (
    "bufio"
    "bytes"
    "context"
    "encoding/json"
    "flag"
    "fmt"
    "io"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

type payload struct {
    TS   string `json:"ts"`
    Line string `json:"line"`
    File string `json:"file"`
}

func shipLine(ctx context.Context, client *http.Client, endpoint, file, line string) error {
    body, err := json.Marshal(payload{
        TS:   time.Now().Format(time.RFC3339Nano),
        Line: line,
        File: file,
    })
    if err != nil {
        return err
    }
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
    if err != nil {
        return err
    }
    req.Header.Set("Content-Type", "application/json")
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    _, _ = io.Copy(io.Discard, resp.Body)
    if resp.StatusCode >= 300 {
        return fmt.Errorf("status %d", resp.StatusCode)
    }
    return nil
}

func main() {
    file := flag.String("file", "app.log", "log file")
    endpoint := flag.String("endpoint", "http://localhost:8080/ingest", "ingest URL")
    fromStart := flag.Bool("from-start", false, "ship existing content first")
    poll := flag.Duration("poll", 300*time.Millisecond, "poll interval on EOF")
    flag.Parse()

    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    f, err := os.Open(*file)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    defer f.Close()

    if !*fromStart {
        if _, err := f.Seek(0, io.SeekEnd); err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
    }

    r := bufio.NewReader(f)
    client := &http.Client{Timeout: 3 * time.Second}

    fmt.Fprintf(os.Stderr, "shipping %s -> %s\n", *file, *endpoint)

    for {
        select {
        case <-ctx.Done():
            fmt.Fprintln(os.Stderr, "shutdown")
            return
        default:
        }

        line, err := r.ReadString('\n')
        if err == io.EOF {
            // partial line without newline: keep in buffer via Reader
            select {
            case <-ctx.Done():
                return
            case <-time.After(*poll):
            }
            continue
        }
        if err != nil {
            fmt.Fprintln(os.Stderr, "read error:", err)
            select {
            case <-ctx.Done():
                return
            case <-time.After(*poll):
            }
            continue
        }

        if err := shipLine(ctx, client, *endpoint, *file, line); err != nil {
            fmt.Fprintln(os.Stderr, "post failed:", err)
            // continue tailing
        }
    }
}

Minimal ingest receiver (for local test)

ingest.go (separate program or build tag):

//go:build ignore

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/ingest", func(w http.ResponseWriter, r *http.Request) {
        b, _ := io.ReadAll(r.Body)
        fmt.Printf("got: %s\n", b)
        w.WriteHeader(http.StatusNoContent)
    })
    log.Println("ingest :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}
go run ingest.go
# other terminal:
echo 'hello ship' >> app.log
go run . -file app.log -endpoint http://localhost:8080/ingest
# another terminal:
echo 'line two' >> app.log

Step-by-step build path

  1. Open file; seek end unless -from-start.
  2. Loop ReadString('\n'); on EOF sleep/poll.
  3. Marshal JSON; POST with context + timeout.
  4. Wire SIGINT via signal.NotifyContext.
  5. Add batching: accumulate N lines or T ms, one POST.

Batching stretch (sketch)

// flush []string every 100 lines or 1s via time.Ticker
type batchPayload struct {
    Lines []payload `json:"lines"`
}

Verification

go build -o logship .
# with ingest running:
printf 'a\nb\n' >> app.log
# expect two POSTs (or one batch)

Pitfalls

Pitfall Fix
No HTTP timeout hang forever
Exit on first POST error use continue + log
Truncate/rotation not handled detect inode/size shrink; reopen (stretch)
Busy loop on EOF poll sleep / ticker
Missing newline at EOF keep partial buffer (bufio does)

Learning goals

  • Follow-mode file IO
  • Resilient HTTP clients
  • Graceful shutdown with context
  • Path to production shippers (batch, backoff, rotation)