Compare commits

..

16 Commits
v0.3.1 ... main

Author SHA1 Message Date
694f8ba1d3 Add comprehensive godoc documentation to all packages
- Add package-level documentation with usage examples and architecture details
- Document all public types, functions, and methods following godoc conventions
- Remove unused logger.Error type and NewError function
- Apply consistent documentation style across all packages

Packages updated:
- apitls: TLS certificate management with automatic renewal
- config: Environment-based configuration system
- config/depenv: Deployment environment handling
- ekko: Enhanced Echo web framework wrapper
- kafka: Kafka client wrapper with TLS support
- logger: Structured logging with OpenTelemetry integration
- tracing: OpenTelemetry distributed tracing setup
- types: Shared data structures for NTP Pool project
- xff/fastlyxff: Fastly CDN IP range management

All tests pass after documentation changes.
2025-06-19 23:52:03 -07:00
09b52f92d7 version: add documentation and tests 2025-06-06 20:19:08 -07:00
785abdec8d ulid: simplify, add function without a timestamp 2025-06-06 20:02:23 -07:00
ce203a4618 Add README 2025-06-06 19:56:43 -07:00
3c994a7343 Add copilot/claude instructions 2025-06-06 19:50:30 -07:00
f69c3e9c3c ulid: add documentation and more tests 2025-06-06 19:31:28 -07:00
fac5b1f275 metrics: add tests and documentation 2025-06-06 19:24:30 -07:00
a37559b93e health: add documentation 2025-06-06 19:16:14 -07:00
faac09ac0c timeutil: Add documentation 2025-06-06 19:08:16 -07:00
62a7605869 config: add depenv.MonitorDomain() and config.ManageURL() methods 2025-04-19 23:07:08 -07:00
0996167865 modernize + gofumpt 2025-04-19 22:19:02 -07:00
87344dd601 version: KongVersionCmd type 2025-04-12 00:24:19 -07:00
39e6611602 build: update goreleaser 2025-04-12 00:23:33 -07:00
355d246010 depenv: implement UnmarshalText 2025-04-12 00:22:57 -07:00
e5836a8b97 depenv: ntppool configuration for deployment environments 2025-01-26 11:08:44 -08:00
f6d160a7f8 health: fix shutdown of health check server 2025-01-03 14:01:52 +01:00
34 changed files with 1954 additions and 175 deletions

1
.github/copilot-instructions.md vendored Symbolic link
View File

@ -0,0 +1 @@
../CLAUDE.md

163
CLAUDE.md Normal file
View File

@ -0,0 +1,163 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Commands
### Testing
- Run all tests: `go test ./...`
- Run tests with verbose output: `go test -v ./...`
- Run tests for specific package: `go test ./config`
- Run specific test: `go test -run TestConfigBool ./config`
### Building
- Build all packages: `go build ./...`
- Check module dependencies: `go mod tidy`
- Verify dependencies: `go mod verify`
### Code Quality
- Format code: `go fmt ./...`
- Vet code: `go vet ./...`
- Run static analysis: `staticcheck ./...` (if available)
## Architecture
This is a common library (`go.ntppool.org/common`) providing shared infrastructure for the NTP Pool project. The codebase emphasizes observability, security, and modern Go practices.
### Core Components
**Web Service Foundation:**
- `ekko/` - Enhanced Echo web framework with pre-configured middleware (OpenTelemetry, Prometheus, logging, security headers)
- `health/` - Standalone health check HTTP server with `/__health` endpoint
- `metricsserver/` - Prometheus metrics exposure via `/metrics` endpoint
**Observability Stack:**
- `logger/` - Structured logging with OpenTelemetry trace integration and multiple output formats
- `tracing/` - OpenTelemetry distributed tracing with OTLP export support
- `metricsserver/` - Prometheus metrics with custom registry
**Configuration & Environment:**
- `config/` - Environment-based configuration with code-generated accessors (`config_accessor.go`)
- `version/` - Build metadata and version information with Cobra CLI integration
**Security & Communication:**
- `apitls/` - TLS certificate management with automatic renewal via certman
- `kafka/` - Kafka client wrapper with TLS support for log streaming
- `xff/fastlyxff/` - Fastly CDN IP range management for trusted proxy handling
**Utilities:**
- `ulid/` - Thread-safe ULID generation with monotonic ordering
- `timeutil/` - JSON-serializable duration types
- `types/` - Shared data structures (LogScoreAttributes for NTP server scoring)
### Key Patterns
**Functional Options:** Used extensively in `ekko/` for flexible service configuration
**Interface-Based Design:** `CertificateProvider` in `apitls/` for pluggable certificate management
**Context Propagation:** Throughout the codebase for cancellation and tracing
**Graceful Shutdown:** Implemented in web servers and background services
### Dependencies
The codebase heavily uses:
- Echo web framework with custom middleware stack
- OpenTelemetry for observability (traces, metrics, logs)
- Prometheus for metrics collection
- Kafka for message streaming
- Cobra for CLI applications
### Code Generation
`config/config_accessor.go` is generated - modify `config.go` and regenerate accessors when adding new configuration options.
## Package Overview
### `apitls/`
TLS certificate management with automatic renewal support via certman. Provides a CA pool for trusted certificates and interfaces for pluggable certificate providers. Used for secure inter-service communication.
### `config/`
Environment-based configuration system with code-generated accessor methods. Handles deployment mode, hostname configuration, and TLS settings. Provides URL building utilities for web and management interfaces.
### `ekko/`
Enhanced Echo web framework wrapper with pre-configured middleware stack including OpenTelemetry tracing, Prometheus metrics, structured logging, gzip compression, and security headers. Supports HTTP/2 with graceful shutdown.
### `health/`
Standalone HTTP health check server that runs independently from the main application. Exposes `/__health` endpoint with configurable health handlers, timeouts, and graceful shutdown capabilities.
### `kafka/`
Kafka client wrapper with TLS support for secure log streaming. Provides connection management, broker discovery, and reader/writer factories with compression and batching optimizations.
### `logger/`
Structured logging system with OpenTelemetry trace integration. Supports multiple output formats (text, OTLP) with configurable log levels, systemd compatibility, and context-aware logging.
### `metricsserver/`
Dedicated Prometheus metrics HTTP server with custom registry isolation. Exposes `/metrics` endpoint with OpenMetrics support and graceful shutdown handling.
### `timeutil/`
JSON-serializable duration types that support both string parsing ("30s", "5m") and numeric nanosecond values. Compatible with configuration files and REST APIs.
### `tracing/`
OpenTelemetry distributed tracing setup with support for OTLP export via gRPC or HTTP. Handles resource detection, propagation, and automatic instrumentation with configurable TLS.
### `types/`
Shared data structures for the NTP Pool project. Currently contains `LogScoreAttributes` for NTP server scoring with JSON and SQL database compatibility.
### `ulid/`
Thread-safe ULID (Universally Unique Lexicographically Sortable Identifier) generation using cryptographically secure randomness. Optimized for simplicity and performance in high-concurrency environments.
### `version/`
Build metadata and version information system with Git integration. Provides CLI commands for Cobra and Kong frameworks, Prometheus build info metrics, and semantic version validation.
### `xff/fastlyxff/`
Fastly CDN IP range management for trusted proxy handling. Parses Fastly's IP ranges JSON file and generates Echo framework trust options for proper client IP extraction.
## Go Development Best Practices
### Code Style
- Follow standard Go formatting (`go fmt ./...`)
- Use `go vet ./...` for static analysis
- Run `staticcheck ./...` when available
- Prefer short, descriptive variable names
- Use interfaces for testability and flexibility
### Error Handling
- Always handle errors explicitly
- Use `errors.Join()` for combining multiple errors
- Wrap errors with context using `fmt.Errorf("context: %w", err)`
- Return early on errors to reduce nesting
### Testing
- Write table-driven tests when testing multiple scenarios
- Use `t.Helper()` in test helper functions
- Test error conditions, not just happy paths
- Use `testing.Short()` for integration tests that can be skipped
### Concurrency
- Use contexts for cancellation and timeouts
- Prefer channels for communication over shared memory
- Use `sync.Once` for one-time initialization
- Always call `defer cancel()` after `context.WithCancel()`
### Performance
- Use `sync.Pool` for frequently allocated objects
- Prefer slices over arrays for better performance
- Use `strings.Builder` for string concatenation in loops
- Profile before optimizing with `go tool pprof`
### Observability
- Use structured logging with key-value pairs
- Add OpenTelemetry spans for external calls
- Include trace IDs in error messages
- Use metrics for monitoring application health
### Dependencies
- Keep dependencies minimal and well-maintained
- Use `go mod tidy` to clean up unused dependencies
- Pin major versions to avoid breaking changes
- Prefer standard library when possible
### Security
- Never log sensitive information (passwords, tokens)
- Use `crypto/rand` for cryptographic randomness
- Validate all inputs at API boundaries
- Use TLS for all network communication

20
README.md Normal file
View File

@ -0,0 +1,20 @@
Common library for the NTP Pool project with shared infrastructure components.
## Packages
- **apitls** - TLS setup for NTP Pool internal services with embedded CA
- **config** - NTP Pool project configuration with environment variables
- **ekko** - Enhanced Echo web framework with observability middleware
- **health** - Standalone health check HTTP server
- **kafka** - Kafka client wrapper with TLS support
- **logger** - Structured logging with OpenTelemetry integration
- **metricsserver** - Prometheus metrics HTTP server
- **timeutil** - JSON-serializable duration types
- **tracing** - OpenTelemetry distributed tracing setup
- **types** - Shared data structures for NTP Pool
- **ulid** - Thread-safe ULID generation
- **version** - Build metadata and version information
- **xff/fastlyxff** - Fastly CDN IP range management
[![Go Reference](https://pkg.go.dev/badge/go.ntppool.org/common.svg)](https://pkg.go.dev/go.ntppool.org/common)

View File

@ -1,3 +1,14 @@
// Package apitls provides TLS certificate management with automatic renewal support.
//
// This package handles TLS certificate provisioning and management for secure
// inter-service communication within the NTP Pool project infrastructure.
// It provides both server and client certificate management through the
// CertificateProvider interface and includes a trusted CA certificate pool
// for validating certificates.
//
// The package integrates with certman for automatic certificate renewal
// and includes embedded CA certificates for establishing trust relationships
// between services.
package apitls
import (
@ -13,11 +24,32 @@ import (
//go:embed ca.pem
var caBytes []byte
// CertificateProvider defines the interface for providing TLS certificates
// for both server and client connections. Implementations should handle
// certificate retrieval, caching, and renewal as needed.
//
// This interface supports both server-side certificate provisioning
// (via GetCertificate) and client-side certificate authentication
// (via GetClientCertificate).
type CertificateProvider interface {
// GetCertificate retrieves a server certificate based on the client hello information.
// This method is typically used in tls.Config.GetCertificate for server-side TLS.
GetCertificate(hello *tls.ClientHelloInfo) (*tls.Certificate, error)
// GetClientCertificate retrieves a client certificate for mutual TLS authentication.
// This method is used in tls.Config.GetClientCertificate for client-side TLS.
GetClientCertificate(certRequestInfo *tls.CertificateRequestInfo) (*tls.Certificate, error)
}
// CAPool returns a certificate pool containing trusted CA certificates
// for validating TLS connections within the NTP Pool infrastructure.
//
// The CA certificates are embedded in the binary and include the trusted
// certificate authorities used for inter-service communication.
// This pool should be used in tls.Config.RootCAs for client connections
// or tls.Config.ClientCAs for server connections requiring client certificates.
//
// Returns an error if the embedded CA certificates cannot be parsed or loaded.
func CAPool() (*x509.CertPool, error) {
capool := x509.NewCertPool()
if !capool.AppendCertsFromPEM(caBytes) {
@ -30,7 +62,6 @@ func CAPool() (*x509.CertPool, error) {
// GetCertman sets up certman for the specified cert / key pair. It is
// used in the monitor-api and (for now) in the client
func GetCertman(certFile, keyFile string) (*certman.CertMan, error) {
cm, err := certman.New(certFile, keyFile)
if err != nil {
return nil, err

View File

@ -1,5 +1,18 @@
// Package config provides NTP Pool specific
// configuration tools.
// Package config provides environment-based configuration management for NTP Pool services.
//
// This package handles configuration loading from environment variables and provides
// utilities for constructing URLs for web and management interfaces. It supports
// deployment-specific settings including hostname configuration, TLS settings,
// and deployment modes.
//
// Configuration is loaded automatically from environment variables:
// - deployment_mode: The deployment environment (devel, production, etc.)
// - manage_hostname: Hostname for management interface
// - web_hostname: Comma-separated list of web hostnames (first is primary)
// - manage_tls: Enable TLS for management interface (yes, no, true, false)
// - web_tls: Enable TLS for web interface (yes, no, true, false)
//
// The package includes code generation for accessor methods using the accessory tool.
package config
import (
@ -11,8 +24,11 @@ import (
"go.ntppool.org/common/logger"
)
//go:generate accessory -type Config
//go:generate go tool github.com/masaushi/accessory -type Config
// Config holds environment-based configuration for NTP Pool services.
// It manages hostnames, TLS settings, and deployment modes loaded from
// environment variables. The struct includes code-generated accessor methods.
type Config struct {
deploymentMode string `accessor:"getter"`
@ -26,6 +42,16 @@ type Config struct {
valid bool `accessor:"getter"`
}
// New creates a new Config instance by loading configuration from environment variables.
// It automatically parses hostnames, TLS settings, and deployment mode from the environment.
// The configuration is considered valid if at least one web hostname is provided.
//
// Environment variables used:
// - deployment_mode: Deployment environment identifier
// - manage_hostname: Management interface hostname
// - web_hostname: Comma-separated web hostnames (first becomes primary)
// - manage_tls: Management interface TLS setting
// - web_tls: Web interface TLS setting
func New() *Config {
c := Config{}
c.deploymentMode = os.Getenv("deployment_mode")
@ -46,10 +72,30 @@ func New() *Config {
return &c
}
// WebURL constructs a complete URL for the web interface using the primary web hostname.
// It automatically selects HTTP or HTTPS based on the web_tls configuration setting.
//
// Parameters:
// - path: URL path component (should start with "/")
// - query: Optional URL query parameters (can be nil)
//
// Returns a complete URL string suitable for web interface requests.
func (c *Config) WebURL(path string, query *url.Values) string {
return baseURL(c.webHostname, c.webTLS, path, query)
}
// ManageURL constructs a complete URL for the management interface using the management hostname.
// It automatically selects HTTP or HTTPS based on the manage_tls configuration setting.
//
// Parameters:
// - path: URL path component (should start with "/")
// - query: Optional URL query parameters (can be nil)
//
// Returns a complete URL string suitable for management interface requests.
func (c *Config) ManageURL(path string, query *url.Values) string {
return baseURL(c.manageHostname, c.webTLS, path, query)
}
func baseURL(host string, tls bool, path string, query *url.Values) string {
uri := url.URL{}
uri.Host = host

View File

@ -7,7 +7,6 @@ import (
)
func TestBaseURL(t *testing.T) {
os.Setenv("web_hostname", "www.ntp.dev, web.ntppool.dev")
os.Setenv("web_tls", "yes")
@ -22,5 +21,4 @@ func TestBaseURL(t *testing.T) {
if u != "https://www.ntp.dev/foo?foo=bar" {
t.Fatalf("unexpected WebURL: %s", u)
}
}

18
config/depenv/context.go Normal file
View File

@ -0,0 +1,18 @@
package depenv
import "context"
type contextKey struct{}
// NewContext adds the deployment environment to the context
func NewContext(ctx context.Context, d DeploymentEnvironment) context.Context {
return context.WithValue(ctx, contextKey{}, d)
}
// FromContext retrieves the deployment environment from the context
func FromContext(ctx context.Context) DeploymentEnvironment {
if d, ok := ctx.Value(contextKey{}).(DeploymentEnvironment); ok {
return d
}
return DeployUndefined
}

133
config/depenv/depenv.go Normal file
View File

@ -0,0 +1,133 @@
// Package depenv provides deployment environment management for NTP Pool services.
//
// This package handles different deployment environments (development, test, production)
// and provides environment-specific configuration including API endpoints, management URLs,
// and monitoring domains. It supports string-based environment identification and
// automatic URL construction for various service endpoints.
//
// The package defines three main deployment environments:
// - DeployDevel: Development environment with dev-specific endpoints
// - DeployTest: Test/beta environment for staging
// - DeployProd: Production environment with live endpoints
//
// Environment detection supports both short and long forms:
// - "dev" or "devel" → DeployDevel
// - "test" or "beta" → DeployTest
// - "prod" → DeployProd
package depenv
import (
"fmt"
"os"
)
var manageServers = map[DeploymentEnvironment]string{
DeployDevel: "https://manage.askdev.grundclock.com",
DeployTest: "https://manage.beta.grundclock.com",
DeployProd: "https://manage.ntppool.org",
}
var apiServers = map[DeploymentEnvironment]string{
DeployDevel: "https://dev-api.ntppool.dev",
DeployTest: "https://beta-api.ntppool.dev",
DeployProd: "https://api.ntppool.dev",
}
// var validationServers = map[DeploymentEnvironment]string{
// DeployDevel: "https://v.ntp.dev/d/",
// DeployTest: "https://v.ntp.dev/b/",
// DeployProd: "https://v.ntp.dev/p/",
// }
const (
// DeployUndefined represents an unrecognized or unset deployment environment.
DeployUndefined DeploymentEnvironment = iota
// DeployDevel represents the development environment.
DeployDevel
// DeployTest represents the test/beta environment.
DeployTest
// DeployProd represents the production environment.
DeployProd
)
// DeploymentEnvironment represents a deployment environment type.
// It provides methods for environment-specific URL construction and
// supports text marshaling/unmarshaling for configuration files.
type DeploymentEnvironment uint8
// DeploymentEnvironmentFromString parses a string into a DeploymentEnvironment.
// It supports both short and long forms of environment names:
// - "dev" or "devel" → DeployDevel
// - "test" or "beta" → DeployTest
// - "prod" → DeployProd
// - any other value → DeployUndefined
func DeploymentEnvironmentFromString(s string) DeploymentEnvironment {
switch s {
case "devel", "dev":
return DeployDevel
case "test", "beta":
return DeployTest
case "prod":
return DeployProd
default:
return DeployUndefined
}
}
// String returns the canonical string representation of the deployment environment.
// Returns "prod", "test", "devel", or panics for invalid environments.
func (d DeploymentEnvironment) String() string {
switch d {
case DeployProd:
return "prod"
case DeployTest:
return "test"
case DeployDevel:
return "devel"
default:
panic("invalid DeploymentEnvironment")
}
}
// APIHost returns the API server URL for this deployment environment.
// It first checks the API_HOST environment variable for overrides,
// then falls back to the environment-specific default API endpoint.
func (d DeploymentEnvironment) APIHost() string {
if apiHost := os.Getenv("API_HOST"); apiHost != "" {
return apiHost
}
return apiServers[d]
}
// ManageURL constructs a management interface URL for this deployment environment.
// It combines the environment-specific management server base URL with the provided path.
//
// The path parameter should start with "/" for proper URL construction.
func (d DeploymentEnvironment) ManageURL(path string) string {
return manageServers[d] + path
}
// MonitorDomain returns the monitoring domain for this deployment environment.
// The domain follows the pattern: {environment}.mon.ntppool.dev
// For example: "devel.mon.ntppool.dev" for the development environment.
func (d DeploymentEnvironment) MonitorDomain() string {
return d.String() + ".mon.ntppool.dev"
}
// UnmarshalText implements the encoding.TextUnmarshaler interface.
// It allows DeploymentEnvironment to be unmarshaled from configuration files
// and other text-based formats. Empty strings are treated as valid (no-op).
//
// Returns an error if the text represents an invalid deployment environment.
func (d *DeploymentEnvironment) UnmarshalText(text []byte) error {
s := string(text)
if s == "" {
return nil
}
env := DeploymentEnvironmentFromString(s)
if env == DeployUndefined {
return fmt.Errorf("invalid deployment environment: %s", s)
}
*d = env
return nil
}

View File

@ -0,0 +1,40 @@
package depenv
import (
"fmt"
"strings"
)
var monitorApiServers = map[DeploymentEnvironment]string{
DeployDevel: "https://api.devel.mon.ntppool.dev",
DeployTest: "https://api.test.mon.ntppool.dev",
DeployProd: "https://api.mon.ntppool.dev",
}
func (d DeploymentEnvironment) MonitorAPIHost() string {
return monitorApiServers[d]
}
func GetDeploymentEnvironmentFromName(clientName string) (DeploymentEnvironment, error) {
clientName = strings.ToLower(clientName)
if !strings.HasSuffix(clientName, ".mon.ntppool.dev") {
return DeployUndefined, fmt.Errorf("invalid client name %s", clientName)
}
if clientName == "api.mon.ntppool.dev" {
return DeployProd, nil
}
prefix := clientName[:strings.Index(clientName, ".mon.ntppool.dev")]
parts := strings.Split(prefix, ".")
if len(parts) != 2 {
return DeployUndefined, fmt.Errorf("invalid client name %s", clientName)
}
if d := DeploymentEnvironmentFromString(parts[1]); d != DeployUndefined {
return d, nil
}
return DeployUndefined, fmt.Errorf("invalid client name %s (unknown environment %s)", clientName, parts[1])
}

View File

@ -1,3 +1,32 @@
// Package ekko provides an enhanced Echo web framework wrapper with pre-configured middleware.
//
// This package wraps the Echo web framework with a comprehensive middleware stack including:
// - OpenTelemetry distributed tracing with request context propagation
// - Prometheus metrics collection with per-service subsystems
// - Structured logging with trace ID correlation
// - Security headers (HSTS, content security policy)
// - Gzip compression for response optimization
// - Recovery middleware with detailed error logging
// - HTTP/2 support with H2C (HTTP/2 Cleartext) capability
//
// The package uses functional options pattern for flexible configuration
// and supports graceful shutdown with configurable timeouts. It's designed
// as the standard web service foundation for NTP Pool project services.
//
// Example usage:
//
// ekko, err := ekko.New("myservice",
// ekko.WithPort(8080),
// ekko.WithPrometheus(prometheus.DefaultRegisterer),
// ekko.WithEchoSetup(func(e *echo.Echo) error {
// e.GET("/health", healthHandler)
// return nil
// }),
// )
// if err != nil {
// log.Fatal(err)
// }
// err = ekko.Start(ctx)
package ekko
import (
@ -16,9 +45,29 @@ import (
"go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
"golang.org/x/net/http2"
"golang.org/x/sync/errgroup"
)
// New creates a new Ekko instance with the specified service name and functional options.
// The name parameter is used for OpenTelemetry service identification, Prometheus metrics
// subsystem naming, and server identification headers.
//
// Default configuration includes:
// - 60 second write timeout
// - 30 second read header timeout
// - HTTP/2 support with H2C
// - Standard middleware stack (tracing, metrics, logging, security)
//
// Use functional options to customize behavior:
// - WithPort(): Set server port (required for Start())
// - WithPrometheus(): Enable Prometheus metrics
// - WithEchoSetup(): Configure routes and handlers
// - WithLogFilters(): Filter access logs
// - WithOtelMiddleware(): Custom OpenTelemetry middleware
// - WithWriteTimeout(): Custom write timeout
// - WithReadHeaderTimeout(): Custom read header timeout
// - WithGzipConfig(): Custom gzip compression settings
func New(name string, options ...func(*Ekko)) (*Ekko, error) {
ek := &Ekko{
writeTimeout: 60 * time.Second,
@ -31,13 +80,25 @@ func New(name string, options ...func(*Ekko)) (*Ekko, error) {
return ek, nil
}
// Setup Echo; only intended for testing
// SetupEcho creates and configures an Echo instance without starting the server.
// This method is primarily intended for testing scenarios where you need access
// to the configured Echo instance without starting the HTTP server.
//
// The returned Echo instance includes all configured middleware and routes
// but requires manual server lifecycle management.
func (ek *Ekko) SetupEcho(ctx context.Context) (*echo.Echo, error) {
return ek.setup(ctx)
}
// Setup Echo and start the server. Will return if the http server
// returns or the context is done.
// Start creates the Echo instance and starts the HTTP server with graceful shutdown support.
// The server runs until either an error occurs or the provided context is cancelled.
//
// The server supports HTTP/2 with H2C (HTTP/2 Cleartext) and includes a 5-second
// graceful shutdown timeout when the context is cancelled. Server configuration
// (port, timeouts, middleware) must be set via functional options during New().
//
// Returns an error if server startup fails or if shutdown doesn't complete within
// the timeout period. Returns nil for clean shutdown via context cancellation.
func (ek *Ekko) Start(ctx context.Context) error {
log := logger.Setup()
@ -50,7 +111,8 @@ func (ek *Ekko) Start(ctx context.Context) error {
g.Go(func() error {
e.Server.Addr = fmt.Sprintf(":%d", ek.port)
log.Info("server starting", "port", ek.port)
err := e.Server.ListenAndServe()
// err := e.Server.ListenAndServe()
err := e.StartH2CServer(e.Server.Addr, &http2.Server{})
if err == http.ErrServerClosed {
return nil
}
@ -120,7 +182,13 @@ func (ek *Ekko) setup(ctx context.Context) (*echo.Echo, error) {
e.Use(middleware.Gzip())
}
e.Use(middleware.Secure())
secureConfig := middleware.DefaultSecureConfig
// secureConfig.ContentSecurityPolicy = "default-src *"
secureConfig.ContentSecurityPolicy = ""
secureConfig.HSTSMaxAge = int(time.Hour * 168 * 30 / time.Second)
secureConfig.HSTSPreloadEnabled = true
e.Use(middleware.SecureWithConfig(secureConfig))
e.Use(
func(next echo.HandlerFunc) echo.HandlerFunc {

View File

@ -9,6 +9,9 @@ import (
slogecho "github.com/samber/slog-echo"
)
// Ekko represents an enhanced Echo web server with pre-configured middleware stack.
// It encapsulates server configuration, middleware options, and lifecycle management
// for NTP Pool web services. Use New() with functional options to configure.
type Ekko struct {
name string
prom prometheus.Registerer
@ -22,50 +25,76 @@ type Ekko struct {
readHeaderTimeout time.Duration
}
// RouteFn defines a function type for configuring Echo routes and handlers.
// It receives a configured Echo instance and should register all application
// routes, middleware, and handlers. Return an error to abort server startup.
type RouteFn func(e *echo.Echo) error
// WithPort sets the HTTP server port. This option is required when using Start().
// The port should be available and the process should have permission to bind to it.
func WithPort(port int) func(*Ekko) {
return func(ek *Ekko) {
ek.port = port
}
}
// WithPrometheus enables Prometheus metrics collection using the provided registerer.
// Metrics include HTTP request duration, request count, and response size histograms.
// The service name is used as the metrics subsystem for namespacing.
func WithPrometheus(reg prometheus.Registerer) func(*Ekko) {
return func(ek *Ekko) {
ek.prom = reg
}
}
// WithEchoSetup configures application routes and handlers via a setup function.
// The provided function receives the configured Echo instance after all middleware
// is applied and should register routes, custom middleware, and handlers.
func WithEchoSetup(rfn RouteFn) func(*Ekko) {
return func(ek *Ekko) {
ek.routeFn = rfn
}
}
// WithLogFilters configures access log filtering to reduce log noise.
// Filters can exclude specific paths, methods, or status codes from access logs.
// Useful for excluding health checks, metrics endpoints, and other high-frequency requests.
func WithLogFilters(f []slogecho.Filter) func(*Ekko) {
return func(ek *Ekko) {
ek.logFilters = f
}
}
// WithOtelMiddleware replaces the default OpenTelemetry middleware with a custom implementation.
// The default middleware provides distributed tracing for all requests. Use this option
// when you need custom trace configuration or want to disable tracing entirely.
func WithOtelMiddleware(mw echo.MiddlewareFunc) func(*Ekko) {
return func(ek *Ekko) {
ek.otelmiddleware = mw
}
}
// WithWriteTimeout configures the HTTP server write timeout.
// This is the maximum duration before timing out writes of the response.
// Default is 60 seconds. Should be longer than expected response generation time.
func WithWriteTimeout(t time.Duration) func(*Ekko) {
return func(ek *Ekko) {
ek.writeTimeout = t
}
}
// WithReadHeaderTimeout configures the HTTP server read header timeout.
// This is the amount of time allowed to read request headers.
// Default is 30 seconds. Should be sufficient for slow clients and large headers.
func WithReadHeaderTimeout(t time.Duration) func(*Ekko) {
return func(ek *Ekko) {
ek.readHeaderTimeout = t
}
}
// WithGzipConfig provides custom gzip compression configuration.
// By default, gzip compression is enabled with standard settings.
// Use this option to customize compression level, skip patterns, or disable compression.
func WithGzipConfig(gzipConfig *middleware.GzipConfig) func(*Ekko) {
return func(ek *Ekko) {
ek.gzipConfig = gzipConfig

4
go.mod
View File

@ -1,8 +1,6 @@
module go.ntppool.org/common
go 1.23
toolchain go1.23.4
go 1.23.5
require (
github.com/abh/certman v0.4.0

View File

@ -1,3 +1,9 @@
// Package health provides a standalone HTTP server for health checks.
//
// This package implements a simple health check server that can be used
// to expose health status endpoints for monitoring and load balancing.
// It supports custom health check handlers and provides structured logging
// with graceful shutdown capabilities.
package health
import (
@ -11,11 +17,19 @@ import (
"golang.org/x/sync/errgroup"
)
// Server is a standalone HTTP server dedicated to health checks.
// It runs separately from the main application server to ensure health
// checks remain available even if the main server is experiencing issues.
//
// The server includes built-in timeouts, graceful shutdown, and structured
// logging for monitoring and debugging health check behavior.
type Server struct {
log *slog.Logger
healthFn http.HandlerFunc
}
// NewServer creates a new health check server with the specified health handler.
// If healthFn is nil, a default handler that returns HTTP 200 "ok" is used.
func NewServer(healthFn http.HandlerFunc) *Server {
if healthFn == nil {
healthFn = basicHealth
@ -27,10 +41,13 @@ func NewServer(healthFn http.HandlerFunc) *Server {
return srv
}
// SetLogger replaces the default logger with a custom one.
func (srv *Server) SetLogger(log *slog.Logger) {
srv.log = log
}
// Listen starts the health server on the specified port and blocks until ctx is cancelled.
// The server exposes the health handler at "/__health" with graceful shutdown support.
func (srv *Server) Listen(ctx context.Context, port int) error {
srv.log.Info("starting health listener", "port", port)
@ -59,11 +76,10 @@ func (srv *Server) Listen(ctx context.Context, port int) error {
<-ctx.Done()
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
g.Go(func() error {
if err := hsrv.Shutdown(ctx); err != nil {
shCtx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
if err := hsrv.Shutdown(shCtx); err != nil {
srv.log.Error("health check server shutdown failed", "err", err)
return err
}
@ -73,8 +89,7 @@ func (srv *Server) Listen(ctx context.Context, port int) error {
return g.Wait()
}
// HealthCheckListener runs simple http server on the specified port for
// health check probes
// HealthCheckListener runs a simple HTTP server on the specified port for health check probes.
func HealthCheckListener(ctx context.Context, port int, log *slog.Logger) error {
srv := NewServer(nil)
srv.SetLogger(log)

View File

@ -8,7 +8,6 @@ import (
)
func TestHealthHandler(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/__health", nil)
w := httptest.NewRecorder()

View File

@ -1,3 +1,32 @@
// Package kafconn provides a Kafka client wrapper with TLS support for secure log streaming.
//
// This package handles Kafka connections with mutual TLS authentication for the NTP Pool
// project's log streaming infrastructure. It provides factories for creating Kafka readers
// and writers with automatic broker discovery, TLS configuration, and connection management.
//
// The package is designed specifically for the NTP Pool pipeline infrastructure and includes
// hardcoded bootstrap servers and group configurations. It uses certman for automatic
// certificate renewal and provides compression and batching optimizations.
//
// Key features:
// - Mutual TLS authentication with automatic certificate renewal
// - Broker discovery and connection pooling
// - Reader and writer factory methods with optimized configurations
// - LZ4 compression for efficient data transfer
// - Configurable batch sizes and load balancing
//
// Example usage:
//
// tlsSetup := kafconn.TLSSetup{
// CA: "/path/to/ca.pem",
// Cert: "/path/to/client.pem",
// Key: "/path/to/client.key",
// }
// kafka, err := kafconn.NewKafka(ctx, tlsSetup)
// if err != nil {
// log.Fatal(err)
// }
// writer, err := kafka.NewWriter("logs")
package kafconn
import (
@ -24,12 +53,17 @@ const (
// kafkaMinBatchSize = 1000
)
// TLSSetup contains file paths for TLS certificate configuration.
// All fields are required for establishing secure Kafka connections.
type TLSSetup struct {
CA string
Key string
Cert string
CA string // Path to CA certificate file for server verification
Key string // Path to client private key file
Cert string // Path to client certificate file
}
// Kafka represents a configured Kafka client with TLS support.
// It manages connections, brokers, and provides factory methods for readers and writers.
// The client handles broker discovery, connection pooling, and TLS configuration automatically.
type Kafka struct {
tls TLSSetup
@ -42,11 +76,9 @@ type Kafka struct {
l *log.Logger
// wr *kafka.Writer
}
func (k *Kafka) tlsConfig() (*tls.Config, error) {
cm, err := certman.New(k.tls.Cert, k.tls.Key)
if err != nil {
return nil, err
@ -118,6 +150,19 @@ func (k *Kafka) kafkaTransport(ctx context.Context) (*kafka.Transport, error) {
return transport, nil
}
// NewKafka creates a new Kafka client with TLS configuration and establishes initial connections.
// It performs broker discovery, validates TLS certificates, and prepares the client for creating
// readers and writers.
//
// The function validates TLS configuration, establishes a connection to the bootstrap server,
// discovers all available brokers, and configures transport layers for optimal performance.
//
// Parameters:
// - ctx: Context for connection establishment and timeouts
// - tls: TLS configuration with paths to CA, certificate, and key files
//
// Returns a configured Kafka client ready for creating readers and writers, or an error
// if TLS setup fails, connection cannot be established, or broker discovery fails.
func NewKafka(ctx context.Context, tls TLSSetup) (*Kafka, error) {
l := log.New(os.Stdout, "kafka: ", log.Ldate|log.Ltime|log.LUTC|log.Lmsgprefix|log.Lmicroseconds)
@ -173,6 +218,12 @@ func NewKafka(ctx context.Context, tls TLSSetup) (*Kafka, error) {
return k, nil
}
// NewReader creates a new Kafka reader with the client's broker list and TLS configuration.
// The provided config is enhanced with the discovered brokers and configured dialer.
// The reader supports automatic offset management, consumer group coordination, and reconnection.
//
// The caller should configure the reader's Topic, GroupID, and other consumer-specific settings
// in the provided config. The client automatically sets Brokers and Dialer fields.
func (k *Kafka) NewReader(config kafka.ReaderConfig) *kafka.Reader {
config.Brokers = k.brokerAddrs()
config.Dialer = k.dialer
@ -188,8 +239,17 @@ func (k *Kafka) brokerAddrs() []string {
return addrs
}
// NewWriter creates a new Kafka writer for the specified topic with optimized configuration.
// The writer uses LZ4 compression, least-bytes load balancing, and batching for performance.
//
// Configuration includes:
// - Batch size: 2000 messages for efficient throughput
// - Compression: LZ4 for fast compression with good ratios
// - Balancer: LeastBytes for optimal partition distribution
// - Transport: TLS-configured transport with connection pooling
//
// The writer is ready for immediate use and handles connection management automatically.
func (k *Kafka) NewWriter(topic string) (*kafka.Writer, error) {
// https://pkg.go.dev/github.com/segmentio/kafka-go#Writer
w := &kafka.Writer{
Addr: kafka.TCP(k.brokerAddrs()...),
@ -205,6 +265,12 @@ func (k *Kafka) NewWriter(topic string) (*kafka.Writer, error) {
return w, nil
}
// CheckPartitions verifies that the Kafka connection can read partition metadata.
// This method is useful for health checks and connection validation.
//
// Returns an error if partition metadata cannot be retrieved, which typically
// indicates connection problems, authentication failures, or broker unavailability.
// Logs a warning if no partitions are available but does not return an error.
func (k *Kafka) CheckPartitions() error {
partitions, err := k.conn.ReadPartitions()
if err != nil {

View File

@ -17,7 +17,6 @@ type logfmt struct {
}
func newLogFmtHandler(next slog.Handler) slog.Handler {
buf := bytes.NewBuffer([]byte{})
h := &logfmt{

View File

@ -9,7 +9,6 @@ import (
)
func TestLogFmt(t *testing.T) {
var buf bytes.Buffer
jsonh := slog.NewJSONHandler(&buf, nil)
h := newLogFmtHandler(jsonh)
@ -39,5 +38,4 @@ func TestLogFmt(t *testing.T) {
t.Log("didn't find message in output")
t.Fail()
}
}

View File

@ -1,3 +1,25 @@
// Package logger provides structured logging with OpenTelemetry trace integration.
//
// This package offers multiple logging configurations for different deployment scenarios:
// - Text logging to stderr with optional timestamp removal for systemd
// - OTLP (OpenTelemetry Protocol) logging for observability pipelines
// - Multi-logger setup that outputs to both text and OTLP simultaneously
// - Context-aware logging with trace ID correlation
//
// The package automatically detects systemd environments and adjusts timestamp handling
// accordingly. It supports debug level configuration via environment variables and
// provides compatibility bridges for legacy logging interfaces.
//
// Key features:
// - Automatic OpenTelemetry trace and span ID inclusion in log entries
// - Configurable log levels via DEBUG environment variable (with optional prefix)
// - Systemd-compatible output (no timestamps when INVOCATION_ID is present)
// - Thread-safe logger setup with sync.Once protection
// - Context propagation for request-scoped logging
//
// Environment variables:
// - DEBUG: Enable debug level logging (configurable prefix via ConfigPrefix)
// - INVOCATION_ID: Systemd detection for timestamp handling
package logger
import (
@ -13,19 +35,26 @@ import (
"go.opentelemetry.io/contrib/bridges/otelslog"
)
// ConfigPrefix allows customizing the environment variable prefix for configuration.
// When set, environment variables like DEBUG become {ConfigPrefix}_DEBUG.
// This enables multiple services to have independent logging configuration.
var ConfigPrefix = ""
var textLogger *slog.Logger
var otlpLogger *slog.Logger
var multiLogger *slog.Logger
var (
textLogger *slog.Logger
otlpLogger *slog.Logger
multiLogger *slog.Logger
)
var setupText sync.Once // this sets the default
var setupOtlp sync.Once // this never sets the default
var setupMulti sync.Once // this sets the default, and will always run after the others
var mu sync.Mutex
var (
setupText sync.Once // this sets the default
setupOtlp sync.Once // this never sets the default
setupMulti sync.Once // this sets the default, and will always run after the others
mu sync.Mutex
)
func setupStdErrHandler() slog.Handler {
var programLevel = new(slog.LevelVar) // Info by default
programLevel := new(slog.LevelVar) // Info by default
envVar := "DEBUG"
if len(ConfigPrefix) > 0 {
@ -63,10 +92,15 @@ func setupOtlpLogger() *slog.Logger {
return otlpLogger
}
// SetupMultiLogger will setup and make default a logger that
// logs as described in Setup() as well as an OLTP logger.
// The "multi logger" is made the default the first time
// this function is called
// SetupMultiLogger creates a logger that outputs to both text (stderr) and OTLP simultaneously.
// This is useful for services that need both human-readable logs and structured observability data.
//
// The multi-logger combines:
// - Text handler: Stderr output with OpenTelemetry trace integration
// - OTLP handler: Structured logs sent via OpenTelemetry Protocol
//
// On first call, this logger becomes the default logger returned by Setup().
// The function is thread-safe and uses sync.Once to ensure single initialization.
func SetupMultiLogger() *slog.Logger {
setupMulti.Do(func() {
textHandler := Setup().Handler()
@ -85,28 +119,38 @@ func SetupMultiLogger() *slog.Logger {
return multiLogger
}
// SetupOLTP configures and returns a logger sending logs
// via OpenTelemetry (configured via the tracing package).
// SetupOLTP creates a logger that sends structured logs via OpenTelemetry Protocol.
// This logger is designed for observability pipelines and log aggregation systems.
//
// This was made to work with Loki + Grafana that makes it
// hard to view the log attributes in the UI, so the log
// message is formatted similarly to the text logger. The
// attributes are duplicated as OLTP attributes in the
// log messages. https://github.com/grafana/loki/issues/14788
// The OTLP logger formats log messages similarly to the text logger for better
// compatibility with Loki + Grafana, while still providing structured attributes.
// Log attributes are available both in the message format and as OTLP attributes.
//
// This logger does not become the default logger and must be used explicitly.
// It requires OpenTelemetry tracing configuration to be set up via the tracing package.
//
// See: https://github.com/grafana/loki/issues/14788 for formatting rationale.
func SetupOLTP() *slog.Logger {
return setupOtlpLogger()
}
// Setup returns an slog.Logger configured for text formatting
// to stderr.
// OpenTelemetry trace_id and span_id's are logged as attributes
// when available.
// When the application is running under systemd timestamps are
// omitted. On first call the slog default logger is set to this
// logger as well.
// Setup creates and returns the standard text logger for the application.
// This is the primary logging function that most applications should use.
//
// If SetupMultiLogger has been called Setup() will return
// the "multi logger"
// Features:
// - Text formatting to stderr with human-readable output
// - Automatic OpenTelemetry trace_id and span_id inclusion when available
// - Systemd compatibility: omits timestamps when INVOCATION_ID environment variable is present
// - Debug level support via DEBUG environment variable (respects ConfigPrefix)
// - Thread-safe initialization with sync.Once
//
// On first call, this logger becomes the slog default logger. If SetupMultiLogger()
// has been called previously, Setup() returns the multi-logger instead of the text logger.
//
// The logger automatically detects execution context:
// - Systemd: Removes timestamps (systemd adds its own)
// - Debug mode: Enables debug level logging based on environment variables
// - OpenTelemetry: Includes trace correlation when tracing is active
func Setup() *slog.Logger {
setupText.Do(func() {
h := setupStdErrHandler()
@ -125,15 +169,33 @@ func Setup() *slog.Logger {
type loggerKey struct{}
// NewContext adds the logger to the context. Use this
// to for example make a request specific logger available
// to other functions through the context
// NewContext stores a logger in the context for request-scoped logging.
// This enables passing request-specific loggers (e.g., with request IDs,
// user context, or other correlation data) through the call stack.
//
// Use this to create context-aware logging where different parts of the
// application can access the same enriched logger instance.
//
// Example:
//
// logger := slog.With("request_id", requestID)
// ctx := logger.NewContext(ctx, logger)
// // Pass ctx to downstream functions
func NewContext(ctx context.Context, l *slog.Logger) context.Context {
return context.WithValue(ctx, loggerKey{}, l)
}
// FromContext retrieves a logger from the context. If there is none,
// it returns the default logger
// FromContext retrieves a logger from the context.
// If no logger is stored in the context, it returns the default logger from Setup().
//
// This function provides a safe way to access context-scoped loggers without
// needing to check for nil values. It ensures that logging is always available,
// falling back to the application's standard logger configuration.
//
// Example:
//
// log := logger.FromContext(ctx)
// log.Info("processing request") // Uses context logger or default
func FromContext(ctx context.Context) *slog.Logger {
if l, ok := ctx.Value(loggerKey{}).(*slog.Logger); ok {
return l

View File

@ -5,12 +5,24 @@ import (
"log/slog"
)
// stdLoggerish provides a bridge between legacy log interfaces and slog.
// It implements common logging methods (Println, Printf, Fatalf) that
// delegate to structured logging with a consistent key prefix.
type stdLoggerish struct {
key string
log *slog.Logger
f func(string, ...any)
key string // Prefix key for all log messages
log *slog.Logger // Underlying structured logger
f func(string, ...any) // Log function (Info or Debug level)
}
// NewStdLog creates a legacy-compatible logger that bridges to structured logging.
// This is useful for third-party libraries that expect a standard log.Logger interface.
//
// Parameters:
// - key: Prefix added to all log messages for identification
// - debug: If true, logs at debug level; otherwise logs at info level
// - log: Underlying slog.Logger (uses Setup() if nil)
//
// The returned logger implements Println, Printf, and Fatalf methods.
func NewStdLog(key string, debug bool, log *slog.Logger) *stdLoggerish {
if log == nil {
log = Setup()
@ -27,15 +39,19 @@ func NewStdLog(key string, debug bool, log *slog.Logger) *stdLoggerish {
return sl
}
func (l stdLoggerish) Println(msg ...interface{}) {
// Println logs the arguments using the configured log level with the instance key.
func (l stdLoggerish) Println(msg ...any) {
l.f(l.key, "msg", msg)
}
func (l stdLoggerish) Printf(msg string, args ...interface{}) {
// Printf logs a formatted message using the configured log level with the instance key.
func (l stdLoggerish) Printf(msg string, args ...any) {
l.f(l.key, "msg", fmt.Sprintf(msg, args...))
}
func (l stdLoggerish) Fatalf(msg string, args ...interface{}) {
// Fatalf logs a formatted error message and panics.
// Note: This implementation panics instead of calling os.Exit for testability.
func (l stdLoggerish) Fatalf(msg string, args ...any) {
l.log.Error(l.key, "msg", fmt.Sprintf(msg, args...))
panic("fatal error") // todo: does this make sense at all?
}

View File

@ -1,17 +0,0 @@
package logger
type Error struct {
Msg string
Data []any
}
func NewError(msg string, data ...any) *Error {
return &Error{
Msg: msg,
Data: data,
}
}
func (e *Error) Error() string {
return "not implemented"
}

View File

@ -1,3 +1,8 @@
// Package metricsserver provides a standalone HTTP server for exposing Prometheus metrics.
//
// This package implements a dedicated metrics server that exposes application metrics
// via HTTP. It uses a custom Prometheus registry to avoid conflicts with other metric
// collectors and provides graceful shutdown capabilities.
package metricsserver
import (
@ -13,10 +18,13 @@ import (
"go.ntppool.org/common/logger"
)
// Metrics provides a custom Prometheus registry and HTTP handlers for metrics exposure.
// It isolates application metrics from the default global registry.
type Metrics struct {
r *prometheus.Registry
}
// New creates a new Metrics instance with a custom Prometheus registry.
func New() *Metrics {
r := prometheus.NewRegistry()
@ -27,12 +35,14 @@ func New() *Metrics {
return m
}
// Registry returns the custom Prometheus registry.
// Use this to register your application's metrics collectors.
func (m *Metrics) Registry() *prometheus.Registry {
return m.r
}
// Handler returns an HTTP handler for the /metrics endpoint with OpenMetrics support.
func (m *Metrics) Handler() http.Handler {
log := logger.NewStdLog("prom http", false, nil)
return promhttp.HandlerFor(m.r, promhttp.HandlerOpts{
@ -42,11 +52,9 @@ func (m *Metrics) Handler() http.Handler {
})
}
// ListenAndServe starts a goroutine with a server running on
// the specified port. The server will shutdown and return when
// the provided context is done
// ListenAndServe starts a metrics server on the specified port and blocks until ctx is done.
// The server exposes the metrics handler and shuts down gracefully when the context is cancelled.
func (m *Metrics) ListenAndServe(ctx context.Context, port int) error {
log := logger.Setup()
srv := &http.Server{

View File

@ -0,0 +1,242 @@
package metricsserver
import (
"context"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/prometheus/client_golang/prometheus"
)
func TestNew(t *testing.T) {
metrics := New()
if metrics == nil {
t.Fatal("New returned nil")
}
if metrics.r == nil {
t.Error("metrics registry is nil")
}
}
func TestRegistry(t *testing.T) {
metrics := New()
registry := metrics.Registry()
if registry == nil {
t.Fatal("Registry() returned nil")
}
if registry != metrics.r {
t.Error("Registry() did not return the metrics registry")
}
// Test that we can register a metric
counter := prometheus.NewCounter(prometheus.CounterOpts{
Name: "test_counter",
Help: "A test counter",
})
err := registry.Register(counter)
if err != nil {
t.Errorf("failed to register metric: %v", err)
}
// Test that the metric is registered
metricFamilies, err := registry.Gather()
if err != nil {
t.Errorf("failed to gather metrics: %v", err)
}
found := false
for _, mf := range metricFamilies {
if mf.GetName() == "test_counter" {
found = true
break
}
}
if !found {
t.Error("registered metric not found in registry")
}
}
func TestHandler(t *testing.T) {
metrics := New()
// Register a test metric
counter := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "test_requests_total",
Help: "Total number of test requests",
},
[]string{"method"},
)
metrics.Registry().MustRegister(counter)
counter.WithLabelValues("GET").Inc()
// Test the handler
handler := metrics.Handler()
if handler == nil {
t.Fatal("Handler() returned nil")
}
// Create a test request
req := httptest.NewRequest("GET", "/metrics", nil)
recorder := httptest.NewRecorder()
// Call the handler
handler.ServeHTTP(recorder, req)
// Check response
resp := recorder.Result()
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("expected status 200, got %d", resp.StatusCode)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
t.Fatalf("failed to read response body: %v", err)
}
bodyStr := string(body)
// Check for our test metric
if !strings.Contains(bodyStr, "test_requests_total") {
t.Error("test metric not found in metrics output")
}
// Check for OpenMetrics format indicators
if !strings.Contains(bodyStr, "# TYPE") {
t.Error("metrics output missing TYPE comments")
}
}
func TestListenAndServe(t *testing.T) {
metrics := New()
// Register a test metric
counter := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "test_requests_total",
Help: "Total number of test requests",
},
[]string{"method"},
)
metrics.Registry().MustRegister(counter)
counter.WithLabelValues("GET").Inc()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Start server in a goroutine
errCh := make(chan error, 1)
go func() {
// Use a high port number to avoid conflicts
errCh <- metrics.ListenAndServe(ctx, 9999)
}()
// Give the server a moment to start
time.Sleep(100 * time.Millisecond)
// Test metrics endpoint
resp, err := http.Get("http://localhost:9999/metrics")
if err != nil {
t.Fatalf("failed to GET /metrics: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("expected status 200, got %d", resp.StatusCode)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
t.Fatalf("failed to read response body: %v", err)
}
bodyStr := string(body)
// Check for our test metric
if !strings.Contains(bodyStr, "test_requests_total") {
t.Error("test metric not found in metrics output")
}
// Cancel context to stop server
cancel()
// Wait for server to stop
select {
case err := <-errCh:
if err != nil {
t.Errorf("server returned error: %v", err)
}
case <-time.After(5 * time.Second):
t.Error("server did not stop within timeout")
}
}
func TestListenAndServeContextCancellation(t *testing.T) {
metrics := New()
ctx, cancel := context.WithCancel(context.Background())
// Start server
errCh := make(chan error, 1)
go func() {
errCh <- metrics.ListenAndServe(ctx, 9998)
}()
// Give server time to start
time.Sleep(100 * time.Millisecond)
// Cancel context
cancel()
// Server should stop gracefully
select {
case err := <-errCh:
if err != nil {
t.Errorf("server returned error on graceful shutdown: %v", err)
}
case <-time.After(5 * time.Second):
t.Error("server did not stop within timeout after context cancellation")
}
}
// Benchmark the metrics handler response time
func BenchmarkMetricsHandler(b *testing.B) {
metrics := New()
// Register some test metrics
for i := 0; i < 10; i++ {
counter := prometheus.NewCounter(prometheus.CounterOpts{
Name: fmt.Sprintf("bench_counter_%d", i),
Help: "A benchmark counter",
})
metrics.Registry().MustRegister(counter)
counter.Add(float64(i * 100))
}
handler := metrics.Handler()
b.ResetTimer()
for i := 0; i < b.N; i++ {
req := httptest.NewRequest("GET", "/metrics", nil)
recorder := httptest.NewRecorder()
handler.ServeHTTP(recorder, req)
if recorder.Code != http.StatusOK {
b.Fatalf("unexpected status code: %d", recorder.Code)
}
}
}

View File

@ -2,7 +2,7 @@
set -euo pipefail
go install github.com/goreleaser/goreleaser/v2@v2.5.0
go install github.com/goreleaser/goreleaser/v2@v2.8.2
if [ ! -z "${harbor_username:-}" ]; then
DOCKER_FILE=~/.docker/config.json

View File

@ -1,3 +1,4 @@
// Package timeutil provides JSON-serializable time utilities.
package timeutil
import (
@ -6,16 +7,39 @@ import (
"time"
)
// Duration is a wrapper around time.Duration that supports JSON marshaling/unmarshaling.
//
// When marshaling to JSON, it outputs the duration as a string using time.Duration.String().
// When unmarshaling from JSON, it accepts both:
// - String values that can be parsed by time.ParseDuration (e.g., "30s", "5m", "1h30m")
// - Numeric values that represent nanoseconds as a float64
//
// This makes it compatible with configuration files and APIs that need to represent
// durations in a human-readable format.
//
// Example usage:
//
// type Config struct {
// Timeout timeutil.Duration `json:"timeout"`
// }
//
// // JSON: {"timeout": "30s"}
// // or: {"timeout": 30000000000}
type Duration struct {
time.Duration
}
// MarshalJSON implements json.Marshaler.
// It marshals the duration as a string using time.Duration.String().
func (d Duration) MarshalJSON() ([]byte, error) {
return json.Marshal(time.Duration(d.Duration).String())
}
// UnmarshalJSON implements json.Unmarshaler.
// It accepts both string values (parsed via time.ParseDuration) and
// numeric values (interpreted as nanoseconds).
func (d *Duration) UnmarshalJSON(b []byte) error {
var v interface{}
var v any
if err := json.Unmarshal(b, &v); err != nil {
return err
}

View File

@ -18,5 +18,4 @@ func TestDuration(t *testing.T) {
if foo.Foo.Seconds() != 30 {
t.Fatalf("parsed time.Duration wasn't 30 seconds: %s", foo.Foo)
}
}

View File

@ -1,3 +1,36 @@
// Package tracing provides OpenTelemetry distributed tracing setup with OTLP export support.
//
// This package handles the complete OpenTelemetry SDK initialization including:
// - Trace provider configuration with batching and resource detection
// - Log provider setup for structured log export via OTLP
// - Automatic resource discovery (service name, version, host, container, process info)
// - Support for both gRPC and HTTP OTLP exporters with TLS configuration
// - Propagation context setup for distributed tracing across services
// - Graceful shutdown handling for all telemetry components
//
// The package supports various deployment scenarios:
// - Development: Local OTLP collectors or observability backends
// - Production: Secure OTLP export with mutual TLS authentication
// - Container environments: Automatic container and Kubernetes resource detection
//
// Configuration is primarily handled via standard OpenTelemetry environment variables:
// - OTEL_SERVICE_NAME: Service identification
// - OTEL_EXPORTER_OTLP_PROTOCOL: Protocol selection (grpc, http/protobuf)
// - OTEL_TRACES_EXPORTER: Exporter type (otlp, autoexport)
// - OTEL_RESOURCE_ATTRIBUTES: Additional resource attributes
//
// Example usage:
//
// cfg := &tracing.TracerConfig{
// ServiceName: "my-service",
// Environment: "production",
// Endpoint: "https://otlp.example.com:4317",
// }
// shutdown, err := tracing.InitTracer(ctx, cfg)
// if err != nil {
// log.Fatal(err)
// }
// defer shutdown(ctx)
package tracing
// todo, review:
@ -43,34 +76,68 @@ var errInvalidOTLPProtocol = errors.New("invalid OTLP protocol - should be one o
// https://github.com/open-telemetry/opentelemetry-go/blob/main/exporters/otlp/otlptrace/otlptracehttp/example_test.go
// TpShutdownFunc represents a function that gracefully shuts down telemetry providers.
// It should be called during application shutdown to ensure all telemetry data is flushed
// and exporters are properly closed. The context can be used to set shutdown timeouts.
type TpShutdownFunc func(ctx context.Context) error
// Tracer returns the configured OpenTelemetry tracer for the NTP Pool project.
// This tracer should be used for creating spans and distributed tracing throughout
// the application. It uses the global tracer provider set up by InitTracer/SetupSDK.
func Tracer() trace.Tracer {
traceProvider := otel.GetTracerProvider()
return traceProvider.Tracer("ntppool-tracer")
}
// Start creates a new span with the given name and options using the configured tracer.
// This is a convenience function that wraps the standard OpenTelemetry span creation.
// It returns a new context containing the span and the span itself for further configuration.
//
// The returned context should be used for downstream operations to maintain trace correlation.
func Start(ctx context.Context, spanName string, opts ...trace.SpanStartOption) (context.Context, trace.Span) {
return Tracer().Start(ctx, spanName, opts...)
}
// GetClientCertificate defines a function type for providing client certificates for mutual TLS.
// This is used when exporting telemetry data to secured OTLP endpoints that require
// client certificate authentication.
type GetClientCertificate func(*tls.CertificateRequestInfo) (*tls.Certificate, error)
// TracerConfig provides configuration options for OpenTelemetry tracing setup.
// It supplements standard OpenTelemetry environment variables with additional
// NTP Pool-specific configuration including TLS settings for secure OTLP export.
type TracerConfig struct {
ServiceName string
Environment string
Endpoint string
EndpointURL string
ServiceName string // Service name for resource identification (overrides OTEL_SERVICE_NAME)
Environment string // Deployment environment (development, staging, production)
Endpoint string // OTLP endpoint hostname/port (e.g., "otlp.example.com:4317")
EndpointURL string // Complete OTLP endpoint URL (e.g., "https://otlp.example.com:4317/v1/traces")
CertificateProvider GetClientCertificate
RootCAs *x509.CertPool
CertificateProvider GetClientCertificate // Client certificate provider for mutual TLS
RootCAs *x509.CertPool // CA certificate pool for server verification
}
// InitTracer initializes the OpenTelemetry SDK with the provided configuration.
// This is the main entry point for setting up distributed tracing in applications.
//
// The function configures trace and log providers, sets up OTLP exporters,
// and returns a shutdown function that must be called during application termination.
//
// Returns a shutdown function and an error. The shutdown function should be called
// with a context that has an appropriate timeout for graceful shutdown.
func InitTracer(ctx context.Context, cfg *TracerConfig) (TpShutdownFunc, error) {
// todo: setup environment from cfg
return SetupSDK(ctx, cfg)
}
// SetupSDK performs the complete OpenTelemetry SDK initialization including resource
// discovery, exporter configuration, provider setup, and shutdown function creation.
//
// The function automatically discovers system resources (service info, host, container,
// process details) and configures both trace and log exporters. It supports multiple
// OTLP protocols (gRPC, HTTP) and handles TLS configuration for secure deployments.
//
// The returned shutdown function coordinates graceful shutdown of all telemetry
// components in the reverse order of their initialization.
func SetupSDK(ctx context.Context, cfg *TracerConfig) (shutdown TpShutdownFunc, err error) {
if cfg == nil {
cfg = &TracerConfig{}
@ -196,7 +263,6 @@ func SetupSDK(ctx context.Context, cfg *TracerConfig) (shutdown TpShutdownFunc,
}
func newOLTPExporter(ctx context.Context, cfg *TracerConfig) (sdktrace.SpanExporter, error) {
log := logger.Setup()
var tlsConfig *tls.Config
@ -238,7 +304,7 @@ func newOLTPExporter(ctx context.Context, cfg *TracerConfig) (sdktrace.SpanExpor
}
client = otlptracegrpc.NewClient(opts...)
case "http/protobuf":
case "http/protobuf", "http/json":
opts := []otlptracehttp.Option{
otlptracehttp.WithCompression(otlptracehttp.GzipCompression),
}

View File

@ -7,7 +7,6 @@ import (
)
func TestInit(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
@ -18,5 +17,4 @@ func TestInit(t *testing.T) {
t.FailNow()
}
defer shutdownFn(ctx)
}

View File

@ -1,3 +1,17 @@
// Package types provides shared data structures for the NTP Pool project.
//
// This package contains common types used across different NTP Pool services
// for data exchange, logging, and database operations. The types are designed
// to support JSON serialization for API responses and SQL database storage
// with automatic marshaling/unmarshaling.
//
// Current types include:
// - LogScoreAttributes: NTP server scoring metadata for monitoring and analysis
//
// All types implement appropriate interfaces for:
// - JSON serialization (json.Marshaler/json.Unmarshaler)
// - SQL database storage (database/sql/driver.Valuer/sql.Scanner)
// - String representation for logging and debugging
package types
import (
@ -6,17 +20,26 @@ import (
"errors"
)
// LogScoreAttributes contains metadata about NTP server scoring and monitoring results.
// This structure captures both NTP protocol-specific information (leap, stratum) and
// operational data (errors, warnings, response status) for analysis and alerting.
//
// The type supports JSON serialization for API responses and database storage
// via the database/sql/driver interfaces. Fields use omitempty tags to minimize
// JSON payload size when values are at their zero state.
type LogScoreAttributes struct {
Leap int8 `json:"leap,omitempty"`
Stratum int8 `json:"stratum,omitempty"`
NoResponse bool `json:"no_response,omitempty"`
Error string `json:"error,omitempty"`
Warning string `json:"warning,omitempty"`
Leap int8 `json:"leap,omitempty"` // NTP leap indicator (0=no warning, 1=+1s, 2=-1s, 3=unsynchronized)
Stratum int8 `json:"stratum,omitempty"` // NTP stratum level (1=primary, 2-15=secondary, 16=unsynchronized)
NoResponse bool `json:"no_response,omitempty"` // True if server failed to respond to NTP queries
Error string `json:"error,omitempty"` // Error message if scoring failed
Warning string `json:"warning,omitempty"` // Warning message for non-fatal issues
FromLSID int `json:"from_ls_id,omitempty"`
FromSSID int `json:"from_ss_id,omitempty"`
FromLSID int `json:"from_ls_id,omitempty"` // Source log server ID for traceability
FromSSID int `json:"from_ss_id,omitempty"` // Source scoring system ID for traceability
}
// String returns a JSON representation of the LogScoreAttributes for logging and debugging.
// Returns an empty string if JSON marshaling fails.
func (lsa *LogScoreAttributes) String() string {
b, err := json.Marshal(lsa)
if err != nil {
@ -25,11 +48,18 @@ func (lsa *LogScoreAttributes) String() string {
return string(b)
}
// Value implements the database/sql/driver.Valuer interface for database storage.
// It serializes the LogScoreAttributes to JSON for storage in SQL databases.
// Returns the JSON bytes or an error if marshaling fails.
func (lsa *LogScoreAttributes) Value() (driver.Value, error) {
return json.Marshal(lsa)
}
func (lsa *LogScoreAttributes) Scan(value interface{}) error {
// Scan implements the database/sql.Scanner interface for reading from SQL databases.
// It deserializes JSON data from the database back into LogScoreAttributes.
// Supports both []byte and string input types, with nil values treated as no-op.
// Returns an error if the input type is unsupported or JSON unmarshaling fails.
func (lsa *LogScoreAttributes) Scan(value any) error {
var source []byte
_t := LogScoreAttributes{}

View File

@ -1,48 +1,44 @@
// Package ulid provides thread-safe ULID (Universally Unique Lexicographically Sortable Identifier) generation.
//
// ULIDs are 128-bit identifiers that are lexicographically sortable and contain
// a timestamp component. This package uses cryptographically secure random
// generation optimized for simplicity and performance in concurrent environments.
package ulid
import (
cryptorand "crypto/rand"
"encoding/binary"
"io"
mathrand "math/rand"
"os"
"sync"
"time"
oklid "github.com/oklog/ulid/v2"
"go.ntppool.org/common/logger"
)
var monotonicPool = sync.Pool{
New: func() interface{} {
log := logger.Setup()
var seed int64
err := binary.Read(cryptorand.Reader, binary.BigEndian, &seed)
if err != nil {
log.Error("crypto/rand error", "err", err)
os.Exit(10)
}
rand := mathrand.New(mathrand.NewSource(seed))
inc := uint64(mathrand.Int63())
// log.Printf("seed: %d", seed)
// log.Printf("inc: %d", inc)
// inc = inc & ^uint64(1<<63) // only want 63 bits
mono := oklid.Monotonic(rand, inc)
return mono
},
}
// MakeULID generates a new ULID with the specified timestamp using cryptographically secure randomness.
// The function is thread-safe and optimized for high-concurrency environments.
//
// This implementation prioritizes simplicity and performance over strict monotonicity within
// the same millisecond. Each ULID is guaranteed to be unique and lexicographically sortable
// across different timestamps.
//
// Returns a pointer to the generated ULID or an error if generation fails.
// Generation should only fail under extreme circumstances (entropy exhaustion).
func MakeULID(t time.Time) (*oklid.ULID, error) {
mono := monotonicPool.Get().(io.Reader)
id, err := oklid.New(oklid.Timestamp(t), mono)
id, err := oklid.New(oklid.Timestamp(t), cryptorand.Reader)
if err != nil {
return nil, err
}
return &id, nil
}
// Make generates a new ULID with the current timestamp using cryptographically secure randomness.
// This is a convenience function equivalent to MakeULID(time.Now()).
//
// The function is thread-safe and optimized for high-concurrency environments.
//
// Returns a pointer to the generated ULID or an error if generation fails.
// Generation should only fail under extreme circumstances (entropy exhaustion).
func Make() (*oklid.ULID, error) {
id, err := oklid.New(oklid.Now(), cryptorand.Reader)
if err != nil {
return nil, err
}

View File

@ -1,25 +1,336 @@
package ulid
import (
cryptorand "crypto/rand"
"sort"
"sync"
"testing"
"time"
oklid "github.com/oklog/ulid/v2"
)
func TestULID(t *testing.T) {
func TestMakeULID(t *testing.T) {
tm := time.Now()
ul1, err := MakeULID(tm)
if err != nil {
t.Logf("makeULID failed: %s", err)
t.Fail()
t.Fatalf("MakeULID failed: %s", err)
}
ul2, err := MakeULID(tm)
if err != nil {
t.Logf("MakeULID failed: %s", err)
t.Fail()
t.Fatalf("MakeULID failed: %s", err)
}
if ul1 == nil || ul2 == nil {
t.Fatal("MakeULID returned nil ULID")
}
if ul1.String() == ul2.String() {
t.Logf("ul1 and ul2 got the same string: %s", ul1.String())
t.Fail()
t.Errorf("ul1 and ul2 should be different: %s", ul1.String())
}
// Verify they have the same timestamp
if ul1.Time() != ul2.Time() {
t.Errorf("ULIDs with same input time should have same timestamp: %d != %d", ul1.Time(), ul2.Time())
}
t.Logf("ulid string 1 and 2: %s | %s", ul1.String(), ul2.String())
}
func TestMake(t *testing.T) {
// Test Make() function (uses current time)
ul1, err := Make()
if err != nil {
t.Fatalf("Make failed: %s", err)
}
if ul1 == nil {
t.Fatal("Make returned nil ULID")
}
// Sleep a bit and generate another
time.Sleep(2 * time.Millisecond)
ul2, err := Make()
if err != nil {
t.Fatalf("Make failed: %s", err)
}
// Should be different ULIDs
if ul1.String() == ul2.String() {
t.Errorf("ULIDs from Make() should be different: %s", ul1.String())
}
// Second should be later (or at least not earlier)
if ul1.Time() > ul2.Time() {
t.Errorf("second ULID should not have earlier timestamp: %d > %d", ul1.Time(), ul2.Time())
}
t.Logf("Make() ULIDs: %s | %s", ul1.String(), ul2.String())
}
func TestMakeULIDUniqueness(t *testing.T) {
tm := time.Now()
seen := make(map[string]bool)
for i := 0; i < 1000; i++ {
ul, err := MakeULID(tm)
if err != nil {
t.Fatalf("MakeULID failed on iteration %d: %s", i, err)
}
str := ul.String()
if seen[str] {
t.Errorf("duplicate ULID generated: %s", str)
}
seen[str] = true
}
}
func TestMakeUniqueness(t *testing.T) {
seen := make(map[string]bool)
for i := 0; i < 1000; i++ {
ul, err := Make()
if err != nil {
t.Fatalf("Make failed on iteration %d: %s", i, err)
}
str := ul.String()
if seen[str] {
t.Errorf("duplicate ULID generated: %s", str)
}
seen[str] = true
}
}
func TestMakeULIDTimestampProgression(t *testing.T) {
t1 := time.Now()
ul1, err := MakeULID(t1)
if err != nil {
t.Fatalf("MakeULID failed: %s", err)
}
// Wait to ensure different timestamp
time.Sleep(2 * time.Millisecond)
t2 := time.Now()
ul2, err := MakeULID(t2)
if err != nil {
t.Fatalf("MakeULID failed: %s", err)
}
if ul1.Time() >= ul2.Time() {
t.Errorf("second ULID should have later timestamp: %d >= %d", ul1.Time(), ul2.Time())
}
if ul1.Compare(*ul2) >= 0 {
t.Errorf("second ULID should be greater: %s >= %s", ul1.String(), ul2.String())
}
}
func TestMakeULIDConcurrency(t *testing.T) {
const numGoroutines = 10
const numULIDsPerGoroutine = 100
var wg sync.WaitGroup
ulidChan := make(chan *oklid.ULID, numGoroutines*numULIDsPerGoroutine)
tm := time.Now()
// Start multiple goroutines generating ULIDs concurrently
for i := 0; i < numGoroutines; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for j := 0; j < numULIDsPerGoroutine; j++ {
ul, err := MakeULID(tm)
if err != nil {
t.Errorf("MakeULID failed: %s", err)
return
}
ulidChan <- ul
}
}()
}
wg.Wait()
close(ulidChan)
// Collect all ULIDs and check uniqueness
seen := make(map[string]bool)
count := 0
for ul := range ulidChan {
str := ul.String()
if seen[str] {
t.Errorf("duplicate ULID generated in concurrent test: %s", str)
}
seen[str] = true
count++
}
if count != numGoroutines*numULIDsPerGoroutine {
t.Errorf("expected %d ULIDs, got %d", numGoroutines*numULIDsPerGoroutine, count)
}
}
func TestMakeConcurrency(t *testing.T) {
const numGoroutines = 10
const numULIDsPerGoroutine = 100
var wg sync.WaitGroup
ulidChan := make(chan *oklid.ULID, numGoroutines*numULIDsPerGoroutine)
// Start multiple goroutines generating ULIDs concurrently
for i := 0; i < numGoroutines; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for j := 0; j < numULIDsPerGoroutine; j++ {
ul, err := Make()
if err != nil {
t.Errorf("Make failed: %s", err)
return
}
ulidChan <- ul
}
}()
}
wg.Wait()
close(ulidChan)
// Collect all ULIDs and check uniqueness
seen := make(map[string]bool)
count := 0
for ul := range ulidChan {
str := ul.String()
if seen[str] {
t.Errorf("duplicate ULID generated in concurrent test: %s", str)
}
seen[str] = true
count++
}
if count != numGoroutines*numULIDsPerGoroutine {
t.Errorf("expected %d ULIDs, got %d", numGoroutines*numULIDsPerGoroutine, count)
}
}
func TestMakeULIDErrorHandling(t *testing.T) {
// Test with various timestamps
timestamps := []time.Time{
time.Unix(0, 0), // Unix epoch
time.Now(), // Current time
time.Now().Add(time.Hour), // Future time
}
for i, tm := range timestamps {
ul, err := MakeULID(tm)
if err != nil {
t.Errorf("MakeULID failed with timestamp %d: %s", i, err)
}
if ul == nil {
t.Errorf("MakeULID returned nil ULID with timestamp %d", i)
}
}
}
func TestMakeULIDLexicographicOrdering(t *testing.T) {
var ulids []*oklid.ULID
var timestamps []time.Time
// Generate ULIDs with increasing timestamps
for i := 0; i < 10; i++ {
tm := time.Now().Add(time.Duration(i) * time.Millisecond)
timestamps = append(timestamps, tm)
ul, err := MakeULID(tm)
if err != nil {
t.Fatalf("MakeULID failed: %s", err)
}
ulids = append(ulids, ul)
// Small delay to ensure different timestamps
time.Sleep(time.Millisecond)
}
// Sort ULID strings lexicographically
ulidStrings := make([]string, len(ulids))
for i, ul := range ulids {
ulidStrings[i] = ul.String()
}
originalOrder := make([]string, len(ulidStrings))
copy(originalOrder, ulidStrings)
sort.Strings(ulidStrings)
// Verify lexicographic order matches chronological order
for i := 0; i < len(originalOrder); i++ {
if originalOrder[i] != ulidStrings[i] {
t.Errorf("lexicographic order doesn't match chronological order at index %d: %s != %s",
i, originalOrder[i], ulidStrings[i])
}
}
}
// Benchmark ULID generation performance
func BenchmarkMakeULID(b *testing.B) {
tm := time.Now()
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := MakeULID(tm)
if err != nil {
b.Fatalf("MakeULID failed: %s", err)
}
}
}
// Benchmark Make function
func BenchmarkMake(b *testing.B) {
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := Make()
if err != nil {
b.Fatalf("Make failed: %s", err)
}
}
}
// Benchmark concurrent ULID generation
func BenchmarkMakeULIDConcurrent(b *testing.B) {
tm := time.Now()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
_, err := MakeULID(tm)
if err != nil {
b.Fatalf("MakeULID failed: %s", err)
}
}
})
}
// Benchmark concurrent Make function
func BenchmarkMakeConcurrent(b *testing.B) {
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
_, err := Make()
if err != nil {
b.Fatalf("Make failed: %s", err)
}
}
})
}
// Benchmark random number generation
func BenchmarkCryptoRand(b *testing.B) {
buf := make([]byte, 10) // ULID entropy size
b.ResetTimer()
for i := 0; i < b.N; i++ {
cryptorand.Read(buf)
}
}

View File

@ -1,3 +1,17 @@
// Package version provides build metadata and version information management.
//
// This package manages application version information including semantic version,
// Git revision, build time, and provides integration with CLI frameworks (Cobra, Kong)
// and Prometheus metrics for operational visibility.
//
// Version information can be injected at build time using ldflags:
//
// go build -ldflags "-X go.ntppool.org/common/version.VERSION=v1.0.0 \
// -X go.ntppool.org/common/version.buildTime=2023-01-01T00:00:00Z \
// -X go.ntppool.org/common/version.gitVersion=abc123"
//
// The package also automatically extracts build information from Go's debug.BuildInfo
// when available, providing fallback values for VCS time and revision.
package version
import (
@ -12,23 +26,28 @@ import (
"golang.org/x/mod/semver"
)
// VERSION has the current software version (set in the build process)
var VERSION string
var buildTime string
var gitVersion string
var gitModified bool
// VERSION contains the current software version (typically set during the build process via ldflags).
// If not set, defaults to "dev-snapshot". The version should follow semantic versioning.
var (
VERSION string // Semantic version (e.g., "1.0.0" or "v1.0.0")
buildTime string // Build timestamp (RFC3339 format)
gitVersion string // Git commit hash
gitModified bool // Whether the working tree was modified during build
)
// info holds the consolidated version information extracted from build variables and debug.BuildInfo.
var info Info
// Info represents structured version and build information.
// This struct is used for JSON serialization and programmatic access to build metadata.
type Info struct {
Version string `json:",omitempty"`
GitRev string `json:",omitempty"`
GitRevShort string `json:",omitempty"`
BuildTime string `json:",omitempty"`
Version string `json:",omitempty"` // Semantic version with "v" prefix
GitRev string `json:",omitempty"` // Full Git commit hash
GitRevShort string `json:",omitempty"` // Shortened Git commit hash (7 characters)
BuildTime string `json:",omitempty"` // Build timestamp
}
func init() {
info.BuildTime = buildTime
info.GitRev = gitVersion
@ -78,10 +97,16 @@ func init() {
Version()
}
// VersionCmd creates a Cobra command for displaying version information.
// The name parameter is used as a prefix in the output (e.g., "myapp v1.0.0").
// Returns a configured cobra.Command that can be added to any CLI application.
func VersionCmd(name string) *cobra.Command {
versionCmd := &cobra.Command{
Use: "version",
Short: "Print version and build information",
Long: `Print detailed version information including semantic version,
Git revision, build time, and Go version. Build information is automatically
extracted from Go's debug.BuildInfo when available.`,
Run: func(cmd *cobra.Command, args []string) {
ver := Version()
fmt.Printf("%s %s\n", name, ver)
@ -90,6 +115,23 @@ func VersionCmd(name string) *cobra.Command {
return versionCmd
}
// KongVersionCmd provides a Kong CLI framework compatible version command.
// The Name field should be set to the application name for proper output formatting.
type KongVersionCmd struct {
Name string `kong:"-"` // Application name, excluded from Kong parsing
}
// Run executes the version command for Kong CLI framework.
// Prints the application name and version information to stdout.
func (cmd *KongVersionCmd) Run() error {
fmt.Printf("%s %s\n", cmd.Name, Version())
return nil
}
// RegisterMetric registers a Prometheus gauge metric with build information.
// If name is provided, it creates a metric named "{name}_build_info", otherwise "build_info".
// The metric includes labels for version, build time, Git time, and Git revision.
// This is useful for exposing build information in monitoring systems.
func RegisterMetric(name string, registry prometheus.Registerer) {
if len(name) > 0 {
name = strings.ReplaceAll(name, "-", "_")
@ -100,13 +142,13 @@ func RegisterMetric(name string, registry prometheus.Registerer) {
buildInfo := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: name,
Help: "Build information",
Help: "Build information including version, build time, and git revision",
},
[]string{
"version",
"buildtime",
"gittime",
"git",
"version", // Combined version/git format (e.g., "v1.0.0/abc123")
"buildtime", // Build timestamp from ldflags
"gittime", // Git commit timestamp from VCS info
"git", // Full Git commit hash
},
)
registry.MustRegister(buildInfo)
@ -121,12 +163,20 @@ func RegisterMetric(name string, registry prometheus.Registerer) {
).Set(1)
}
// v caches the formatted version string to avoid repeated computation.
var v string
// VersionInfo returns the structured version information.
// This provides programmatic access to version details for JSON serialization
// or other structured uses.
func VersionInfo() Info {
return info
}
// Version returns a human-readable version string suitable for display.
// The format includes semantic version, Git revision, build time, and Go version.
// Example: "v1.0.0/abc123f-M (2023-01-01T00:00:00Z, go1.21.0)"
// The "-M" suffix indicates the working tree was modified during build.
func Version() string {
if len(v) > 0 {
return v
@ -154,10 +204,23 @@ func Version() string {
return v
}
// CheckVersion compares a version against a minimum required version.
// Returns true if the version meets or exceeds the minimum requirement.
//
// Special handling:
// - "dev-snapshot" is always considered valid (returns true)
// - Git hash suffixes (e.g., "v1.0.0/abc123") are stripped before comparison
// - Uses semantic version comparison rules
//
// Both version and minimumVersion should follow semantic versioning with "v" prefix.
func CheckVersion(version, minimumVersion string) bool {
if version == "dev-snapshot" {
return true
}
// Strip Git hash suffix if present (e.g., "v1.0.0/abc123" -> "v1.0.0")
if idx := strings.Index(version, "/"); idx >= 0 {
version = version[0:idx]
}
if semver.Compare(version, minimumVersion) < 0 {
// log.Debug("version too old", "v", cl.Version.Version)
return false

311
version/version_test.go Normal file
View File

@ -0,0 +1,311 @@
package version
import (
"runtime"
"strings"
"testing"
"github.com/prometheus/client_golang/prometheus"
dto "github.com/prometheus/client_model/go"
)
func TestCheckVersion(t *testing.T) {
tests := []struct {
In string
Min string
Expected bool
}{
// Basic version comparisons
{"v3.8.4", "v3.8.5", false},
{"v3.9.3", "v3.8.5", true},
{"v3.8.5", "v3.8.5", true},
// Dev snapshot should always pass
{"dev-snapshot", "v3.8.5", true},
{"dev-snapshot", "v99.99.99", true},
// Versions with Git hashes should be stripped
{"v3.8.5/abc123", "v3.8.5", true},
{"v3.8.4/abc123", "v3.8.5", false},
{"v3.9.0/def456", "v3.8.5", true},
// Pre-release versions
{"v3.8.5-alpha", "v3.8.5", false},
{"v3.8.5", "v3.8.5-alpha", true},
{"v3.8.5-beta", "v3.8.5-alpha", true},
}
for _, d := range tests {
r := CheckVersion(d.In, d.Min)
if r != d.Expected {
t.Errorf("CheckVersion(%q, %q) = %t, expected %t", d.In, d.Min, r, d.Expected)
}
}
}
func TestVersionInfo(t *testing.T) {
info := VersionInfo()
// Check that we get a valid Info struct
if info.Version == "" {
t.Error("VersionInfo().Version should not be empty")
}
// Version should start with "v" or be "dev-snapshot"
if !strings.HasPrefix(info.Version, "v") && info.Version != "dev-snapshot" {
t.Errorf("Version should start with 'v' or be 'dev-snapshot', got: %s", info.Version)
}
// GitRevShort should be <= 7 characters if set
if info.GitRevShort != "" && len(info.GitRevShort) > 7 {
t.Errorf("GitRevShort should be <= 7 characters, got: %s", info.GitRevShort)
}
// GitRevShort should be prefix of GitRev if both are set
if info.GitRev != "" && info.GitRevShort != "" {
if !strings.HasPrefix(info.GitRev, info.GitRevShort) {
t.Errorf("GitRevShort should be prefix of GitRev: %s not prefix of %s",
info.GitRevShort, info.GitRev)
}
}
}
func TestVersion(t *testing.T) {
version := Version()
if version == "" {
t.Error("Version() should not return empty string")
}
// Should contain Go version
if !strings.Contains(version, runtime.Version()) {
t.Errorf("Version should contain Go version %s, got: %s", runtime.Version(), version)
}
// Should contain the VERSION variable (or dev-snapshot)
info := VersionInfo()
if !strings.Contains(version, info.Version) {
t.Errorf("Version should contain %s, got: %s", info.Version, version)
}
// Should be in expected format: "version (extras)"
if !strings.Contains(version, "(") || !strings.Contains(version, ")") {
t.Errorf("Version should be in format 'version (extras)', got: %s", version)
}
}
func TestVersionCmd(t *testing.T) {
appName := "testapp"
cmd := VersionCmd(appName)
// Test basic command properties
if cmd.Use != "version" {
t.Errorf("Expected command use to be 'version', got: %s", cmd.Use)
}
if cmd.Short == "" {
t.Error("Command should have a short description")
}
if cmd.Long == "" {
t.Error("Command should have a long description")
}
if cmd.Run == nil {
t.Error("Command should have a Run function")
}
// Test that the command can be executed without error
cmd.SetArgs([]string{})
err := cmd.Execute()
if err != nil {
t.Errorf("VersionCmd execution should not return error, got: %s", err)
}
}
func TestKongVersionCmd(t *testing.T) {
cmd := &KongVersionCmd{Name: "testapp"}
// Test that Run() doesn't return an error
err := cmd.Run()
if err != nil {
t.Errorf("KongVersionCmd.Run() should not return error, got: %s", err)
}
}
func TestRegisterMetric(t *testing.T) {
// Create a test registry
registry := prometheus.NewRegistry()
// Test registering metric without name
RegisterMetric("", registry)
// Gather metrics
metricFamilies, err := registry.Gather()
if err != nil {
t.Fatalf("Failed to gather metrics: %s", err)
}
// Find the build_info metric
var buildInfoFamily *dto.MetricFamily
for _, family := range metricFamilies {
if family.GetName() == "build_info" {
buildInfoFamily = family
break
}
}
if buildInfoFamily == nil {
t.Fatal("build_info metric not found")
}
if buildInfoFamily.GetHelp() == "" {
t.Error("build_info metric should have help text")
}
metrics := buildInfoFamily.GetMetric()
if len(metrics) == 0 {
t.Fatal("build_info metric should have at least one sample")
}
// Check that the metric has the expected labels
metric := metrics[0]
labels := metric.GetLabel()
expectedLabels := []string{"version", "buildtime", "gittime", "git"}
labelMap := make(map[string]string)
for _, label := range labels {
labelMap[label.GetName()] = label.GetValue()
}
for _, expectedLabel := range expectedLabels {
if _, exists := labelMap[expectedLabel]; !exists {
t.Errorf("Expected label %s not found in metric", expectedLabel)
}
}
// Check that the metric value is 1
if metric.GetGauge().GetValue() != 1 {
t.Errorf("Expected build_info metric value to be 1, got %f", metric.GetGauge().GetValue())
}
}
func TestRegisterMetricWithName(t *testing.T) {
// Create a test registry
registry := prometheus.NewRegistry()
// Test registering metric with custom name
appName := "my-test-app"
RegisterMetric(appName, registry)
// Gather metrics
metricFamilies, err := registry.Gather()
if err != nil {
t.Fatalf("Failed to gather metrics: %s", err)
}
// Find the my_test_app_build_info metric
expectedName := "my_test_app_build_info"
var buildInfoFamily *dto.MetricFamily
for _, family := range metricFamilies {
if family.GetName() == expectedName {
buildInfoFamily = family
break
}
}
if buildInfoFamily == nil {
t.Fatalf("%s metric not found", expectedName)
}
}
func TestVersionConsistency(t *testing.T) {
// Call Version() multiple times and ensure it returns the same result
v1 := Version()
v2 := Version()
if v1 != v2 {
t.Errorf("Version() should return consistent results: %s != %s", v1, v2)
}
}
func TestVersionInfoConsistency(t *testing.T) {
// Ensure VersionInfo() is consistent with Version()
info := VersionInfo()
version := Version()
// Version string should contain the semantic version
if !strings.Contains(version, info.Version) {
t.Errorf("Version() should contain VersionInfo().Version: %s not in %s",
info.Version, version)
}
// If GitRevShort is set, version should contain it
if info.GitRevShort != "" {
if !strings.Contains(version, info.GitRevShort) {
t.Errorf("Version() should contain GitRevShort: %s not in %s",
info.GitRevShort, version)
}
}
}
// Test edge cases
func TestCheckVersionEdgeCases(t *testing.T) {
// Test with empty strings
if CheckVersion("", "v1.0.0") {
t.Error("Empty version should not be >= v1.0.0")
}
// Test with malformed versions (should be handled gracefully)
// Note: semver.Compare might panic or return unexpected results for invalid versions
// but our function should handle the common cases
tests := []struct {
version string
minimum string
desc string
}{
{"v1.0.0/", "v1.0.0", "version with trailing slash"},
{"v1.0.0/abc/def", "v1.0.0", "version with multiple slashes"},
}
for _, test := range tests {
// This should not panic
result := CheckVersion(test.version, test.minimum)
t.Logf("%s: CheckVersion(%q, %q) = %t", test.desc, test.version, test.minimum, result)
}
}
// Benchmark version operations
func BenchmarkVersion(b *testing.B) {
// Reset the cached version to test actual computation
v = ""
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = Version()
}
}
func BenchmarkVersionInfo(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = VersionInfo()
}
}
func BenchmarkCheckVersion(b *testing.B) {
version := "v1.2.3/abc123"
minimum := "v1.2.0"
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = CheckVersion(version, minimum)
}
}
func BenchmarkCheckVersionDevSnapshot(b *testing.B) {
version := "dev-snapshot"
minimum := "v1.2.0"
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = CheckVersion(version, minimum)
}
}

View File

@ -1,3 +1,27 @@
// Package fastlyxff provides Fastly CDN IP range management for trusted proxy handling.
//
// This package parses Fastly's public IP ranges JSON file and generates Echo framework
// trust options for proper client IP extraction from X-Forwarded-For headers.
// It's designed specifically for services deployed behind Fastly's CDN that need
// to identify real client IPs for logging, rate limiting, and security purposes.
//
// Fastly publishes their edge server IP ranges in a JSON format that this package
// consumes to automatically configure trusted proxy ranges. This ensures that
// X-Forwarded-For headers are only trusted when they originate from legitimate
// Fastly edge servers.
//
// Key features:
// - Automatic parsing of Fastly's IP ranges JSON format
// - Support for both IPv4 and IPv6 address ranges
// - Echo framework integration via TrustOption generation
// - CIDR notation parsing and validation
//
// The JSON file typically contains IP ranges in this format:
//
// {
// "addresses": ["23.235.32.0/20", "43.249.72.0/22", ...],
// "ipv6_addresses": ["2a04:4e40::/32", "2a04:4e42::/32", ...]
// }
package fastlyxff
import (
@ -9,15 +33,29 @@ import (
"github.com/labstack/echo/v4"
)
// FastlyXFF represents Fastly's published IP ranges for their CDN edge servers.
// This structure matches the JSON format provided by Fastly for their public IP ranges.
// It contains separate lists for IPv4 and IPv6 CIDR ranges.
type FastlyXFF struct {
IPv4 []string `json:"addresses"`
IPv6 []string `json:"ipv6_addresses"`
IPv4 []string `json:"addresses"` // IPv4 CIDR ranges (e.g., "23.235.32.0/20")
IPv6 []string `json:"ipv6_addresses"` // IPv6 CIDR ranges (e.g., "2a04:4e40::/32")
}
// TrustedNets holds parsed network prefixes for efficient IP range checking.
// This type is currently unused but reserved for future optimizations
// where frequent IP range lookups might benefit from pre-parsed prefixes.
type TrustedNets struct {
prefixes []netip.Prefix
prefixes []netip.Prefix // Parsed network prefixes for efficient lookups
}
// New loads and parses Fastly IP ranges from a JSON file.
// The file should contain Fastly's published IP ranges in their standard JSON format.
//
// Parameters:
// - fileName: Path to the Fastly IP ranges JSON file
//
// Returns the parsed FastlyXFF structure or an error if the file cannot be
// read or the JSON format is invalid.
func New(fileName string) (*FastlyXFF, error) {
b, err := os.ReadFile(fileName)
if err != nil {
@ -34,6 +72,19 @@ func New(fileName string) (*FastlyXFF, error) {
return &d, nil
}
// EchoTrustOption converts Fastly IP ranges into Echo framework trust options.
// This method generates trust configurations that tell Echo to accept X-Forwarded-For
// headers only from Fastly's edge servers, ensuring accurate client IP extraction.
//
// The generated trust options should be used with Echo's IP extractor:
//
// options, err := fastlyRanges.EchoTrustOption()
// if err != nil {
// return err
// }
// e.IPExtractor = echo.ExtractIPFromXFFHeader(options...)
//
// Returns a slice of Echo trust options or an error if any CIDR range cannot be parsed.
func (xff *FastlyXFF) EchoTrustOption() ([]echo.TrustOption, error) {
ranges := []echo.TrustOption{}

View File

@ -3,14 +3,12 @@ package fastlyxff
import "testing"
func TestFastlyIPRanges(t *testing.T) {
fastlyxff, err := New("fastly.json")
if err != nil {
t.Fatalf("could not load test data: %s", err)
}
data, err := fastlyxff.EchoTrustOption()
if err != nil {
t.Fatalf("could not parse test data: %s", err)
}
@ -19,5 +17,4 @@ func TestFastlyIPRanges(t *testing.T) {
t.Logf("only got %d prefixes, expected more", len(data))
t.Fail()
}
}