Observability¶

SystemForge integrates with omniobserve to provide vendor-agnostic observability including metrics, traces, and logs with support for multiple backends.

Supported Backends¶

OTLP - OpenTelemetry Protocol (vendor-agnostic)
Datadog
New Relic
Dynatrace

Quick Start¶

1. Import the Provider¶

Import the observability package and the provider you want to use:

import (
    "github.com/grokify/systemforge/observability"
    _ "github.com/plexusone/omniobserve/observops/otlp"     // OTLP
    // or
    _ "github.com/plexusone/omniobserve/observops/datadog"  // Datadog
    // or
    _ "github.com/plexusone/omniobserve/observops/newrelic" // New Relic
)

2. Create Observability Instance¶

obs, err := observability.New(observability.Config{
    Provider:       "otlp",
    ServiceName:    "my-app",
    ServiceVersion: "1.0.0",
    Endpoint:       "localhost:4317",
})
if err != nil {
    log.Fatal(err)
}
defer obs.Shutdown(context.Background())

3. Integrate with SystemForge Components¶

CoreAuth (OAuth Server)¶

server, err := coreauth.NewEmbedded(cfg,
    coreauth.WithObservability(obs),
)

Rate Limiting¶

limiter := ratelimit.New(storage,
    ratelimit.WithObservability(obs),
)
router.Use(limiter.Middleware())

JWT Middleware¶

router.Use(middleware.HTTPAuthWithObservability(jwtService, obs))

API Key Middleware¶

config := middleware.DefaultAPIKeyMiddlewareConfig()
config.Service = apikeyService
config.Observability = obs
router.Use(middleware.APIKeyMiddleware(config))

HTTP Request Tracing¶

router.Use(obs.Middleware())

slog Integration¶

// Use observability-integrated logging
handler := obs.SlogHandler(
    observops.WithSlogLocalHandler(slog.NewJSONHandler(os.Stdout, nil)),
)
slog.SetDefault(slog.New(handler))

Configuration¶

Environment Variables¶

Variable	Description	Example
`OBSERVABILITY_PROVIDER`	Backend provider	`otlp`, `datadog`, `newrelic`, `dynatrace`
`OBSERVABILITY_ENDPOINT`	Backend endpoint	`localhost:4317`
`OBSERVABILITY_API_KEY`	API key (if required)
`OBSERVABILITY_SERVICE_NAME`	Service name	`my-app`
`OBSERVABILITY_SERVICE_VERSION`	Service version	`1.0.0`
`OBSERVABILITY_DISABLED`	Disable observability	`true`, `false`
`OBSERVABILITY_INSECURE`	Disable TLS	`true`, `false`
`OBSERVABILITY_DEBUG`	Enable debug logging	`true`, `false`

Configuration from Environment¶

obs, err := observability.New(observability.ConfigFromEnv())

Metrics¶

CoreAuth Metrics¶

Metric	Type	Labels	Description
`systemforge.coreauth.auth_requests_total`	Counter	grant_type, client_id, status	Authentication requests
`systemforge.coreauth.auth_latency_ms`	Histogram	grant_type, endpoint	Request latency
`systemforge.coreauth.tokens_issued_total`	Counter	grant_type, client_id	Tokens issued
`systemforge.coreauth.token_validations_total`	Counter	result	Token validations
`systemforge.coreauth.sessions_active`	Gauge		Active sessions

Rate Limiting Metrics¶

Metric	Type	Labels	Description
`systemforge.coreapi.ratelimit_requests_total`	Counter	policy_id, client_id, allowed	Rate limit checks
`systemforge.coreapi.ratelimit_quota_usage`	Gauge	policy_id, client_id, window	Current usage ratio

Session Middleware Metrics¶

Metric	Type	Labels	Description
`systemforge.session.jwt_validations_total`	Counter	result	JWT validations
`systemforge.session.jwt_validation_latency_ms`	Histogram		Validation latency
`systemforge.session.apikey_validations_total`	Counter	result	API key validations

Traces¶

Span Names¶

Span	Description
`systemforge.coreauth.authorize`	OAuth authorization endpoint
`systemforge.coreauth.token`	OAuth token endpoint
`systemforge.coreauth.introspect`	Token introspection
`systemforge.coreauth.revoke`	Token revocation
`systemforge.http.request`	HTTP request
`systemforge.session.jwt_validation`	JWT validation
`systemforge.session.apikey_validation`	API key validation
`systemforge.ratelimit.check`	Rate limit check

Testing with Jaeger¶

Run Jaeger locally to test OTLP integration:

docker run -d --name jaeger \
  -p 16686:16686 -p 4317:4317 \
  jaegertracing/all-in-one:latest

Configure your app:

OBSERVABILITY_PROVIDER=otlp \
OBSERVABILITY_ENDPOINT=localhost:4317 \
OBSERVABILITY_INSECURE=true \
go run ./cmd/server

View traces at http://localhost:16686

Recording Custom Metrics¶

Use the underlying observops.Provider for custom metrics:

meter := obs.Meter()

counter, _ := meter.Counter("my_custom_counter",
    observops.WithDescription("My custom counter"),
)

counter.Add(ctx, 1, observops.WithAttributes(
    observops.Attribute("key", "value"),
))

Creating Custom Spans¶

ctx, span := obs.StartSpan(ctx, "my.custom.operation",
    observops.WithSpanKind(observops.SpanKindInternal),
    observops.WithSpanAttributes(
        observops.Attribute("user.id", userID),
    ),
)
defer span.End()

// ... your code here ...

if err != nil {
    span.RecordError(err)
    span.SetStatus(observops.StatusCodeError, err.Error())
}

Disabling Observability¶

To disable observability entirely:

obs, _ := observability.New(observability.Config{
    Disabled: true,
})

Or via environment:

OBSERVABILITY_DISABLED=true

When disabled, all operations are no-ops with minimal overhead.

ProductGraph Integration¶

SystemForge integrates with ProductGraph for product analytics, enabling frontend-backend correlation and user journey tracking.

Quick Start¶

import (
    "github.com/grokify/systemforge/observability"
    "github.com/grokify/systemforge/productgraph"
)

// Create observability instance
obs, err := observability.New(observability.ConfigFromEnv())
if err != nil {
    log.Fatal(err)
}

// Configure ProductGraph
err = obs.SetProductGraph(productgraph.Config{
    ProjectID: "my-project",
    Endpoint:  "https://api.productgraph.io/v1/events",
    APIKey:    os.Getenv("PRODUCTGRAPH_API_KEY"),
})
// Or from environment:
// err = obs.SetProductGraphFromEnv()

defer obs.Shutdown(context.Background())

Middleware¶

Use the combined middleware for automatic request tracking with frontend correlation:

router := chi.NewRouter()

// Combined middleware (recommended)
router.Use(obs.ProductGraphMiddleware())

// Or use components separately:
// router.Use(productgraph.CorrelationMiddleware)           // Extract headers
// router.Use(obs.ProductGraphRequestTrackerMiddleware())   // Track requests

Frontend Correlation Headers¶

ProductGraph correlates backend requests with frontend sessions using these headers:

Header	Description
`X-Session-ID`	Frontend session identifier
`X-Request-ID`	Per-request identifier (auto-generated if missing)
`X-User-ID`	Authenticated user identifier

Access correlation IDs in handlers:

func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    sessionID := productgraph.SessionIDFromContext(ctx)
    requestID := productgraph.RequestIDFromContext(ctx)
    userID := productgraph.UserIDFromContext(ctx)
}

Manual Event Tracking¶

// Track API calls
obs.TrackAPICall(ctx, "POST", "/api/checkout", 200, 150*time.Millisecond)

// Track errors
obs.TrackError(ctx, "ValidationError", "invalid card number")

// Track user journey steps
obs.TrackJourneyStep(ctx, "checkout_flow", "payment", "Enter Payment")

// Track custom events
obs.TrackProductGraphEvent(ctx, productgraph.Event{
    EventType: productgraph.EventTypeUIClick,
    UIElement: "checkout_button",
})

Configuration¶

Environment Variables¶

Variable	Description	Default
`PRODUCTGRAPH_PROJECT_ID`	Project identifier
`PRODUCTGRAPH_ENDPOINT`	API endpoint
`PRODUCTGRAPH_API_KEY`	Authentication key
`PRODUCTGRAPH_BATCH_SIZE`	Events per batch	`50`
`PRODUCTGRAPH_BATCH_INTERVAL`	Flush interval (seconds)	`5`
`PRODUCTGRAPH_DEBUG`	Enable debug logging	`false`

Event Types¶

ProductGraph follows OTel semantic conventions for event types:

Event Type	Description
`page.view`	Page navigation
`ui.click`, `ui.input`, `ui.scroll`, `ui.submit`	User interactions
`api.request`, `api.response`	API calls
`journey.step`	User journey progression
`error`	Application errors
`performance`	Performance metrics

See ProductGraph Design Docs for detailed architecture.