Observability

Francis is instrumented with OpenTelemetry . A single distributed trace follows a request end-to-end as it crosses hosts and the runtime, so you can see exactly where time goes: the actor invocation, the cross-host or host-to-runtime call, the turn-based lock wait, the actor’s own method, state operations, placement lookups, and alarms.

Instrumentation is built into the framework and is always present, but it does nothing until OpenTelemetry is configured. With no exporter configured, spans are created against a no-op tracer with negligible overhead, so there is nothing to turn off in development.

What gets traced#

A trace is assembled from spans created at every meaningful boundary. The main span names are:

Spans are tagged with attributes such as francis.actor.type, francis.actor.id, francis.actor.method, francis.request.id, francis.host.id, and francis.peer.address.
Provider spans carry francis.provider.method, and statement spans carry db.query.text.
SQL statements are always recorded in traces. Parameter values are excluded by default and can be included with QueryLog.IncludeParameters, or with instrument.Options.IncludeParameters when instrumenting an existing database handle directly.

Note: alarms start their own trace.
A fired alarm is not triggered by a caller request, so alarm.dispatch and alarm.execute begin a fresh trace rather than attaching to an unrelated one. The trace still spans the runtime and the owning host.

SQL visibility#

Two complementary levels make database work observable:

• Provider spans (provider.<Method>) are emitted for every provider method call made through the local-host, runtime, or ClusterAdmin construction paths, no matter how the database connection was created.
  A slow health check or lease renewal shows up as its own span, with the error attached when it fails.
• Statement spans (sqlite.*, postgresql.*) are emitted for every SQL statement, but only when Francis opens the database connection itself (a ConnectionString in the provider options, or the runtime binary’s provider.connectionString).
  On Postgres, postgresql.pool.acquire spans do the same for time spent waiting for a free connection in the pool.

If your application creates its own database handle and passes it to the provider (DB field in the provider options), statement spans are not automatic. To enable them, use the instrumentation helpers from go-sql-utils when opening the connection:

import (
	"github.com/italypaleale/go-sql-utils/instrument"
	gosqlsqlite "github.com/italypaleale/go-sql-utils/sqlite"
	sqliteinstrument "github.com/italypaleale/go-sql-utils/instrument/sqlite"
	postgresinstrument "github.com/italypaleale/go-sql-utils/instrument/postgres"
)

// SQLite: open the database with the instrumented driver
connector, err := gosqlsqlite.NewConnector(gosqlsqlite.ConnectOpts{
    ConnString: dsn,
    Logger:     logger,
})
db, err := sqliteinstrument.Open(connector, &instrument.Options{})

// Postgres: attach the tracer to the pool config, chaining any tracer you already set
cfg, err := pgxpool.ParseConfig(connString)
cfg.ConnConfig.Tracer = postgresinstrument.NewTracer(&instrument.Options{}, cfg.ConnConfig.Tracer)
pool, err := pgxpool.NewWithConfig(ctx, cfg)

Direct callers of the low-level provider constructors can add provider-operation telemetry explicitly with WrapProvider from Francis’s components/instrument package. The local host, runtime, and ClusterAdmin factories apply that wrapper automatically.

Database logging#

Aside from traces, francis can log SQL statements and provider operations with their durations using two independent configurations:

• QueryLog (or runtime provider.queryLog) controls SQL statement logs for connections opened by the provider.
• OperationLog (or runtime provider.operationLog) controls provider-operation logs for every backend, including memory.

Each configuration has an Enabled switch for Debug records and a SlowThreshold for Warn records. QueryLog also has an IncludeParameters switch, which defaults to false because parameter values may contain sensitive data.

SQL text in logs is normalized to one line with consecutive whitespace condensed. Every SQL log includes code.file.path and code.line.number attributes for the query call.

Parameter values are emitted as db.query.parameter.<name-or-position> attributes in traces when QueryLog.IncludeParameters is true. Logs include them only alongside SQL text when Debug query logging is active, so an Info-level slow-query warning omits both SQL text and parameters.

provider:
  connectionString: "data.db"
  queryLog:
    enabled: true
    includeParameters: false
    slowThreshold: "250ms"
  operationLog:
    enabled: true
    slowThreshold: "500ms"

log:
  level: debug

When a provider is given an existing database handle (DB option), statement-level logging requires opening the connection with sqliteinstrument.Open or postgresinstrument.NewTracer as shown above, passing the corresponding instrument.Options. Set instrument.Options.IncludeParameters to include parameter values in that case.

Trace context propagation#

Francis propagates the W3C Trace Context on every request it sends between hosts and the runtime, so a trace started on one host continues seamlessly on another. Propagation uses the globally configured OpenTelemetry propagator; it is a no-op when none is set, so nothing is written to the wire unless tracing is enabled.

Standalone runtime#

The standalone runtime binary exports traces, metrics, and logs through the standard OTEL_* environment variables . Export is opt-in: each signal stays off until you set its exporter.

# Send traces to an OTLP collector
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT="http://collector:4318"

# Optionally enable metrics and logs the same way
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp

Common values:

Tuning trace volume#

In production you usually don’t want every internal span for every request, so control the volume with the sampler rather than turning instrumentation off:

# Keep ~10% of traces, respecting the decision made upstream
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

Because the decision is parent-based, a trace that was sampled on a calling host stays sampled as it flows into the runtime and on to a peer host.

Logs#

Regardless of OTLP export, the runtime always writes logs to standard output. The format is controlled by the runtime configuration :

log:
  level: info   # debug, info, warn, or error
  json: true    # structured JSON instead of text

When OTEL_LOGS_EXPORTER is set, log records are also exported via OpenTelemetry and carry the active trace and span IDs, so logs correlate with traces.

Metrics#

When OTEL_METRICS_EXPORTER is set, the runtime emits:

Embedded (local) topology#

In the local topology your application owns the process, so it also owns the OpenTelemetry setup. Francis uses the global tracer and propagator, so it automatically picks up whatever your app configures.

Configure the OpenTelemetry SDK as usual, then install a trace-context propagator so spans flow across hosts:

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
)

// ... build and set your TracerProvider (exporter, sampler, resource) ...
otel.SetTracerProvider(tp)

// Install the W3C trace context propagator so traces flow across hosts
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
	propagation.TraceContext{},
	propagation.Baggage{},
))

If your app never configures OpenTelemetry, Francis’s spans are no-ops and nothing is propagated.

Note: don’t forget the propagator
Setting only a TracerProvider records spans within a single host but does not carry the trace across hosts. Install a TraceContext propagator (as above) for end-to-end distributed traces.