Tenant provisioning

This is an example of an IT workflow. When a new tenant is requested, a manager has to approve it, and three subsystems are provisioned in parallel before the tenant is verified and welcomed. Two of the three are complex enough to be workflows of their own. If anything fails after approval, everything provisioned so far is torn down, including whatever the children built.

Children workflow definitions#

Each child keeps its own status, compensation stack, and timers. WithOutput names the step whose output the parent reads back.

provisionDatabase, err := workflow.New("provision-database",
	workflow.WithOutput("credentials"),
	workflow.WithSteps(
		workflow.Step("create-cluster",
			workflow.WithRun(createCluster),
			workflow.WithCompensate(deleteCluster),
			workflow.WithStepTimeout(10*time.Minute),
			// Only hosts advertising "cloud-api" run this, and its undo runs on one too
			workflow.WithRequiredCapability("cloud-api"),
		),
		workflow.Step("create-schema", workflow.WithRun(createSchema)),
		workflow.Step("credentials",
			workflow.WithRun(issueCredentials),
			workflow.WithCompensate(revokeCredentials),
		),
	),
)

provisionStorage, err := workflow.New("provision-storage",
	workflow.WithOutput("bucket"),
	workflow.WithSteps(
		workflow.Step("bucket", workflow.WithRun(createBucket), workflow.WithCompensate(deleteBucket)),
		workflow.Step("policy", workflow.WithRun(attachPolicy), workflow.WithCompensate(detachPolicy)),
	),
)

Parent workflow definition#

onboarding, err := workflow.New("tenant-onboarding",
	workflow.WithTimeout(7*24*time.Hour),
	workflow.WithSteps(
		// Opens the review ticket
		// Its compensation closes it with the reason
		workflow.Step("request-review",
			workflow.WithRun(openReviewTicket),
			workflow.WithCompensate(closeReviewTicket),
		),

		// Parks the instance until a manager approves, or three days pass
		workflow.WaitForEvent("approval",
			workflow.WithEventTimeout(72*time.Hour),
		),

		// The approval payload says whether it was approved
		workflow.Step("approved", workflow.WithRun(readApproval)),

		// Three subsystems at once, two of them workflows in their own right
		workflow.Parallel("provision",
			workflow.Child("database", workflow.WithDefinition(provisionDatabase)),
			workflow.Child("storage", workflow.WithDefinition(provisionStorage)),
			workflow.Step("dns", workflow.WithRun(createDNS), workflow.WithCompensate(deleteDNS)),
		),

		workflow.Step("verify",
			workflow.WithRun(verifyTenant),
			workflow.WithInputFrom("provision"),
			workflow.WithSkipIf("approved", false),
		),

		workflow.Step("welcome", workflow.WithRun(sendWelcome), workflow.WithOptional()),
	),
)

err = host.RegisterBuiltInActor(provisionDatabase)
err = host.RegisterBuiltInActor(provisionStorage)
err = host.RegisterBuiltInActor(onboarding)

Handlers#

readApproval turns the event payload into the boolean the condition reads:

type approvalPayload struct {
	Approved bool   `json:"approved"`
	By       string `json:"by"`
	Note     string `json:"note,omitempty"`
}

func readApproval(ctx context.Context, t workflow.Task) (any, error) {
	var payload approvalPayload
	err := t.DecodeOutput("approval", &payload)
	if err != nil {
		return nil, errors.Join(actor.ErrJobPermanentFailure, err)
	}

	// A rejection is a recorded output, which is what WithSkipIf reads on the step below
	return payload.Approved, nil
}

verifyTenant reads the parallel group’s output through WithInputFrom("provision"): the database child’s credentials under database, the storage child’s bucket under storage, and the DNS record under dns.

type provisionOutput struct {
	Database databaseCredentials `json:"database"`
	Storage  bucketRef           `json:"storage"`
	DNS      dnsRecord           `json:"dns"`
}

func verifyTenant(ctx context.Context, t workflow.Task) (any, error) {
	var provisioned provisionOutput
	err := t.DecodeOutput("provision", &provisioned)
	if err != nil {
		return nil, errors.Join(actor.ErrJobPermanentFailure, err)
	}

	return nil, healthcheck.Tenant(ctx, provisioned.Database, provisioned.Storage, provisioned.DNS)
}

Starting and driving the workflow#

svc := onboarding.Service(host.Service())

// One instance per tenant, so a retried request finds the run already in flight
id, _, err := svc.Start(ctx, tenantRequest{Name: name, Plan: plan}, workflow.WithInstanceID(tenantID))

When the workflow waits for the external approval, notify the manager in some ways (e.g. an email, a Slack/Teams notification, etc). The approval arrives from outside, as a single call, for example through a webhook that’s invoked from the Slack/Teams message:

err = svc.RaiseEvent(ctx, tenantID, "approval", approvalPayload{Approved: true, By: user, Note: note})

Example failures#

Example failures and how they are handled:

Nobody approves within three days.
The event timeout elapses. The stack holds one entry, request-review, so closeReviewTicket runs with the cause event "approval" timed out and the instance terminates failed. Nothing was provisioned.
The manager rejects.
RaiseEvent carries Approved: false, the wait completes, readApproval returns false, and WithSkipIf skips verify. The provision group still runs, since in this graph a rejected tenant is provisioned in a sandbox, and the instance completes with verify recorded as skipped.
Storage’s policy step fails permanently after the database child has completed.
The storage child deletes its bucket and reports failure, and FailFast fails the provision group.
The parent’s stack holds the group and request-review.
Rolling back the group asks the completed database child to undo itself, which revokes its credentials and deletes its cluster, and runs deleteDNS at the same time. The ticket is closed last. The parent is failed with compensation: completed.
```
// Both children, in their terminal states
page, err := dbSvc.List(ctx, &workflow.ListOptions{Parent: tenantID})
```
The cloud API is under maintenance mid-provisioning.
An operator suspends the parent:
```
err = svc.Suspend(ctx, tenantID, "cloud maintenance")
```
That stops it starting anything new. The children keep running, and their results wait. The seven-day instance timeout is paused, and Resume continues where the instance left off.