Skip to main content

AWS CloudTrail (S3 + SQS)

AWS CloudTrail organisation trail delivered to S3 and announced on SQS. Reads management events, data events and VPC flow logs for every member account from one queue, with the queue as the cursor so nothing is lost across a restart.

  • Connector id: aws_cloudtrail_s3
  • Category: cloud

Configuration​

FieldLabelTypeRequiredDefaultNotes
regionAWS Regionstringyesus-east-1—
queue_urlSQS queue URLstringyes—The queue your trail bucket's event notification publishes to. Either S3 → SQS directly, or S3 → SNS → SQS; both envelopes are read.
access_keyAccess Key IDstringno—Leave blank to use the runtime IAM role / instance profile.
secret_keySecret Access Keysecretno—encrypted at rest · Required only when supplying a static access key above.
include_data_eventsInclude data eventsbooleannoTrueS3 / Lambda / DynamoDB data events. These answer 'what did they read', and are the highest-volume thing in the trail — most land at info severity.
include_flow_logsInclude VPC flow logsbooleannoTrueVPC flow logs published to the same bucket (key prefix .../vpcflowlogs/...).
visibility_timeout_secondsReceive visibility timeout (seconds)numberno300How long a received message is hidden from other consumers. Messages are not deleted until ingest confirms their events, so this has to cover a poll and the push that follows it — not just the poll. AiSOC stops receiving once it has spent half the timeout, and a message whose visibility lapses first is redelivered and de-duplicated rather than lost. Set a redrive policy on the queue so a notification naming an unreadable object is quarantined to a dead-letter queue instead of retried forever.
Credential handling

secret_key is encrypted at rest by the credential vault before the row is written, and decrypted only at poll time. The value is never logged and never returned by the API — a read of the saved connector shows the field as set, not its contents. See credential vault.

Setup​

  1. In the console, go to Connectors → Add connector and pick AWS CloudTrail (S3 + SQS).
  2. Fill in the fields above and run Test connection. The test is read-only and does not save anything.
  3. Save. Polling starts on a jittered schedule within about a minute; the connector row shows last_sync once the first poll completes.
If last_sync stays empty

The first poll is the honest signal that the credential works. A connector that saves cleanly and never syncs usually means the credential lacks a scope the vendor requires for reads — check the connectors service log for the vendor's own error, which is passed through verbatim rather than summarised.

Vendor-specific setup notes go here: which console page issues the credential, which scopes to grant, rate limits, and anything the schema cannot express. This section is preserved when the page is regenerated.