Syncing Data to AWS S3
Background
Admins can configure Twingate to send audit logs, network events, and DNS filtering logs to their AWS S3 buckets every 5 minutes. This data can be programmatically consumed or sent to a SIEM, providing real-time visibility for troubleshooting or investigating Admin actions within Twingate or end user connections across the network.
Configuring your S3 Bucket
- Follow the AWS S3 User Guide to create your bucket
Configuring AWS S3 permissions
Twingate supports two methods for connecting Twingate to your AWS S3 bucket: an OIDC role and IAM user credentials.
- OIDC role (recommended): Using an OIDC role allows Twingate to access your S3 bucket using an assumed IAM role with temporary credentials. You don’t need to manually rotate Twingate’s access key, and Twingate’s access is kept to a minimum.
- IAM user credentials (not recommended): Twingate previously used static IAM user credentials to sync analytics to S3. This is not recommended as it uses a long-term, manually rotated credential, which doesn’t align with best security practices.
Option 1: OIDC Role (Recommended)
Create an IAM Identity Provider
- Log into AWS
- Navigate to IAM → Identity providers and add a provider
- Choose OpenID connect
- For the provider URL, enter your Network’s URL followed by
oidc
. For example, if your Network’s URL ishttps://acme.twingate.com/
, your provider URL should behttps://acme.twingate.com/oidc
- For the audience field, use your Network’s slug, e.g.
acme
for Networkacme.twingate.com
Create a policy
-
Navigate to IAM → Policies and create a new policy
-
Using the JSON policy editor, specify the following permissions:
{"Version": "2012-10-17","Statement": [{"Sid": "TwingateS3Sync","Effect": "Allow","Action": "s3:PutObject","Resource": "arn:aws:s3:::<S3BucketName>/*"}]} -
Give the policy a meaningful name like
TwingateS3Sync
Create an IAM Role
- Navigate to IAM → Roles and create a new role
- Choose “Web identity” as the trusted entity type
- For the identity provider, select the Twingate OIDC provider you created in the previous step
- For the audience field, use your Network’s slug, e.g.
acme
for Networkacme.twingate.com
- Add a condition
- For the key, enter your Network’s URL followed by
oidc:sub
. For example, if your Network’s URL isacme.twingate.com/
, your provider URL should beacme.twingate.com/oidc:sub
(do not includehttps://
at the start of the URL) - For the condition, select
StringEquals
- For the value, enter
events_sync
- For the key, enter your Network’s URL followed by
- Attach the IAM policy you created in the previous step
- Choose a name
- Create the role
Option 2: IAM User Credentials
- Open the AWS IAM console
- Go to Users
- Create a User who will have access to the S3 bucket
- Select the user who should have access
- Click Create Access Key
- Save the Access Key and Secret Access Key
- Grant your AWS S3 user access to the bucket. Make sure the user has
s3:ListBucket
ands3:PutObject
listed in their policy.
Configuring your AWS S3 sync in Twingate
-
Navigate to the Reports page under Settings
-
Click Sync to S3 Bucket to configure your sync
-
Choose your authentication method
-
If you chose OIDC role, enter your bucket name and Role ARN. If you chose IAM user credentials, enter your bucket name, Access Key, and Secret Access Key.
-
The first synced data should arrive within the next 10 minutes. Any subsequent audit log, network events, and DNS filtering logs will be synced to your S3 bucket every 5 minutes on an ongoing basis.
Troubleshooting
Why is my S3 sync failing?
If you’ve just configured your S3 sync and it immediately fails, you could be running into a configuration issue. Please check that your bucket name, access key, secret access key, and AWS user policies are correct. The AWS user trying to access the bucket should have s3:ListBucket
and s3:PutObject
policies.
What happens if there are no events to sync?
In the case that there are no events to sync, Twingate will not send files to the S3 bucket. You can confirm that the sync works in the Admin Console by navigating to Settings, then Reports. The S3 sync status will be displayed on the upper right corner as seen below.
I just performed an action that should be reflected in my data. Why am I not seeing it?
Events can take up to 10 minutes to be reflected in the sync.
Last updated 2 days ago