Securing Data on AWS: Encryption and KMS

Encryption on AWS is easy to switch on and easy to misunderstand. A bucket that says "encrypted" is protected against some threats and not at all against others, and the difference is rarely explained. For a data engineer the practical questions are three: what is encrypted by default, when is a KMS key worth using, and how does a pipeline keep working once one is.

Two kinds of protection

Encryption in transit protects data moving over a network, using TLS. It stops someone on the path from reading or altering it.

Encryption at rest protects data stored on disk. It stops someone who obtains the physical media, or a raw copy of the storage, from reading it.

Neither protects against a caller who is authorised. If a role may read a bucket, encryption at rest is transparent to it: S3 decrypts the object and hands it over. Encryption is one layer; access control is another, covered in AWS IAM explained.

What AWS KMS is

AWS Key Management Service creates and controls the keys used to encrypt data. Its defining property is that the keys do not leave it: KMS keys are protected by hardware security modules validated to FIPS 140-3 Security Level 3, and never leave the service unencrypted (KMS docs: overview). You do not download a KMS key. You ask KMS to use it.

There are three kinds of key (KMS docs: KMS keys):

Envelope encryption

KMS does not encrypt your gigabytes. Sending large data to a key service would be slow, and KMS operations work on small inputs. The technique that solves this is envelope encryption: encrypt the data with a data key, then encrypt the data key under another key (KMS docs: cryptography essentials).

The script performs it by hand, then lets S3 do the same thing. It runs against moto, a local emulator of KMS and S3; the AES encryption itself is real.

import os
import boto3
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from moto import mock_aws

os.environ.update(AWS_ACCESS_KEY_ID="testing", AWS_SECRET_ACCESS_KEY="testing",
                  AWS_DEFAULT_REGION="us-east-1")
record = b"customer_id=4417,card_last4=0921,amount=129.50"


@mock_aws
def main():
    kms, s3 = boto3.client("kms"), boto3.client("s3")
    key_id = kms.create_key(Description="data lake key")["KeyMetadata"]["KeyId"]

    # --- Envelope encryption, by hand ---
    # 1. Ask KMS for a data key: one copy in plaintext, one copy encrypted under the KMS key
    data_key = kms.generate_data_key(KeyId=key_id, KeySpec="AES_256")
    # 2. Encrypt the data locally with the plaintext data key, then forget that key
    nonce = os.urandom(12)
    ciphertext = AESGCM(data_key["Plaintext"]).encrypt(nonce, record, None)
    stored = {"nonce": nonce, "ciphertext": ciphertext, "encrypted_key": data_key["CiphertextBlob"]}
    del data_key
    print("stored with the data:", sorted(stored), "| ciphertext", len(stored["ciphertext"]), "bytes")
    print("record visible in ciphertext:", record in stored["ciphertext"])

    # 3. To read: send only the small encrypted key to KMS, get the data key back, decrypt locally
    plaintext_key = kms.decrypt(CiphertextBlob=stored["encrypted_key"])["Plaintext"]
    print("decrypted:", AESGCM(plaintext_key).decrypt(stored["nonce"], stored["ciphertext"], None).decode())

    # --- The same thing done for you: S3 server-side encryption with a KMS key ---
    s3.create_bucket(Bucket="demo-lake")
    s3.put_object(Bucket="demo-lake", Key="curated/payment.txt", Body=record,
                  ServerSideEncryption="aws:kms", SSEKMSKeyId=key_id, BucketKeyEnabled=True)
    head = s3.head_object(Bucket="demo-lake", Key="curated/payment.txt")
    print("S3 object encryption:", head["ServerSideEncryption"], "| key recorded:", head["SSEKMSKeyId"].endswith(key_id))
    print("read back through S3:", s3.get_object(Bucket="demo-lake", Key="curated/payment.txt")["Body"].read() == record)


main()

stored with the data: ['ciphertext', 'encrypted_key', 'nonce'] | ciphertext 62 bytes
record visible in ciphertext: False
decrypted: customer_id=4417,card_last4=0921,amount=129.50
S3 object encryption: aws:kms | key recorded: True
read back through S3: True

The steps, in words:

1. Ask KMS for a data key. It returns the key twice: once in plaintext, once encrypted under the KMS key.
2. Encrypt locally with the plaintext data key, then discard it. The record is no longer readable in the ciphertext.
3. Store three things together: the ciphertext, the nonce, and the encrypted data key. None of them is secret alone.
4. To read, send only the small encrypted data key to KMS. If the caller is permitted, KMS returns the plaintext key, and the data is decrypted locally.

The data never travels to KMS, and the KMS key never leaves it. Whoever holds the stored file can do nothing without permission to call KMS, and every such call can be logged.

The last two output lines show S3 doing this on your behalf. With server-side encryption set to aws:kms, S3 obtains the data key, encrypts the object, records which key was used, and decrypts on read.

Encryption options in S3

Since January 2023 S3 has applied SSE-S3 as the base level of encryption for every bucket, automatically (S3 docs: server-side encryption). So "is my data lake encrypted at rest?" has the answer yes, without any action. The real question is whether SSE-S3 is enough.

When a KMS key is worth it

SSE-KMS with a customer managed key adds things SSE-S3 cannot:

• A second lock. With SSE-KMS, downloading an object needs kms:Decrypt on the key as well as S3 permission; uploading needs kms:GenerateDataKey (S3 docs: SSE-KMS). A principal with broad S3 access but no key permission cannot read the data.
• A key policy. Every KMS key has exactly one key policy, the primary way to control access to it. No principal has permission unless explicitly allowed (KMS docs: key policies).
• An audit trail of every use of the key.
• Control of the lifecycle: rotation, disabling, deletion.

Use it for sensitive zones, personal data, financial data, anything under regulation, and keep SSE-S3 for the rest. A reasonable layout is one customer managed key per sensitivity level or per domain, not one per bucket.

Two operational consequences follow.

Every reader needs key permission. The most common failure after enabling SSE-KMS is a pipeline that had S3 access and now gets access denied. The Glue job's role, the Athena users, the Redshift role and any cross-account consumers all need to be allowed on the key.

KMS requests cost money and have quotas. Without a bucket key, S3 uses an individual data key per object and calls KMS for each request. A job reading millions of small objects makes millions of KMS calls. S3 Bucket Keys reduce that traffic, and AWS states they can cut KMS request costs by up to 99 percent (S3 docs: bucket keys). Enable them on any SSE-KMS bucket used for analytics.

In transit

Calls to AWS services made through the SDKs and CLI use HTTPS by default. To make it a rule and not a habit, AWS recommends allowing only encrypted connections by using the aws:SecureTransport condition in bucket policies (S3 docs: security best practices). For databases, require TLS on connections, and keep traffic on private paths where possible; see VPC basics for data engineers.

Beyond S3

Each data service has its own encryption settings, and they should use the same keys by design, not by accident. Glue, for example, uses security configurations to encrypt the data a job writes to S3, its logs and its job bookmarks. Check the equivalent for Athena query results, Redshift, Kinesis streams and anything else that stores a copy of the data. Temporary and derived data is where unencrypted or wrongly keyed copies tend to appear.

Mistakes that hurt

• Deleting a key. The guide calls deletion destructive and irreversible. Data encrypted under a deleted key cannot be recovered. Disable a key if in doubt; schedule deletion only when certain.
• One key for everything, so that key access cannot distinguish sensitive from ordinary data.
• Enabling SSE-KMS without updating the roles that read the data.
• No bucket key on high-volume buckets.
• Treating encryption as access control. Encrypted data that everyone is allowed to decrypt is not protected from them.
• Forgetting the copies: query results, staging areas, exports, logs.

For controlling who sees which tables and columns once access is granted, see AWS Lake Formation; for where encryption fits in the lake's layout, building a data lake on AWS.

Data governance and security for cloud data pipelines are covered in the Data Engineering course.

Sources

• AWS Key Management Service (AWS KMS Developer Guide) — AWS Documentation
• AWS KMS cryptography essentials (AWS KMS Developer Guide) — AWS Documentation
• AWS KMS keys (AWS KMS Developer Guide) — AWS Documentation
• Key policies in AWS KMS (AWS KMS Developer Guide) — AWS Documentation
• Protecting data with server-side encryption (Amazon S3 User Guide) — AWS Documentation
• Using server-side encryption with AWS KMS keys (Amazon S3 User Guide) — AWS Documentation
• Reducing the cost of SSE-KMS with Amazon S3 Bucket Keys (Amazon S3 User Guide) — AWS Documentation
• Security best practices for Amazon S3 (Amazon S3 User Guide) — AWS Documentation