Securing Data on GCP: Encryption and DLP

Ask whether your data on Google Cloud is encrypted and the answer is yes, before you have done anything. That is reassuring and beside the point. Encryption protects stored bytes from someone who gets hold of the disks. It does nothing about an analyst with read access exporting a table of phone numbers. Securing data on GCP is mostly about that second problem: knowing where the sensitive values are, and deciding who sees them in clear.

Four layers

The first two are largely done for you. The last two are your work.

Encryption at rest is already on

Google encrypts all customer content at rest, with no action on your part, using AES-256. Stored data is split into chunks, and each chunk is encrypted with its own data encryption key. Those keys are in turn encrypted by key encryption keys, an arrangement called envelope encryption (Google Cloud: default encryption at rest).

In this default mode Google owns and manages the keys. For most data, that is enough.

Customer-managed keys

With customer-managed encryption keys, usually shortened to CMEK, you create the key in Cloud KMS and the service uses your key to protect your data. You choose its location, set its rotation and control who may use it (Google Cloud: customer-managed encryption keys). Keys can be software keys, hardware keys, or keys held in an external system (Google Cloud: Cloud KMS overview).

In BigQuery, three rules apply (Google Cloud: customer-managed keys in BigQuery):

• The key must be in a location that matches the dataset.
• BigQuery's service account needs the Encrypter/Decrypter role on the key.
• A default key on a dataset applies to new tables only. Existing tables keep what they had.

CREATE SCHEMA `analytics-prod.finance`
OPTIONS (
  default_kms_key_name = 'projects/sec-keys/locations/asia-south1/keyRings/data/cryptoKeys/finance',
  location = 'asia-south1'
);

The control cuts both ways. If BigQuery loses access to the key, queries on those tables fail, no data can be added, and Google's documentation says plainly that data loss can occur. A key disabled by mistake is an outage.

So use customer-managed keys when a regulation, a contract or an internal policy requires that you hold the key. They do not make data more encrypted. They change who controls the key, and they add something that can be broken.

Encryption in transit

Google encrypts data in transit between users and its services and within its own networks (Google Cloud: encryption in transit). What remains yours is the connections you build: a pipeline reading an on-premises database, for instance. For private paths inside Google Cloud, see GCP networking basics.

Finding sensitive data

The service for this is Sensitive Data Protection. Older material calls it Cloud DLP, for Data Loss Prevention. Google's documentation states that Cloud DLP is now part of Sensitive Data Protection and that the API keeps its old name (Google Cloud: Sensitive Data Protection overview).

It does three things:

• Discovery profiles data across a project, a folder or a whole organization, in BigQuery, Cloud SQL and Cloud Storage, and reports where sensitive data lies.
• Inspection scans one resource in depth and reports every instance it finds.
• De-identification transforms the sensitive values.

What it looks for is defined by infoTypes: detectors for kinds of data such as email addresses, telephone numbers and credit card numbers. Google is careful about their limits. Built-in detectors are not perfectly accurate and cannot guarantee compliance with a regulation (Google Cloud: infoTypes and infoType detectors). Treat a scan as evidence, not as proof that nothing was missed.

De-identification: the techniques

The last column matters to data engineers. The last three techniques turn each distinct value into its own token, the same one every time. That keeps referential integrity: two tables can still be joined on the token, without either holding the real value (Google Cloud: transformation reference). Google advises format-preserving encryption only when the original format must be kept, and deterministic encryption otherwise.

The cryptographic hash here is not a bare hash. It is an HMAC-SHA-256 digest, which needs a key (Google Cloud: pseudonymization). The next section shows why that matters.

Masking in BigQuery

BigQuery can mask a column for some users and show it in clear to others, with no change to anyone's queries. It is built on column-level access control. A data policy attaches a masking rule to a column or to a policy tag, and users with the Masked Reader role see the masked form (Google Cloud: introduction to data masking).

The rules include nullify, a default value, an email mask that keeps the domain, the first or last four characters, and a SHA-256 hash that still allows joins. On that last rule the documentation carries a warning: SHA-256 uses no key, so someone can run every possible original value through it and find the one that matches.

For policy tags and how they are organised, see data catalog and governance on GCP.

Why a hash is not always protection

The script applies four techniques to three invented customers, in plain Python. It then does two things: joins orders to customers on a keyed hash, and plays the attacker, who knows that the phone numbers begin with 98765 and tries all 100,000 possibilities.

It demonstrates the techniques. It is not Sensitive Data Protection or BigQuery data masking.

import hashlib
import hmac

CUSTOMERS = [("C001", "asha.rao@example.com", "9876501234"),
             ("C002", "vikram.s@example.org", "9876543210"),
             ("C003", "meera@example.net", "9876577788")]
ORDERS = [("O-11", "9876543210", 1450), ("O-12", "9876501234", 320), ("O-13", "9876543210", 990)]
KEY = b"held-in-a-key-manager-not-in-code"


def email_mask(value):
    return "XXXXX@" + value.split("@")[1]


def last_four(value):
    return "XXXXX" + value[-4:]


def sha256(value):
    return hashlib.sha256(value.encode()).hexdigest()


def keyed_hash(value):
    return hmac.new(KEY, value.encode(), hashlib.sha256).hexdigest()


print("id    email mask         last four  sha-256       keyed hash")
for cid, email, phone in CUSTOMERS:
    print(f"{cid}  {email_mask(email):17}  {last_four(phone)}  {sha256(phone)[:12]}  {keyed_hash(phone)[:12]}")

# 1. a deterministic token still joins
customer_for = {keyed_hash(phone): cid for cid, _, phone in CUSTOMERS}
print()
print("orders joined to customers on the keyed hash of the phone number:")
for order_id, phone, amount in ORDERS:
    print(f"  {order_id}  {customer_for[keyed_hash(phone)]}  {amount}")

# 2. an attacker without the key tries every number that starts 98765
plain_tokens = {sha256(phone) for _, _, phone in CUSTOMERS}
keyed_tokens = {keyed_hash(phone) for _, _, phone in CUSTOMERS}
from_plain = from_keyed = 0
for n in range(100_000):
    guess = sha256(f"98765{n:05d}")
    from_plain += guess in plain_tokens
    from_keyed += guess in keyed_tokens
print()
print(f"tried 100,000 candidate numbers: recovered {from_plain} of 3 from SHA-256, "
      f"{from_keyed} of 3 from the keyed hash")

id    email mask         last four  sha-256       keyed hash
C001  XXXXX@example.com  XXXXX1234  ea9d3b768777  7639d36be6eb
C002  XXXXX@example.org  XXXXX3210  7619ee8cea49  54f839846ac3
C003  XXXXX@example.net  XXXXX7788  f998f13de275  480873cf5669

orders joined to customers on the keyed hash of the phone number:
  O-11  C002  1450
  O-12  C001  320
  O-13  C002  990

tried 100,000 candidate numbers: recovered 3 of 3 from SHA-256, 0 of 3 from the keyed hash

• The join works on tokens. Orders find their customers through the keyed hash, and neither side needs the real number.
• The plain hash gave up all three numbers. A phone number has too few possible values. Trying 100,000 of them takes a moment, and all ten billion ten-digit numbers are within reach of an ordinary computer.
• The keyed hash gave up none. Without the key, the attacker cannot compute a candidate token to compare.
• Partial masks leak too. The last four digits plus an email domain narrow a person down considerably.

A hash is protection only when the input is hard to guess or the hash is keyed. Names, phone numbers, dates of birth and postal codes are all easy to guess.

Who does what

1. Leave default encryption alone. It is on.
2. Add customer-managed keys only where required, and treat the key as production infrastructure.
3. Separate data by sensitivity into different datasets, and grant access to each on purpose. See GCP IAM.
4. Mask or restrict sensitive columns in the tables people query.
5. De-identify before data leaves the governed store: extracts, development copies, training sets.
6. Run discovery to find the sensitive data nobody declared.
7. Turn on the audit logs that record reads, so you can answer who looked. See logging and monitoring on GCP.

Common mistakes

• Treating encryption as privacy. An authorised reader sees everything in clear.
• Enabling customer-managed keys without owning their lifecycle.
• Hashing an identifier with a bare hash and calling the data anonymous.
• Masking in the dashboard while the raw table is open to everyone.
• Copying production data into a development project.
• Trusting a clean scan as proof.

Data governance and data quality are part of the Data Engineering course.

Sources

• Default encryption at rest (Google Cloud Documentation) — Google Cloud
• Customer-managed encryption keys (Google Cloud Documentation) — Google Cloud
• Cloud Key Management Service overview (Google Cloud Documentation) — Google Cloud
• Customer-managed Cloud KMS keys in BigQuery (Google Cloud Documentation) — Google Cloud
• Encryption in transit (Google Cloud Documentation) — Google Cloud
• Sensitive Data Protection overview (Google Cloud Documentation) — Google Cloud
• InfoTypes and infoType detectors (Google Cloud Documentation) — Google Cloud
• Transformation reference (Google Cloud Documentation) — Google Cloud
• Pseudonymization (Google Cloud Documentation) — Google Cloud
• Introduction to data masking (Google Cloud Documentation) — Google Cloud