Azure Data Lake Storage Gen2 Explained

Azure Data Lake Storage Gen2 is the storage layer under almost every analytics platform on Azure: Data Factory lands data in it, and Synapse and Databricks read from it. It is also widely misunderstood, because it is not really a product. This guide explains what it is, how its access control works, and how to organise a lake on it.

A note on the name. Microsoft's current documentation calls it simply Azure Data Lake Storage. "Gen2" survives in connector names and in everyday speech, and means the same thing.

One setting on Blob Storage

Microsoft states it directly: Azure Data Lake Storage is not a dedicated service or account type. It is a set of capabilities on the Blob Storage service of a storage account, unlocked by enabling the hierarchical namespace setting (Microsoft Learn: Data Lake Storage overview).

Everything else follows from that:

• Priced at Blob Storage levels, with Blob Storage's tiering and lifecycle management.
• The data are blobs. What you call a file is stored as a blob; what Blob Storage calls a container, the data lake documentation calls a file system.
• A second endpoint. Besides the blob endpoint, the account has one at dfs.core.windows.net that understands directories.

What the hierarchical namespace changes

Plain object storage has no folders. A blob named sales/2026/10/orders.parquet is one object with slashes in its name. "Renaming the folder" means copying and deleting every object that starts with that prefix.

With a hierarchical namespace, directories are real. Renaming or deleting a directory is a single atomic metadata operation on the directory (Microsoft Learn: hierarchical namespace).

That sounds like a convenience and is in fact the reason the feature exists. Spark and Hive write their output to a temporary location and rename it when the job finishes. Microsoft notes that without a hierarchical namespace, that rename can take longer than the analytics job itself.

Two cautions from the same page:

• It is one-way. Once enabled on an account, it cannot be reverted to a flat namespace.
• Not every workload benefits. Backups and image storage are given as examples that gain nothing.

Addressing data

Analytics engines reach the lake through the Azure Blob File System driver, with addresses of this form (Microsoft Learn: ABFS URI):

abfss://raw@contosolake.dfs.core.windows.net/sales/2026/10/02/orders.parquet

The parts are the scheme, the file system, the account, and the path. The second s in the scheme means the connection always uses TLS.

Access control: two layers

This is the part that causes the most trouble, and it rewards ten minutes of attention. Two mechanisms apply to identities in Microsoft Entra ID (Microsoft Learn: access control model):

Role assignments are coarse. A role such as Storage Blob Data Reader or Storage Blob Data Contributor applies to all the data in an account or a container.

Access control lists are fine. Each directory and file carries entries granting Read, Write and Execute to users and groups, in the POSIX style (Microsoft Learn: ACLs).

The rule that decides everything is the order of evaluation. Role assignments are evaluated first. ACLs are consulted only if the role has not already granted the access.

The script below is a model of that rule. It works out which ACL entries an identity needs at each level of a small tree, for seven operations and three roles, and compares the result with eleven rows of the table in Microsoft's documentation. It is a model written for this article, not Azure Storage.

# A model of the documented rule: role assignments are evaluated first, and ACLs only for what the role does not grant.
LEVELS = ["/", "Oregon/", "Portland/", "Data.txt"]

# each operation is a list of needs: (kind of access, index of the item acted on, permissions needed on that item)
OPERATIONS = {
    "Read Data.txt":          [("read", 3, "R")],
    "Append to Data.txt":     [("read", 3, "R"), ("write", 3, "W")],
    "Delete Data.txt":        [("write", 2, "WX")],          # deleting changes the parent directory
    "Create or update file":  [("write", 2, "WX")],
    "List /":                 [("read", 0, "RX")],
    "List /Oregon/":          [("read", 1, "RX")],
    "List /Oregon/Portland/": [("read", 2, "RX")],
}
ROLE_GRANTS = {"None": set(), "Storage Blob Data Reader": {"read"},
               "Storage Blob Data Contributor": {"read", "write"}}


def acl_needed(role, operation):
    """The ACL entry required at each level, for whatever the role does not already grant."""
    needed = [set() for _ in LEVELS]
    for kind, target, permissions in OPERATIONS[operation]:
        if kind in ROLE_GRANTS[role]:
            continue
        for ancestor in range(target):
            needed[ancestor].add("X")                         # Execute on every directory above the item
        needed[target] |= set(permissions)
    return ["".join(p if p in n else "-" for p in "RWX") if n else "N/A" for n in needed]


def allowed(role, acl, operation):
    return all(want == "N/A" or all(p == "-" or p in have for p in want)
               for want, have in zip(acl_needed(role, operation), acl))


PUBLISHED = {   # rows of Microsoft's permissions table: (operation, role) -> entries for /, Oregon/, Portland/, Data.txt
    ("Read Data.txt", "Storage Blob Data Reader"):       ["N/A", "N/A", "N/A", "N/A"],
    ("Read Data.txt", "None"):                           ["--X", "--X", "--X", "R--"],
    ("Append to Data.txt", "Storage Blob Data Reader"):  ["--X", "--X", "--X", "-W-"],
    ("Append to Data.txt", "None"):                      ["--X", "--X", "--X", "RW-"],
    ("Delete Data.txt", "Storage Blob Data Reader"):     ["--X", "--X", "-WX", "N/A"],
    ("Delete Data.txt", "None"):                         ["--X", "--X", "-WX", "N/A"],
    ("Create or update file", "Storage Blob Data Contributor"): ["N/A", "N/A", "N/A", "N/A"],
    ("Create or update file", "None"):                   ["--X", "--X", "-WX", "N/A"],
    ("List /", "None"):                                  ["R-X", "N/A", "N/A", "N/A"],
    ("List /Oregon/", "None"):                           ["--X", "R-X", "N/A", "N/A"],
    ("List /Oregon/Portland/", "None"):                  ["--X", "--X", "R-X", "N/A"],
}
print(f"{'operation':24} {'role':30} " + " ".join(f"{level:>9}" for level in LEVELS))
agree = True
for (operation, role), published in PUBLISHED.items():
    entries = acl_needed(role, operation)
    agree = agree and entries == published
    print(f"{operation:24} {role:30} " + " ".join(f"{e:>9}" for e in entries))
print("every row equals the published table:", agree)

print("\nfour cases:")
cases = [
    ("no role, R on the file only",                      "None", ["---", "---", "---", "R--"], "Read Data.txt"),
    ("no role, X on each directory and R on the file",   "None", ["--X", "--X", "--X", "R--"], "Read Data.txt"),
    ("Reader role, ACL on the file set to no permissions", "Storage Blob Data Reader", ["---", "---", "---", "---"], "Read Data.txt"),
    ("Reader role, W on the file, no X on Portland/",    "Storage Blob Data Reader", ["--X", "--X", "---", "-W-"], "Append to Data.txt"),
]
for label, role, acl, operation in cases:
    print(f"  {label:52} {operation:20} {'allowed' if allowed(role, acl, operation) else 'denied'}")

operation                role                                   /   Oregon/ Portland/  Data.txt
Read Data.txt            Storage Blob Data Reader             N/A       N/A       N/A       N/A
Read Data.txt            None                                 --X       --X       --X       R--
Append to Data.txt       Storage Blob Data Reader             --X       --X       --X       -W-
Append to Data.txt       None                                 --X       --X       --X       RW-
Delete Data.txt          Storage Blob Data Reader             --X       --X       -WX       N/A
Delete Data.txt          None                                 --X       --X       -WX       N/A
Create or update file    Storage Blob Data Contributor        N/A       N/A       N/A       N/A
Create or update file    None                                 --X       --X       -WX       N/A
List /                   None                                 R-X       N/A       N/A       N/A
List /Oregon/            None                                 --X       R-X       N/A       N/A
List /Oregon/Portland/   None                                 --X       --X       R-X       N/A
every row equals the published table: True

four cases:
  no role, R on the file only                          Read Data.txt        denied
  no role, X on each directory and R on the file       Read Data.txt        allowed
  Reader role, ACL on the file set to no permissions   Read Data.txt        allowed
  Reader role, W on the file, no X on Portland/        Append to Data.txt   denied

All eleven rows match the published table. Four lessons are in it.

Execute on every directory above. To read one file with ACLs alone, an identity needs Execute on the root, on each directory down the path, and Read on the file. The first of the four cases is the classic mistake: Read on the file and nothing above it. Access is denied, and the file's own ACL looks correct.

A role removes the need for ACLs, for what it covers. With the Reader role, reading and listing need no ACL entries at all.

A role does not cover what it does not grant. A Reader who must append still needs Write on the file and Execute on every directory above it. The fourth case fails for one missing Execute.

An ACL cannot take away what a role gives. The third case sets the file's ACL to no permissions, and the read is still allowed. Microsoft flags this as important: because roles are evaluated first, ACLs are ignored once a role grants access. If some folders must be hidden from a group, do not give that group an account-wide data role.

Three more points from the documentation:

• Use groups in ACL entries, never individuals. Membership can then change without rewriting ACLs across a directory tree.
• Default ACLs apply only to new items. A default ACL on a directory is a template for children created afterwards. It does not change what already exists.
• Keys and shared access signatures bypass both layers. With Shared Key or an account SAS, roles and ACLs have no effect. Roles such as Owner and Contributor do not grant access to data, and they can obtain the account keys, which do.

For how Data Factory fits into this, see managed identities in Azure Data Factory.

Laying out the lake

Zones. A common convention is a container or top-level folder per stage of refinement: raw data as received, cleaned data, and data shaped for consumers. Data lake vs data warehouse vs lakehouse covers the idea.

Directories. Microsoft's best practices suggest putting the stable, security-relevant parts of the path first and the date last (Microsoft Learn: best practices):

{Region}/{SubjectMatter}/{yyyy}/{mm}/{dd}/{hh}/

The reasoning ties back to access control. With region and subject first, one ACL on a directory secures everything beneath it. With the date first, the same permission would have to be applied under every day's directory.

For batch processing, the same page suggests separate In, Out and Bad directories under each subject, so that a file's state is visible from where it sits.

Files: format and size

• Format. Avro for write-heavy patterns and whole-record retrieval, such as events from a message bus. Parquet or ORC for read-heavy analytics that touch a subset of columns.
• Size. Larger files perform better and cost less. Microsoft's guidance is files of roughly 256 MB to 100 GB.
• Why small files hurt. Engines pay a per-file overhead for listing and checking access, and read and write operations are billed in 4 MB increments, so an operation on a file of a few kilobytes is charged like one on 4 MB.
• Compact. Where the source produces many small files, a process that merges them into larger Parquet files belongs in the pipeline.

Table formats such as Delta Lake sit on top of these files and add transactions.

Cost: access tiers

Because the data are blobs, Blob Storage's tiers apply (Microsoft Learn: access tiers):

Each step down lowers the storage price and raises the access price. Moving or deleting data before the minimum period brings an early deletion charge. Archive is offline: a blob there cannot be read until it is rehydrated, which can take up to 15 hours.

For a lake the usual pattern is hot for the current period, cooler tiers for history that is seldom queried, moved by a lifecycle policy. Never archive data that a scheduled job still reads.

With Data Factory

Data Factory connects through a linked service for Azure Data Lake Storage Gen2, ideally with the factory's managed identity: Storage Blob Data Reader to read, Storage Blob Data Contributor to write. The Copy activity guide covers loading it. The equivalent service on AWS is covered in Amazon S3 explained.

Mistakes to avoid

• Granting Read on a file and forgetting Execute on its parents.
• Giving a group an account-wide data role, then trying to hide folders with ACLs.
• ACL entries for named individuals.
• Dates at the top of the directory tree.
• Millions of small files.
• Enabling the hierarchical namespace on an account used for something else, without checking, since it cannot be undone.
• Distributing account keys. They bypass every control described here.

Azure data lake design and cloud-based data warehousing are covered in the Data Engineering course.

Sources

• Azure Data Lake Storage overview (Microsoft Learn) — Microsoft Learn
• Azure Data Lake Storage hierarchical namespace (Microsoft Learn) — Microsoft Learn
• Use the Azure Data Lake Storage URI (Microsoft Learn) — Microsoft Learn
• Access control model for Azure Data Lake Storage (Microsoft Learn) — Microsoft Learn
• Access control lists (ACLs) in Azure Data Lake Storage (Microsoft Learn) — Microsoft Learn
• Best practices for using Azure Data Lake Storage (Microsoft Learn) — Microsoft Learn
• Access tiers for blob data (Microsoft Learn) — Microsoft Learn