Folder Structure & Access Design¶

1. Purpose
2. Naming Convention
3. S3 Prefix Structure
4. Consumers View (AWS Transfer Family / SFTP)
5. On-Premises Active Directory Permissions
6. Onboarding Checklist — New Lab (labN)

1. Purpose¶

This document describes the Amazon S3 folder (prefix) structure used to exchange files between on-premises lab shares and the external SFTP user via AWS Transfer Family. It is intended as a reference for onboarding a new lab (labN) into the existing integration.

Note: In this document, lab1, lab2, ... labN refer to individual sequencing instruments integrated, not physical lab rooms or teams. Each instrument is onboarded using the identical pattern described below.

2. Naming Convention¶

Folder names are always written from the lab's point of view, not the consumer's.

3. S3 Prefix Structure¶

Each lab gets exactly two prefixes: one for the lab-to-consumer direction and one for the consumer-to-lab direction. The structure is identical for every lab, which is what makes onboarding a new lab mechanical.

s3://<bucket>/
  uploads/
    lab1/      <- Lab writes (on-prem)     | Consumer reads (SFTP)
    lab2/
    lab3/      <- future lab, same pattern
    labN/
  downloads/
    lab1/      <- Producer writes (SFTP)   | Lab reads (on-prem)
    lab2/
    lab3/
    labN/

4. Consumers View (AWS Transfer Family / SFTP)¶

Consumers connect once, via a single SFTP account. Rather than seeing the uploads/downloads S3 prefix names directly, the consumer's Transfer Family home directory is mapped so that it observes two direction-based folders, /inbound and /outbound, scoped only to those two trees (not the bucket root, and not any other bucket content). This means a new lab's folders appear automatically once created — no per-user SFTP configuration change is required when onboarding labN — while the consumer's visibility stays strictly limited to these two folders.

4.1 How the single Consumer account sees every lab folder¶

The consumer has one SFTP account, but it needs visibility into all current and future lab folders without any per-lab configuration on the SFTP side. This is achieved through two settings on the consumer's AWS Transfer Family user, independent of how many labs exist:

• Home directory mapping is scoped to only the uploads/ and downloads/ trees, not the bucket root — and the consumer does not see the S3 prefix names uploads/ or downloads/ directly. Instead, two logical folder mappings present the consumer with its own, direction-based folder names: /inbound and /outbound. The consumer's SFTP session therefore only ever sees these two top-level folders — never any other content that may exist elsewhere in the bucket — and within them, every lab subfolder (lab1, lab2 … labN) appears automatically as soon as it is created, with no mapping change required.

So when a consumer logs in via SFTP, it observes exactly two top-level folders, /inbound and /outbound, each containing one subfolder per lab (lab1, lab2 … labN). This keeps the consumer's own view direction-based and self-explanatory, while the on-prem/S3 side continues to use the lab-centric uploads/downloads naming described in Section 3.

• The IAM session policy issued to the consumer uses a wildcard resource pattern scoped to the same two trees, e.g. uploads/lab*/* and downloads/lab*/*, rather than naming each lab explicitly or granting bucket-wide access. This means a brand-new lab's folders become visible and usable by the consumer the moment they exist — no IAM or Transfer Family change is required to onboard labN — while access remains confined strictly to the uploads/ and downloads/ prefixes.

Practical effect: Creating the S3 prefixes for a new lab is the only action needed for a consumer to gain access to it under /inbound and /outbound, and a consumer can never see or reach any bucket content outside these two folders.

5. On-Premises Active Directory Permissions¶

Access to the on-prem SMB shares (which sync to/from the S3 prefixes above) is controlled entirely through Active Directory group membership and Samba share configuration on the Storage Gateway appliance. Authentication is via AD; authorization is via Samba share config applied per share.

5.1 Standard AD group per lab¶

Each lab has one AD group granting normal operational access to both of its folders. There is an optional read-only group for some users (QA), for visibility into what has been staged for the consumer and transferred by the external consumer client.

5.2 Summary of AD group naming¶

Note: If a lab has no read-only requirement, only the *-rw group needs to be created; the *-ro group is added later only if a genuine need arises, to avoid unnecessary group sprawl.

6. Onboarding Checklist — New Lab (labN)¶

When a new lab/instrument is added, complete the following steps in order. No changes are needed on the consumer's SFTP/IAM side, since access scoping is pattern-based (uploads/lab*/ and downloads/lab*/).

Note: Configuration changes are performed via IaC (Infrastructure as Code).