Customer Data Intake

Workspace-scoped dataset ingestion and token-based file collection, with separate storage, processing, security, and rollout boundaries for each path.

Amigo exposes two current intake workflows with different purposes and data lifecycles. A third, HMAC-signed route remains in the API contract for separately provisioned integrations but is not a self-service intake option.

Workflow	Best for	Where the upload appears	Processing contract
Dataset intake	Governed tabular snapshots and document collections	Dataset file catalog	Dataset schema, versioning, source batches, and optional processing
Upload links	Collecting individual files from people without Platform credentials	Upload list for that link	Content-type allowlist and size limit; no dataset schema or batch
External HMAC route	Separately approved legacy integrations	Separate integration-upload record	No dataset assignment; availability requires explicit Amigo provisioning

Do not treat these as interchangeable entry points into one pipeline. In particular, an upload-link file does not automatically become a dataset file.

When to Use Each Workflow

Scenario	Recommended workflow
An operator uploads a CSV snapshot against a defined schema	Dataset intake
A Google Drive folder should feed a registered dataset	Dataset intake source
A referral partner needs a browser link and has no Platform credential	Upload link
A source exposes a stable API that Amigo should read repeatedly	Connector
An existing integration was explicitly provisioned for HMAC upload	External HMAC route, after confirming availability with Amigo

The Developer Console exposes dataset intake only where the Intake feature is enabled. Upload-link management is a separate Console surface.

Dataset Intake

Dataset intake is the governed path for files that need a named data contract, catalog status, and version history. Authenticated workspace routes let an operator register a dataset, upload or download files, list file status, and refresh a dataset from mapped sources.

Dataset Contracts

Every dataset has one of two ingestion modes:

• Snapshot - A tabular contract with a primary key and typed fields. CSV is the current processing format. Although the registration contract also accepts xls and xlsx type tokens, the current snapshot validators and processor read CSV bytes; use CSV for processing workflows.
• Document - A set of accepted filename extensions plus a document-processing configuration. The current native text extractor supports PDF, DOCX, and UTF-8 text such as Markdown. OCR and hybrid are accepted configuration values but are not current processing implementations and result in a failed processing verdict.

Registering a file type does not prove that a downstream processor can interpret every file with that extension. Validate representative files before production use.

Upload and Version Behavior

An authenticated upload is associated with one registered dataset and is capped at 100 MB. Dataset contracts can record max_size_mb, but the current upload runtime does not enforce a lower per-dataset value; clients that rely on it must enforce it before upload. The service computes a content hash and returns an existing row for a byte-identical file already present in the same workspace and dataset.

Snapshot versions are allocated at the dataset level. Document versions belong to a logical document: omit document_id to create a document, or provide an existing document_id to add a same-type version.

Processing is configuration-dependent:

• Snapshot uploads can be validated synchronously or land as received for asynchronous processing.
• Document uploads land as received and require the document processor to reach a terminal status.
• A successful upload response confirms storage and cataloging, not successful parsing, publication, or world-model projection.

Malware scanning is deployment-dependent. When a scanner is configured, intake scans before persistence and rejects infected files or an unavailable scanner. When it is not configured, the upload proceeds without a malware verdict. An allowlisted extension, MIME type, or successful schema check is not a validation.

Google Drive Sources

A dataset source maps one or more Google Drive folders to registered datasets. Discovery occurs when an operator triggers a source sync or starts a dataset update; registration alone does not schedule continuous synchronization.

A source sync creates a batch per mapped folder, downloads eligible files, and places newly landed files in the dataset catalog. Unchanged content is deduplicated. Processing a batch is a separate step unless the dataset-update workflow orchestrates it.

The dataset-update endpoint combines source discovery, preparation, and publication behind a durable run. It applies only when an active Drive source maps to the requested dataset. Monitor the returned run rather than assuming that 202 Accepted means the dataset is ready.

Upload Links

Upload links collect files without requiring the uploader to hold a Platform credential. An authenticated operator creates a link for one workspace and customer slug, chooses an expiration and upload-count limit, and shares the returned URL. The token in the URL is the bearer credential, so distribute and store it accordingly.

The public upload route accepts one raw file per request, up to 100 MB, from a fixed MIME-type allowlist covering PDF, CSV, Markdown, JPEG, PNG, Word, and PowerPoint formats. It performs limited header and declared-content-type checks. Those checks do not guarantee that a document is well formed, safe to open, or supported by a downstream parser.

Upload-link files have these boundaries:

• They are stored as link-associated uploads, not dataset catalog rows.
• They do not inherit a dataset schema, version chain, source batch, or dataset processing run.
• A duplicate content hash is reported for operator awareness, but the new upload is still stored.
• scan_status can be skipped when file validation is not configured.
• Event emission after storage is best-effort. Upload success is not proof of downstream event delivery or world-model projection.

Authenticated operators can list and download uploads for a specific link. The download path is scoped by workspace, link ID, and upload ID. It is classified as PHI-sensitive when audit logging is configured, but the download response is not a durable audit receipt.

External HMAC Route

The External HMAC Upload is a separate raw-body path whose public contract is embedded in the Developer Guide. It requires a Platform bearer credential plus a customer-specific HMAC signature and writes a different upload record from dataset intake.

The route is present in the public API schema, but the current service is initialized without customer HMAC secrets. The route is therefore dormant and rejects every customer slug. Do not build a new integration around it unless Amigo explicitly enables and provisions the path.

Even when provisioned, this route does not register a dataset, validate against a dataset contract, create a source batch, or enter the dataset-update lifecycle. It verifies the supplied checksum, records upload metadata, and emits an intake event; it does not promise document parsing or direct entity creation.

Security and Compliance Boundaries

• Dataset, source, link-management, listing, and download routes require a workspace-scoped Platform credential. Public upload routes use the unexpired, unrevoked link token instead.
• Treat upload-link URLs as secrets. Prefer the shortest practical expiry and upload count, and revoke a link when it is no longer needed.
• Content-type checks, schema validation, malware scanning, parsing, and downstream publication are distinct controls. Success at one stage does not imply success at another.
• The intake routes do not, by themselves, establish a HIPAA or HITRUST compliance posture. Deployment controls, contractual coverage, retention, residency, and operating procedures remain separate concerns. See Compliance and Audit and Data Residency.

For endpoint-level request and response behavior, see the Platform intake guide.