> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/data/customer-data-intake.md). # Customer Data Intake Amigo exposes two current intake workflows with different purposes and data lifecycles. A third, HMAC-signed route remains in the API contract for separately provisioned integrations but is not a self-service intake option. | Workflow | Best for | Where the upload appears | Processing contract | | ----------------------- | -------------------------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------ | | **Dataset intake** | Governed tabular snapshots and document collections | Dataset file catalog | Dataset schema, versioning, source batches, and optional processing | | **Upload links** | Collecting individual files from people without Platform credentials | Upload list for that link | Content-type allowlist and size limit; no dataset schema or batch | | **External HMAC route** | Separately approved legacy integrations | Separate integration-upload record | No dataset assignment; availability requires explicit Amigo provisioning | Do not treat these as interchangeable entry points into one pipeline. In particular, an upload-link file does not automatically become a dataset file. ## When to Use Each Workflow | Scenario | Recommended workflow | | ---------------------------------------------------------------------- | ----------------------------------------------------------------- | | An operator uploads a CSV snapshot against a defined schema | **Dataset intake** | | A Google Drive folder should feed a registered dataset | **Dataset intake source** | | A referral partner needs a browser link and has no Platform credential | **Upload link** | | A source exposes a stable API that Amigo should read repeatedly | [Connector](/data/connectors-and-ehr.md) | | An existing integration was explicitly provisioned for HMAC upload | **External HMAC route**, after confirming availability with Amigo | The Developer Console exposes dataset intake only where the Intake feature is enabled. Upload-link management is a separate Console surface. ## Dataset Intake Dataset intake is the governed path for files that need a named data contract, catalog status, and version history. Authenticated workspace routes let an operator register a dataset, upload or download files, list file status, and refresh a dataset from mapped sources. ### Dataset Contracts Every dataset has one of two ingestion modes: * **Snapshot** - A tabular contract with a primary key and typed fields. CSV is the current processing format. Although the registration contract also accepts `xls` and `xlsx` type tokens, the current snapshot validators and processor read CSV bytes; use CSV for processing workflows. * **Document** - A set of accepted filename extensions plus a document-processing configuration. The current native text extractor supports PDF, DOCX, and UTF-8 text such as Markdown. `OCR` and `hybrid` are accepted configuration values but are not current processing implementations and result in a failed processing verdict. Registering a file type does not prove that a downstream processor can interpret every file with that extension. Validate representative files before production use. ### Upload and Version Behavior An authenticated upload is associated with one registered dataset and is capped at 100 MB. Dataset contracts can record `max_size_mb`, but the current upload runtime does not enforce a lower per-dataset value; clients that rely on it must enforce it before upload. The service computes a content hash and returns an existing row for a byte-identical file already present in the same workspace and dataset. Snapshot versions are allocated at the dataset level. Document versions belong to a logical document: omit `document_id` to create a document, or provide an existing `document_id` to add a same-type version. Processing is configuration-dependent: * Snapshot uploads can be validated synchronously or land as `received` for asynchronous processing. * Document uploads land as `received` and require the document processor to reach a terminal status. * A successful upload response confirms storage and cataloging, not successful parsing, publication, or world-model projection. Malware scanning is deployment-dependent. When a scanner is configured, intake scans before persistence and rejects infected files or an unavailable scanner. When it is not configured, the upload proceeds without a malware verdict. An allowlisted extension, MIME type, or successful schema check is not a validation. ### Google Drive Sources A dataset source maps one or more Google Drive folders to registered datasets. Discovery occurs when an operator triggers a source sync or starts a dataset update; registration alone does not schedule continuous synchronization. A source sync creates a batch per mapped folder, downloads eligible files, and places newly landed files in the dataset catalog. Unchanged content is deduplicated. Processing a batch is a separate step unless the dataset-update workflow orchestrates it. The dataset-update endpoint combines source discovery, preparation, and publication behind a durable run. It applies only when an active Drive source maps to the requested dataset. Monitor the returned run rather than assuming that `202 Accepted` means the dataset is ready. ## Upload Links Upload links collect files without requiring the uploader to hold a Platform credential. An authenticated operator creates a link for one workspace and customer slug, chooses an expiration and upload-count limit, and shares the returned URL. The token in the URL is the bearer credential, so distribute and store it accordingly. The public upload route accepts one raw file per request, up to 100 MB, from a fixed MIME-type allowlist covering PDF, CSV, Markdown, JPEG, PNG, Word, and PowerPoint formats. It performs limited header and declared-content-type checks. Those checks do not guarantee that a document is well formed, safe to open, or supported by a downstream parser. Upload-link files have these boundaries: * They are stored as link-associated uploads, not dataset catalog rows. * They do not inherit a dataset schema, version chain, source batch, or dataset processing run. * A duplicate content hash is reported for operator awareness, but the new upload is still stored. * `scan_status` can be `skipped` when file validation is not configured. * Event emission after storage is best-effort. Upload success is not proof of downstream event delivery or world-model projection. Authenticated operators can list and download uploads for a specific link. The download path is scoped by workspace, link ID, and upload ID. It is classified as PHI-sensitive when audit logging is configured, but the download response is not a durable audit receipt. ## External HMAC Route The [External HMAC Upload](https://docs.amigo.ai/developer-guide/platform-api/data-world-model/intake#external-hmac-upload) is a separate raw-body path whose public contract is embedded in the Developer Guide. It requires a Platform bearer credential plus a customer-specific HMAC signature and writes a different upload record from dataset intake. The route is present in the public API schema, but the current service is initialized without customer HMAC secrets. The route is therefore dormant and rejects every customer slug. Do not build a new integration around it unless Amigo explicitly enables and provisions the path. Even when provisioned, this route does not register a dataset, validate against a dataset contract, create a source batch, or enter the dataset-update lifecycle. It verifies the supplied checksum, records upload metadata, and emits an intake event; it does not promise document parsing or direct entity creation. ## Security and Compliance Boundaries * Dataset, source, link-management, listing, and download routes require a workspace-scoped Platform credential. Public upload routes use the unexpired, unrevoked link token instead. * Treat upload-link URLs as secrets. Prefer the shortest practical expiry and upload count, and revoke a link when it is no longer needed. * Content-type checks, schema validation, malware scanning, parsing, and downstream publication are distinct controls. Success at one stage does not imply success at another. * The intake routes do not, by themselves, establish a HIPAA or HITRUST compliance posture. Deployment controls, contractual coverage, retention, residency, and operating procedures remain separate concerns. See [Compliance and Audit](/operations-and-safety/compliance.md) and [Data Residency](/platform-overview/data-residency.md). For endpoint-level request and response behavior, see the [Platform intake guide](https://docs.amigo.ai/developer-guide/platform-api/data-world-model/intake). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.amigo.ai/data/customer-data-intake.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.