Import from your own storage bucket

If your documents already sit in object storage (S3, or anything that speaks the same API), point us at the bucket instead of uploading files one at a time. We read them where they are. You register a bucket once under Your storage, then import from it as often as you like.

What to create on your side#

1. A folder inside the bucket holding the files you want indexed, handbook/policies/ rather than the whole bucket, unless the whole bucket is what you mean. It is the prefix, if your storage calls it that.
2. An access key that can only read that folder. An import lists the folder and reads the objects under it, and that is all it can do. We never write to your bucket, never delete anything in it, and never read outside the folder you registered.

Then Register a bucket and fill in:

• Storage host. The host your storage answers on, on its own: no scheme in front, no bucket and no path in it. We always connect over https, because anything else would put your access key on the wire in the clear. An address on a private network is refused when you register it, and checked again, against the address we actually reach, on every import.
• Bucket, and Region if your storage has regions.
• Folder. A boundary, not just a starting point. An import may narrow it (that is what Only this folder does on the import itself) but never widen it, so a source registered at handbook/ can never be talked into reading the rest of the bucket.
• Access key id and Secret access key.

What happens to the secret#

The secret is write-only. It is stored encrypted, and nothing in the product reads it back: not the list of sources, not the source itself, not the secrets API. Afterwards you can see that a key is stored, never what it is, which is why the field is empty every time you open the source again.

So we cannot recover one for you. To rotate a key, type the new secret over the old one; leaving the field empty keeps the key we already hold. There is no way to remove it and leave the source in place, because a source without a key cannot import anything.

One import is one document#

Every file under the folder becomes part of a single document, the same way a whole website crawl becomes one document made of many pages. Each passage still carries the file it came from, so an answer cites the exact file. If you want one document per file, import a narrower folder per document.

The document appears in the collection immediately, as a draft, and the Processing tab shows the import running. Nothing is searchable until the job finishes and you publish, exactly as with an upload.

Your plan's upload limits apply: the same size cap per file, and a ceiling on how many files one import reads. Hitting the ceiling is reported as a warning on the job rather than as a failure, so narrow the folder and import the rest separately.

Import again to pick up changes#

Import into the same document again whenever the bucket has moved on. Every file is compared against what we already hold: one whose contents have not changed is not fetched again and not indexed again, and if nothing under the folder has changed at all, the job finishes having written nothing. It is cheap enough to repeat as often as you like.

Importing again is not an edit. The title, the labels and the published status stay as you set them.

Re-index is a different thing, and this is the mistake to avoid. Re-index rebuilds the passages from the copies we already hold rather than from your bucket, so it cannot see a file you added, edited or deleted there. There is one exception, and it is not a way in: if our own copies of a document have gone missing, a re-index reads the bucket again to rebuild them, and the job says so. Importing again is what reads your storage.

Changing or removing a source#

Editing a source (a new host, a different bucket, a rotated key) changes what the next import reads and leaves everything already indexed exactly as it is.

Deleting a source erases the registration and the stored key, and nothing else. It is refused while any document imported from that source still exists, and the refusal names one of them. That is deliberate: those documents are your indexed content, and deleting a piece of configuration must never destroy them. Delete the documents you no longer want first, then remove the source.

Nothing in your own bucket is ever changed, whatever you do here. We only ever read from it.