Object storage connectors

How to connect Amazon S3, MinIO (and other S3-compatible services) and Azure Blob Storage so agents can list and read objects and, if you allow it, create new text objects.

Enterprise module

Connectors are an enterprise module. The Amazon S3, MinIO and Azure Blob Storage packages are installed from Build → Connectors (see Install packages).

The three packages work the same way. An agent can list buckets (or containers) and objects, read the text of an object and read its details. With Read only switched off it can also create a new text object. Console never deletes objects: there is no delete tool, and the connector can send only GET, HEAD and PUT requests. A bucket can also be synced into a Knowledge Engine as a knowledge source.

Console signs every request itself with the stored credential. The connector package never holds a secret and never signs. Console also checks each request against the buckets or containers you listed.

Choose a package

Package	Use it for
Amazon S3	Buckets on AWS. One connector serves one Region.
MinIO	MinIO, and other S3-compatible services (Cloudflare R2, Wasabi, DigitalOcean Spaces, Backblaze B2, Ceph) with a typed endpoint. Also AWS endpoints the Amazon S3 package does not cover (FIPS, dual-stack, private link).
Azure Blob Storage	One Azure storage account, in the public cloud, Azure Government or Azure operated by 21Vianet.

All three use path-style addresses only (<endpoint>/<bucket>/<key>). There is no setting for it.

Settings

Setting	Amazon S3	MinIO	Azure Blob Storage
Where	Region: one of the 36 AWS Regions with an S3 endpoint (default us-east-1). Console reaches s3.<region>.amazonaws.com only.	Endpoint (SaaS: https, public address) or Endpoint (on-prem) (https or http, private addresses allowed). No path. Region: the signing region the server expects (default us-east-1; Cloudflare R2 uses auto).	Azure cloud (public, Azure Government, 21Vianet), Storage account (3 to 24 lowercase letters and digits), Blob endpoint (SaaS) or Blob endpoint (on-prem).
Limit to	Buckets, at most 20	Buckets, at most 20	Containers, at most 20
Own CA	not available	CA certificate (PEM), on-prem only	CA certificate (PEM), on-prem only
Other			Tenant ID (Microsoft Entra ID sign-in only)

Common settings for all three:

Setting	Meaning
Buckets / Containers	Limits the connector to these names. Console checks every request against the list. Required for a public bucket or container. Empty means any bucket the credential can reach.
Read only	On by default. The write tool exists only while it is off.
Write prefix	Visible when Read only is off. Writes go only to names with this prefix. It is an extra check, not a boundary. Limit the credential to the same prefix.
Allow overwrite	Visible when Read only is off. Off by default: the write tool never replaces an existing object.
Write limit	Advanced. 1 to 1024 KiB, default 256. The largest size of one write.
Read limit	Advanced. 1 to 128 KiB, default 32. The text one read returns.

Keep both limits

The Buckets or Containers list limits what Console will request. The credential's own policy or role limits what the credential could request. Keep both. The policy is the real boundary.

Sign-in

Package	Sign-in methods
Amazon S3	Access key (Access key ID and Secret access key); Temporary credentials (the same plus a Session token); Public bucket (no credentials)
MinIO	Access key; Public bucket (no credentials)
Azure Blob Storage	Microsoft Entra ID (service principal) (Application (client) ID and Client secret, plus the Tenant ID setting); SAS token; Storage account key; Public container (no credentials)

• Temporary credentials (S3) expire and Console cannot renew them. They suit a trial, not a scheduled knowledge source. When they end, Console reports "The temporary credentials have expired." and you paste a fresh set.
• Public bucket and Public container send no credential, so they can only do what the owner allows everyone to do (normally list and read, never write). The method appears only after you enter at least one name under Buckets or Containers.
• A SAS token cannot be renewed by Console. When it expires the connection fails until you paste a new token. A user delegation SAS is valid for 7 days at most.
• Storage account key gives full access to the account, whatever Read only, Containers or Write prefix say. Prefer Microsoft Entra ID with a role on one container, or a SAS on one container.
• For the Entra method, Console also reaches the Microsoft sign-in address of the chosen cloud over https (login.microsoftonline.com, login.microsoftonline.us or login.partner.microsoftonline.cn).

Click Test connection after you connect. A "denied" answer to a listing still passes, and the account name says what is not allowed (for example "cannot list: enter bucket names"). A wrong key, an expired token, a wrong Region or a missing bucket stops the connection with a message.

Tools

Tool	Access	What it does
List buckets / List containers	read	Lists the names you entered under Buckets / Containers, otherwise the account's own (S3 up to 100 per call, Azure up to 100 per call).
List objects / List blobs	read	Lists objects and "folders" under a prefix. 50 by default, 200 at most per call. A cursor continues the list.
Get object info / Get blob info	read	Size, content type, last change, ETag, storage class or access tier, and up to 20 user metadata entries.
Read object / Read blob	read	Reads text in parts (Read limit, 32 KiB by default, 128 KiB at most). A nextOffset continues. PDF and Word are for the knowledge source, not this tool. Archived and compressed objects are refused with a hint.
Create text object / Create text blob	write	Creates a text object. Exists only while Read only is off. Fails with "exists" if the name is taken, unless Allow overwrite is on and the agent asks for it.

Write tools ask for approval by default. See Tools and approvals and which tools ask.

Written content is text only, stored as text/plain, text/markdown, text/csv or application/json. Names that end in .html, .htm, .svg, .js, .mjs, .css, .xml or .xhtml are stored as text/plain, so a public bucket does not serve them as active content. Written objects carry a metadata entry that says cognipeer-console wrote them.

Treat object content as untrusted

Names, metadata and text come from the bucket and may contain instructions. They reach the model as data.

Size limits

Limit	Value
Text returned by one read	32 KiB default, 128 KiB at most (Read limit)
Text written by one write	256 KiB default, 1 MiB at most (Write limit)
Objects listed per call	50 default, 200 at most
User metadata entries shown	20
Object name	1,024 bytes (Azure: 1,024 characters)
Buckets or containers you can enter	20

Knowledge source limits are in the next section.

Sync a bucket into a knowledge base

Choose the Bucket (or Container), optional Prefixes, the File types, Maximum file size and Recursive.

Option	Value
Prefixes	At most 10. Each is at most 1024 bytes, with no leading /. Each is listed on its own.
File types	Default: md, markdown, txt, csv, json, html, htm, pdf, docx.
Maximum file size	10 MB by default, 20 MB at most. A larger file is skipped, and raising the limit picks it up on the next run.
Recursive	On by default. Off lists one level only.
Schedule	Default every 6 hours. Cannot be shorter than 60 minutes.
Run size	At most 100,000 matching objects. Beyond that the run stops with a message and removes nothing.

• Each run lists the whole selection again. Files whose ETag and size did not change are not downloaded again.
• An object that disappears, or stops matching, is removed from the knowledge base on the next complete run. A large removal is held back for an administrator to confirm. This never touches the objects in the bucket.
• Zero-byte "folder" markers and archived objects are ignored.
• A file the run skipped, for example because the credential could not read it, is tried again only when the object changes. After you fix a permission, change or upload the object again, or create the source again.
• Requests and data transfer are billed by the storage vendor, not by Console. On a large bucket use prefixes and file types.

See Knowledge sources for runs and schedules.

Network reachability

	Cognipeer's cloud (SaaS)	On-prem Console
Amazon S3	s3.<region>.amazonaws.com on port 443, public addresses only	the same
MinIO endpoint	https only, public address	https or http, private addresses allowed
Azure endpoint	https, one of the three Azure blob endings	https or http, any host, private addresses allowed
Your own CA	not available	CA certificate (PEM)

• On SaaS, allow the Console egress address in the firewall of your storage (your Cognipeer contact gives you the address).
• Loopback addresses (localhost, 127.0.0.1) are never reachable. Use the machine's LAN address.
• The operator of an on-prem Console can narrow the reachable private ranges with CONNECTOR_EGRESS_PRIVATE_CIDRS, a comma-separated list such as 10.20.0.0/16,192.168.10.0/24. If the check fails with "could not be reached" while the server is up, ask the operator whether the server's range is allowed.
• With http on-prem, object names, content and the access key ID travel unencrypted. The secret key is not sent. Use https unless the network is fully trusted.
• Requests are signed with the time of the Console server. If the clock is far off, the storage refuses the requests and the message names the clock.
• For Azure, the endpoint is https://<storage-account>.blob.core.windows.net (Azure Government: .blob.core.usgovcloudapi.net; 21Vianet: .blob.core.chinacloudapi.cn). System containers whose names start with $ are not supported.
• A MinIO server must accept path-style requests. A server behind a reverse proxy path cannot be used.

Least-privilege setup

Amazon S3 and MinIO

Create a user only for Console and attach the smallest policy. Never use the root user. Do not grant s3:DeleteObject.

Console needs it to	Permission	Resource
list objects and run the knowledge source	s3:ListBucket	the bucket
read object info and text	s3:GetObject	the objects of the bucket
write new text objects (only when Read only is off)	s3:PutObject	the objects of the bucket
list all buckets of the account (optional)	s3:ListAllMyBuckets	all (*)

Read only, one bucket (replace amzn-s3-demo-bucket):

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket" },
    { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*" }
  ]
}

To also create new objects, add s3:PutObject on arn:aws:s3:::amzn-s3-demo-bucket/*.

• Without s3:ListAllMyBuckets, enter the bucket names under Buckets. The agent then sees that list and the connection check still passes.
• To limit a user to one prefix, limit the object permission to .../docs/* and the listing with a StringLike condition on s3:prefix. Then set Prefixes of the knowledge source to that prefix. A listing of the whole bucket is refused.
• If your objects use SSE-KMS, the user may also need permission on that key.
• Amazon S3 supports 36 listed Regions only. S3 on Outposts, access points, directory buckets, requester-pays buckets, SSE-C and the FIPS, dual-stack and China endpoints are not supported by the Amazon S3 package. Use the MinIO package with a typed endpoint for FIPS, dual-stack or private link.
• To rotate an access key, create a second key, put it into the connector settings, test the connection, then delete the old one.

For MinIO, create the policy and user with mc (replace ALIAS and mybucket), then give Console the user name as the Access key ID:

mc admin policy create ALIAS cognipeer-readonly readonly-mybucket.json
mc admin user add ALIAS cognipeer-reader <a-generated-secret>
mc admin policy attach ALIAS cognipeer-readonly --user cognipeer-reader

Older mc versions name some commands differently. Run mc admin policy --help to see what yours offers, and trust the connection check. If signing fails, check Region first: a signing region the server does not expect is the usual cause. For other S3-compatible services, create a key that can only read one bucket and use the region value your provider documents.

A MinIO server that does not accept the If-None-Match: * header makes Console check first and then write, which can race if two writers create the same key at once. A credential that cannot read objects cannot make that check; nothing is written and the tool says so.

Azure Blob Storage

Method	Least privilege
Microsoft Entra ID (service principal)	Storage Blob Data Reader to list and read. Storage Blob Data Contributor only to write. Assign on the storage account, or on one container (safest; then fill in Containers, because a role on one container cannot list the account's containers). A role assignment can take up to 10 minutes to take effect.
SAS token	On one container: l (List) and r (Read). Add c and w only to write. Never include d.
Storage account key	No smaller key exists. Use it only when nothing else is possible.

For Entra, register an application, create a client secret with an expiry (at most 24 months; Microsoft recommends less than 12), and assign the role. In Console enter the Tenant ID, Application (client) ID and Client secret.

Role	Role ID
Storage Blob Data Reader	2a2b9908-6ea1-4ae2-8e65-a410df84e7d1
Storage Blob Data Contributor	ba92f5b4-2d11-453d-a403-e96b0029c9fe

The Contributor role can delete blobs as well. Console has no delete tool, but the credential could still delete, so assign Contributor only when writing is needed.

Which SAS to use:

Kind	Signed with	Note
User delegation SAS	Microsoft Entra credentials	Valid for 7 days at most. Not affected by key rotation.
Service SAS on one container	the account key	Can follow a stored access policy, so you can revoke it without changing the key. Stops working when the key is regenerated.
Account SAS	the account key	Only to let agents list all containers of the account.

Turning off Shared Key access on the account makes the Storage account key method stop working. The in-product Setup guide on the connector has the portal steps for each method.