Connector knowledge sources
How a connector reads documents from an external system into a Knowledge Engine module on a schedule.
Enterprise module
Connectors are an enterprise module, and a knowledge source needs the Knowledge Engine service as well. If either is off for the workspace, no source runs.
A source is something a connector package can read as documents: the pages of a web site, the files under a storage prefix, the rows of a table. You bind a source to a Knowledge Engine module of the same project and choose how often to read it. New and changed documents are added to the module. What disappeared at the source is removed. Agents then search the module like any other Knowledge Engine module.
Not every connector offers a source. A connector shows the Knowledge capability when its package defines one. If the current settings offer none, the tab says This connector offers no source with its current settings.
Add a source
- Open Build → Connectors, open the connector, then open its Knowledge tab.
- Select Add source.
- Fill in the form, then select Add, or Add and sync to start the first run at once.
The connector must be enabled and must have a connected account. A source reads with the connector's account. Without one the tab says Connect an account first: a source reads with the account of the connector.
The form has four sections.
| Section | What you set |
|---|---|
| Source | Which source of the package to read, and a Name. Left empty, the source's own name is used. The form says whether the source reads Only changes are read after the first run. or Everything is read on every run. |
| Filters | Which part of the source to read. The filters come from the package: a site address, a path prefix, a table. A source without filters says This source has no filters. |
| Target and schedule | The Knowledge Engine module that receives the documents, the Schedule, and Scheduled runs are on. |
| Permissions | The confirmation described below, and When the source is removed. |
Choose the knowledge base
The module list shows the active Knowledge Engine modules of the project. If the list cannot be read you can type the module key. If the project has no active module, create one in Knowledge Engine first. The module cannot be changed after the source exists: Delete the source and add it again to use another one. The source must target a module of the same project.
Schedule
| Schedule | Interval |
|---|---|
| Manual | Only when you select Sync now. |
| 15 min | 15 minutes |
| 1 hour | 60 minutes |
| 6 hours | 360 minutes |
| Daily | 1440 minutes |
The interval is counted from the start of the last run. A package can set a minimum interval. Shorter schedules are then off, and the form says This source allows a run at most every {minutes} minutes: shorter schedules are off. A new source starts with the shortest of 1 hour, 6 hours or daily that the package allows, or Manual if none fits. Turn off Scheduled runs are on to stop scheduled runs and still sync by hand.
Who sees the documents
The Knowledge Engine has no document-level permissions. Everyone with access to the module sees every document the source adds. Before you can add a source you confirm this: I understand: everyone with access to the module sees every document this source adds. Bind a source only to a module whose readers may see all of that source.
When the source is removed
When the source is removed sets the default for deleting the source: Documents stay or Documents are deleted. You can change it later, and the delete dialog asks again.
What is pulled and how it becomes a document
Each item the package lists becomes one document in the module. Console converts it to text before the module chunks and embeds it.
| Item content | Result |
|---|---|
| Markdown or plain text | Used as it is. |
| HTML | Converted to Markdown. The whole page is kept, tables and lists included. |
A file of a text type (.md, .markdown, .txt, .csv, .json, or a text/* type) | Read as text. |
| An HTML file | Converted to Markdown. |
| Any other file | Converted to text by the Knowledge Engine's file converter. |
Console skips an item and counts it as skipped when it has no content, when its text is blank, or when a file is larger than 20 MB. An item whose content cannot be converted counts as failed.
The document name is the item's title, cut to 255 characters. If there is no title, the file name or item id is used. Every document also carries metadata: source (always connector), connectorId, connectorKey, sourceId, sourceKey, externalId, title, and revision and sourceUrl when the package gives them. The package may add its own flat values, at most 32 keys. Text values are cut to 512 characters, and lists to 32 entries of 128 characters. These keys are reserved and never overwritten: the chunk fields the Knowledge Engine writes itself and the keys above.
Sync runs
Select Sync now on a source, or wait for its schedule. A run is queued and then runs on a Console node. While a run is queued or running, the lists refresh every two seconds.
A run is either full or incremental. The package decides: an incremental source reads only what changed since the last checkpoint, and a full one reads everything each time. A long run is split into segments, at most 12 per run. If a run stops before it has read everything, it ends Partly done and the next run continues from the checkpoint. If the vendor asks Console to wait more than 60 seconds, the run also ends there.
For each item Console compares the document with what it already has. The counters of a run are:
| Counter | Meaning |
|---|---|
| listed | Items the source listed. |
| added | New documents. |
| updated | Documents whose content changed. |
| unchanged | Items already in the module and not changed. |
| deleted | Documents removed because the item is gone from the source. |
| skipped | Items not ingested (no content, blank, too large). |
| failed | Items that could not be ingested. |
The source list shows them as +added ~updated −deleted · unchanged.
If the first three documents of a run all fail to ingest, the run stops. The module is then not taking documents at all.
Run statuses
Select a source to see its newest runs in Runs: status, who started it (Manual, Schedule or On add), start time, duration and result.
| Run status | Meaning |
|---|---|
| Queued | Waiting for a worker. |
| Running | Reading. Use Cancel run to stop it. |
| Succeeded | Finished with no failed item. |
| Partly done | Finished or stopped early, with failed items or unread items left. The message says why. |
| Failed | Stopped by an error. The run shows the error code and message. |
| Canceled | You canceled it. |
| Held | Stopped before removing documents. See below. |
A run that stops answering is ended as failed (Interrupted, or The run did not start if it never started).
Source statuses
| Source status | Meaning |
|---|---|
| Idle | Nothing is running. |
| Queued, Running | A run is waiting or running. |
| Needs approval | The last run is held and waits for an admin. |
| Error | The last run failed. |
| Paused | A condition blocks the source. The reason is shown. |
A source is paused, and scheduled runs skip it, while one of these is true. It runs again by itself when the cause is fixed.
| Paused reason | What to do |
|---|---|
| The module was not found | The module was deleted. Delete the source and add it for another module. |
| The module is not active | Activate the module. |
| The module is re-indexing | Wait for the re-index to finish. |
| The account needs attention | Connect the account again on the connector. |
| The connector is disabled | Enable the connector. |
| The connector package is not installed | Install the package again. |
When a run is held
When the source gives a complete listing and many documents are no longer in it, Console does not remove them by itself. If more than 5 documents and more than 20% of the source's documents are gone, the run ends Held with the message Run stopped: {count} documents ({percent}%) would be removed from {kb}. A bulk move at the source can cause this. Below that, the missing documents are removed during the run.
An admin decides:
- Remove them removes the documents from the module. This cannot be undone.
- Keep them changes nothing now. The next run looks again.
Only an admin can remove them. Only the latest run of a source can be confirmed, and only while it is held.
Re-sync
Sync now starts a run immediately. A second request while a run is active answers A run is already in progress. Checkpoints belong to the filters they were taken with, so a source with changed filters does not reuse them. Change a source with Edit; the module cannot be changed.
Delete a source
Select Delete on the source. The dialog is Delete source {name}? The source stops syncing. The documents it added stay in the module unless you choose Also delete the {count} documents from {kb}. Deleting a connector deletes its sources too, each as its When the source is removed setting says.