# Sources (https://docs.neuralisapp.com/brain-core/sources)

A **source** is a mounted data location of one [connector kind](https://docs.neuralisapp.com/brain-core/connectors.md):
the project's data folder, an extra host directory, the vector memory, a
machine sandbox. Its slug is its URI scheme. brain-core keeps source content
and metadata searchable **without an embedding model** or embedding API costs —
text search, field filters, brain memory and skills discovered in indexed
sources all work — and adds semantic search when you configure one. Hosting and
any chat model have their own costs.

The index is multi-tenant: records carry project, user and agent scope, and
search applies the caller's current permissions. Qdrant stores the indexed
content independently of its optional semantic vectors, so a storage failure
is still an indexing failure; disabling embedding does not make storage optional.

## Govern external data before connecting it [#govern-external-data-before-connecting-it]

Sources can hold an organisation's shared corpus or one user's private data. Before copying or syncing, establish the dataset owner and approved location, intended audience and source scope, sensitivity and retention, credential scope, copy permission, and provenance/refresh record. If copying is not authorised, do not create or index the source. Public availability does not by itself permit collecting sensitive personal data, and external text is data rather than trusted instruction.

Corpus content belongs only behind a configured source/connector with explicit policy and sync lifecycle. A writable project-data folder is not automatically a governed corpus. Agent identity and durable memory must not contain dataset rows or shadow copies; memory can retain a compact canonical source-URI pointer with owner/scope, provenance, permitted use, refresh date and governing retention/policy reference. Packages add skills, deterministic tools and bounded workflows around those source URIs instead of embedding the data in package prose.

## Declared sources [#declared-sources]

brain-core's manifest declares three sources that exist in every project
without anyone attaching them: `data` (the project data zone — agent memory,
conversations, runtime output, audit logs), `packages` (the `_packages/`
drop-zone that WASM and source packages load from) and `brain` (the vector
memory). Packages can declare source instances like these in their own
manifests — slug, connector kind, root, scope, a required one-line description,
and either **auto** (created in every project) or **discoverable** (offered for
on-demand attachment, like the privileged platform app zone, gated by
`sources.privileged`).

Every source therefore carries a contract-backed **owning package**, its
`origin_package`. The match is deliberately unspoofable: a rooted on-disk
source must match BOTH the declared slug AND the resolved root, so a user mount
that merely borrows a well-known slug (say, naming a mount `app`) is never
mistaken for the package-owned source; rootless sandbox kinds (vector memory,
machine sandboxes) match by kind alone. `origin_package` appears on the sources
listing and on each source's runtime-stack row as a `[pkg: …]` tag, so the
Sources panel groups sources by owning package and an agent can tell a native
source from one a teammate attached. A source's description follows the same
chain: an explicit per-source description wins, then the declared description,
then the connector kind's default — so a
[runtime-stack](https://docs.neuralisapp.com/package-system/runtime-stack.md) row is never blank, and a good
description directly improves how an agent picks the right source.

## Scope and naming [#scope-and-naming]

A source belongs to a **project**, a **user**, or an **agent**. Each connector
kind declares which scopes it allows and the default; project-scoped sources
are visible to every project member, while user- and agent-scoped sources
require ownership or a scoped-observation feature grant.

Source slugs must be unique because they are URI schemes. The Files UI
suggests scope-qualified keys when you attach a source: a project source named
after its folder (`repo`), a user source qualified by the user (`repo-alex`),
an agent source qualified by the agent (`repo-nova`). Never reuse one slug
across scopes — the scheme is the identity.

## Discoverable sources [#discoverable-sources]

The "add a source" picker offers a union of two suggestion families:
**declared discoverable sources** a package surfaces (carrying their
`origin_package` and prefilling the declared label and description), and
**host-mount suggestions** — directories the operator bound into the runtime
(the Neuralis tree, the projects zone, the whole host, or any named mount),
which have no owning package. Either way, the access controls that gate
*seeing* a high-blast-radius suggestion and *attaching* a source rooted in a
protected zone remain authoritative — a declaration can tighten visibility,
never loosen it.

A suggestion may also carry defaults for after the attach: a sync include
allowlist, a sync trigger, and path rules. Path rules offered this way are
ordinary editable rules — only a trusted package manifest can author a
non-removable one, and a create request carrying that marker is rejected. An
include narrows what is **indexed**, while what an agent can **read** is decided
by the source root and the path rules: a wide root with a narrow include is
still a wide root. Where a zone offers both a narrow and a wide proposal, the
narrow one is listed first.

## Enabled toggle versus URI policy [#enabled-toggle-versus-uri-policy]

Three independent signals answer three different questions:

- **URI policy** answers *which URIs may this caller touch*. Per-source path
  rules are evaluated against the caller at four enforcement layers — route,
  UI, tool (including the sandboxed shell), and the vector index; connectors
  are deliberately not one of them. A `read: false` path is hidden on **both**
  axes: its content is refused (read/diff/show), **and** its name is dropped from
  the file tree and search, so a denied path never leaks its existence. One
  authoring note: a rule on `foo/**` hides everything *inside* `foo`, but not the
  folder node `foo` itself — to hide the name too, add a rule on `foo` (or
  `**/foo`). The rule shape is on [URI policies](https://docs.neuralisapp.com/package-system/uri-policies.md);
  what brain-core itself seeds is on [brain-core URI policies](https://docs.neuralisapp.com/brain-core/uri-policies.md).
- **The enabled flag** answers *is this source live at all*: `true`, `false`, or
  `'auto'` (default `true`). A disabled source returns a structured error
  (`code: 'source_disabled'`, with `enable_source` as the suggested next step)
  from every `fs_*` tool and route, regardless of what the policy would allow.
  `'auto'` exists for machine sources only: the state follows a live probe of
  whether the sandbox is running, so a stopped machine reads as disabled
  without anyone flipping a switch. The toggle is `POST /sources/enabled` and a
  switch in the Sources panel.
- **Connector liveness** answers *can the connector reach its backing store
  right now* — display-only. A disconnected source (a missing local mount, a
  sandbox container that isn't running) dims in the file tree and the Sources
  panel, and the agent's runtime view marks it offline with a reason.
  Already-synced content stays listed and readable.

When a source is first attached, every loaded package's matching manifest
`uriPolicies` baselines are unioned into its persisted configuration. From
then on the persisted config is authoritative: owners and admins edit rules in
the Sources panel or through the policy routes, and a baseline is re-seeded
only when a source is added or replaced. Non-removable rules from a trusted
manifest — or from a source connector's own declaration — are the exception:
they are re-asserted on every boot, so one added by a newer package version
reaches existing sources and one that went missing comes back. Ordinary rules
never are; a rule you edited away stays away.

## When content is indexed [#when-content-is-indexed]

Writes that go through brain-core (`fs_write`, uploads) update the index
**inline**. Background sync exists for everything else: files changed on disk
by shells, skills, external editors, or other processes. Inline writes honour
the **same exclude/include filters** as sync, so a file an agent creates under
an excluded path is written and tracked for review but **not embedded** — one
rule decides what is searchable, however a file arrives. Excluded files stay
fully readable and editable through the connector.

Each source declares its sync trigger:

| Trigger | Behavior |
| --- | --- |
| `manual` | Sync only when explicitly triggered (UI button, route, or skill) |
| `auto` | Event-driven: syncs on the first run, when the filter configuration changes, and on a periodic full-reconcile cadence (`reconcileEvery`, default 24h) — never a full tree walk on every tick |
| `interval:<N><s\|m\|h\|d>` | Sync at most once per interval (for example `interval:15m`) |

The former `on_change` trigger was replaced by `auto`; a source configuration
that still names it is refused as invalid — set its trigger to `auto`.

`auto` sources are watched at the operating-system level: any change — an
agent write, a shell command, a git checkout, an external editor — marks the
touched files dirty, and after a short debounce a **targeted** sync reconciles
exactly those files (re-embed if changed, tombstone if deleted, move detection
for renames; a deleted folder removes everything indexed under it). Excluded
trees are never entered. On Linux one watch covers at most
`brainWatchMaxFolders` folders (default 200 000; files cost nothing, about
2.5 KB of memory per folder); macOS and Windows use one handle per source. A
larger source, or one past the computer's watch limit, is watched in part —
the Sources panel says **Live: partial** — and the rest arrives through the
regular reconcile; while the watcher walks a new source the panel says
**Live: starting**. A watcher that loses events falls back to a full
reconcile.

A background scheduler polls source configurations (every 60 s by default)
and enqueues due jobs on one shared sync queue; 30 seconds after start every
auto source runs one delta sync, so changes made while the app was stopped are
picked up. The queue is the single path for every sync-shaped operation —
scheduled runs, the Sync button, reindex, and delete-time purges. It runs a
bounded number of jobs at a time (default two), rotates fairly across projects,
deduplicates repeat requests per source, and gives every job its own
cancellation handle: triggering a sync returns a job reference immediately
(HTTP 202), and cancelling stops the run within moments. A cancelled run is not
an error; everything indexed before it stays valid. Renames are detected by
exact content hash, so a moved file keeps its index entry and chunk embeddings.
A source whose folder is unreachable, or suddenly empty while the index still
holds its files, fails with `root_unreachable` or `root_empty` and removes
nothing — **Sync anyway** confirms a folder that really is empty.

A platform-wide **auto-sync mode** (`on`, `off`, `packages-only`) controls
the scheduler; manual sync is available in every mode
([config settings](https://docs.neuralisapp.com/brain-core/config-settings.md#sync-and-watching)).

Sync outcomes also reach your [notification inbox](https://docs.neuralisapp.com/brain-core/events.md). A
failed sync and a source whose semantic embedding becomes owed notify you by
default; the next successful sync (or the embedding catching up) resolves the
warning. A watcher-driven update never resolves a failed full sync. After a
restart, resume checks the remaining work again; an old in-flight result
cannot overwrite a newer edit or revive a deleted record.

### Filters, caps and the credential shapes [#filters-caps-and-the-credential-shapes]

Filters are two lists. **Exclude** patterns (gitignore syntax) always apply on
top of the built-in defaults; an optional **include** allowlist narrows the
source further, and an exclude match still wins. Either edit forces a
reconcile on the next run, and every full sync removes the search data of
files the filters no longer cover — while records that still carry recovery
history (pending or deleted records, saved baselines, sole-copy `brain://`
content) keep their content without search vectors. Enabled/auto inactivity
never purges an index. A new project's `data` source starts with the
platform's numbers-only machine-written files (usage counters) already
excluded; an existing project's filters are never rewritten.

Default excludes keep `node_modules`, package-manager stores (`.pnpm-store`,
`.yarn/cache`), build output, lockfiles, `.env` files and key material out of
the index. A binary file is never indexed as text: a known binary extension is
skipped before the read, and any other file whose first 8 192 characters hold a
NUL byte (tunable as `brainSyncBinarySniffChars`) is skipped right after it and
counted on the source's last-sync line. Per-file caps (5 MB, 200k indexed
characters) bound the embedding cost; append-shaped files (`.jsonl`,
`.ndjson`) have their own higher limit (`brainSyncMaxAppendFileMb`, 20 MB) and
an append embeds only the new tail. Every walk stops at an effective **scan
cap** — a per-source *Max scan files* override, else 200 000 — and a count that
reached it shows as `N+`.

Some patterns are more than a default. A **runtime sync guarantee** is applied
under every source's own excludes on every check, so it also covers sources
attached before a pattern shipped, and no `include` can bring those paths back.
It holds the dotfile credential stores and browser profiles — `.ssh`, `.gnupg`,
`.aws`, `.azure`, `.kube`, `.docker`, `.netrc`, `.git-credentials`, `.npmrc`,
`.pypirc`, the login keyrings, the NSS certificate and private-key store (`.pki`
and `.local/share/pki`), the `gcloud` / `gh` / `rclone` config trees, and the
Chromium, Chrome and Firefox profiles — written as path shapes, because a home
directory synced through a local source and a sandbox `/config` synced through
a virtual desktop are the same exposure.

**What cannot be read cannot be embedded.** Indexing also consults a source's
non-removable `read` rules directly: a path such a rule denies is never newly
embedded, on every source, with no exclude entry to add and none that can be
edited away. The exclude list is a **preference** you own; the non-removable
rule is a **security** declaration owned by the package that owns the
connector. A per-**role** denial does *not* keep a file out of the index —
somebody can read it, so it stays searchable and each caller's results are
filtered for them. A non-removable rule hides content and blocks new indexing,
but alone never purges old records. Conversation summaries follow this: a
conversation's active `SUMMARY.md` is indexed, while the summary revision
archive behind it is never embedded or searchable.

After each successful run the source stores an index snapshot (indexed,
unindexed, and total file counts) that the UI and the agent's runtime stack
read cheaply, without a live directory walk.

## The embedding pipeline [#the-embedding-pipeline]

Semantic search is optional. The model, its dimension, the endpoints and the
spend cap are [config settings](https://docs.neuralisapp.com/brain-core/config-settings.md); the provider keys
are [credentials](https://docs.neuralisapp.com/brain-core/credentials.md). Three properties make it safe to
operate:

- **One active model, recorded.** The model that built the collection stays the
  **active** model; each vector carries its model identity, so two models never
  mix in one active generation. The configured model is only the **target** —
  changing it starts nothing, and the admin Vector section reports the switch
  as pending, with a sample-based cost and time estimate (a range, not a quote)
  on request.
- **A model switch is an explicit rebuild.** An owner starts it; the target
  model embeds eligible content into a **new** collection while the old one
  keeps serving, writes and deletes meanwhile go to both, and the two are
  swapped in one step when the new one is complete — the dimension may change
  freely. Retained recovery content, including `brain://` content and its
  version history, is preserved. A rebuild can be cancelled, and one
  interrupted by a restart waits, paused, until resumed. While it runs storage
  is doubled.
- **No silent provider substitution.** A missing credential raises
  `EMBEDDING_CREDENTIAL_MISSING` naming the credential id and its scope; an
  unsupported dimension raises `UNSUPPORTED_EMBEDDING_DIMENSION`. The platform
  never swaps in another provider or key behind your back. Content and metadata
  are still indexed, and semantic requests use text search with a warning.

Enabling a model later does not silently charge for the whole corpus: review
the estimate and explicitly start the initial build (a Rebuild in the admin
Vector section) or the backlog catch-up (**Embed pending…** on a source, which
embeds only missing or stale files). Saving a credential alone does not start a
paid rebuild. Each indexed file produces a parent document point plus content
chunks; the parent keeps the first 200k indexed characters and the chunks hold
the rest, where every search mode reaches it. Unchanged chunks are skipped by
content hash, which keeps re-syncs cheap.

That skip is why the per-source controls are three different things.
**Rescan changes** walks the whole source again, but an unchanged file keeps
its vectors. **Forget sync state** only clears what the last sync saw, so the
next sync compares every file — it deletes no file and no vector. **Re-embed…**
is the one that spends: it embeds every file in its scope (a source, folder or
file) again with the active model, after a sample-based estimate and an
explicit confirmation; it needs `sources.sync` and is not offered on
`brain://`.

**Media.** With a multimodal active model, sync also embeds media from the
file's own bytes: images and PDF pages by default, video and audio where a
source switches them on (`sync.media`), within `brainSyncMaxPdfPages` and
`brainSyncMaxMediaMb`. A kind is embedded only when the ACTIVE model supports
it, and the Sources panel says how many media files were not embedded and why.
Embedding endpoints embed text only.

**Clearing embeddings** removes only semantic vectors and preserves indexed
content, metadata, text search and retained review history. **A full
vector-store reset is destructive**: it removes the owned collections and their
indexed content, including `brain://` content stored there. A pending journal
or baseline outside those collections is not a full backup of the brain — back
up the installation first. A partially failed reset reports its partial
outcome and preserves the local state for recovery.

## Contribution categories at sync time [#contribution-categories-at-sync-time]

The indexer assigns every file a `category`. Files whose path matches the
shared contribution layout get a first-class contribution category — the
discovery substrate for [source packages](https://docs.neuralisapp.com/package-system/source-packages.md):

| Path pattern | Category |
| --- | --- |
| `SKILL.md` (any directory) + `commands/*.md` (legacy single-file form) | `skill` |
| `instructions/**/*.md` | `instruction` |
| `AGENTS.md`, `AGENTS.local.md`, `AGENT.md`, `CLAUDE.md`, `CLAUDE.local.md`, `GEMINI.md`, `.github/copilot-instructions.md` | `instruction` (identity form) |
| `rules/**/*.md`, `.cursor/rules/*.mdc`, `.cursorrules`, `.windsurfrules` | `rule` |
| `agents/**/*.md`, `*.agent.md` | `agent` |
| `docs/**/*.md` (READMEs and templates excluded) | `docs` |
| `package.json`, `neuralis.package.json`, known plugin manifests | `package` |
| `.mcp.json`, `hooks.json` | `connector` / `hook` (indexed only) |

These patterns also match inside well-known tool directories (`.claude`,
`.cursor`, `.codex`, `.gemini`, `.github`, and friends). The filename-identity
forms always keep their category; a markdown under `agents/`, `rules/`,
`instructions/` or `docs/` and a `*.agent.md` need frontmatter `id` or `name`,
otherwise the file is indexed as a plain `file`. A contribution category is
always derived from the path and content, never taken from a request.

## Deleting a source [#deleting-a-source]

`DELETE /sources/:slug` first cancels any running or queued sync job for the
source, then removes its configuration and connector binding. **The underlying
folder is never touched** — brain-core owns the wiring, not your content.
Vector points that belonged to the source are left in place by default; pass
`?purge=1` (the *Purge vector index entries* checkbox, off by default) to
remove them as a background job with one bounded index filter — the call
returns the job reference immediately. Leftover orphan points can be cleaned
up later with [Vector Health](https://docs.neuralisapp.com/brain-core/app-surfaces.md#vector-health) repair.
Removing a source resolves its open notifications.

<Callout type="info" title="Sources are config, files are content">
Attaching, disabling, or deleting a source never modifies the files behind
it. Source operations edit configuration and index state only — content
mutations go exclusively through the write tools and routes, where policies
and approvals apply. `brain://` is a source like any other: agents write
memory with `fs_write`, refine it with an `edit`, and retrieve it with
`fs_search`.
</Callout>
