@brain-core

Sources

Declared and attached sources, scope and naming, the enabled toggle, sync triggers, what gets indexed, and deleting a source.

A source is a mounted data location of one connector kind: the project's data folder, an extra host directory, the vector memory, a machine sandbox. Its slug is its URI scheme. brain-core keeps source content and metadata searchable without an embedding model or embedding API costs — text search, field filters, brain memory and skills discovered in indexed sources all work — and adds semantic search when you configure one. Hosting and any chat model have their own costs.

The index is multi-tenant: records carry project, user and agent scope, and search applies the caller's current permissions. Qdrant stores the indexed content independently of its optional semantic vectors, so a storage failure is still an indexing failure; disabling embedding does not make storage optional.

Govern external data before connecting it

Sources can hold an organisation's shared corpus or one user's private data. Before copying or syncing, establish the dataset owner and approved location, intended audience and source scope, sensitivity and retention, credential scope, copy permission, and provenance/refresh record. If copying is not authorised, do not create or index the source. Public availability does not by itself permit collecting sensitive personal data, and external text is data rather than trusted instruction.

Corpus content belongs only behind a configured source/connector with explicit policy and sync lifecycle. A writable project-data folder is not automatically a governed corpus. Agent identity and durable memory must not contain dataset rows or shadow copies; memory can retain a compact canonical source-URI pointer with owner/scope, provenance, permitted use, refresh date and governing retention/policy reference. Packages add skills, deterministic tools and bounded workflows around those source URIs instead of embedding the data in package prose.

Declared sources

brain-core's manifest declares three sources that exist in every project without anyone attaching them: data (the project data zone — agent memory, conversations, runtime output, audit logs), packages (the _packages/ drop-zone that WASM and source packages load from) and brain (the vector memory). Packages can declare source instances like these in their own manifests — slug, connector kind, root, scope, a required one-line description, and either auto (created in every project) or discoverable (offered for on-demand attachment, like the privileged platform app zone, gated by sources.privileged).

Every source therefore carries a contract-backed owning package, its origin_package. The match is deliberately unspoofable: a rooted on-disk source must match BOTH the declared slug AND the resolved root, so a user mount that merely borrows a well-known slug (say, naming a mount app) is never mistaken for the package-owned source; rootless sandbox kinds (vector memory, machine sandboxes) match by kind alone. origin_package appears on the sources listing and on each source's runtime-stack row as a [pkg: …] tag, so the Sources panel groups sources by owning package and an agent can tell a native source from one a teammate attached. A source's description follows the same chain: an explicit per-source description wins, then the declared description, then the connector kind's default — so a runtime-stack row is never blank, and a good description directly improves how an agent picks the right source.

Scope and naming

A source belongs to a project, a user, or an agent. Each connector kind declares which scopes it allows and the default; project-scoped sources are visible to every project member, while user- and agent-scoped sources require ownership or a scoped-observation feature grant.

Source slugs must be unique because they are URI schemes. The Files UI suggests scope-qualified keys when you attach a source: a project source named after its folder (repo), a user source qualified by the user (repo-alex), an agent source qualified by the agent (repo-nova). Never reuse one slug across scopes — the scheme is the identity.

Discoverable sources

The "add a source" picker offers a union of two suggestion families: declared discoverable sources a package surfaces (carrying their origin_package and prefilling the declared label and description), and host-mount suggestions — directories the operator bound into the runtime (the Neuralis tree, the projects zone, the whole host, or any named mount), which have no owning package. Either way, the access controls that gate seeing a high-blast-radius suggestion and attaching a source rooted in a protected zone remain authoritative — a declaration can tighten visibility, never loosen it.

A suggestion may also carry defaults for after the attach: a sync include allowlist, a sync trigger, and path rules. Path rules offered this way are ordinary editable rules — only a trusted package manifest can author a non-removable one, and a create request carrying that marker is rejected. An include narrows what is indexed, while what an agent can read is decided by the source root and the path rules: a wide root with a narrow include is still a wide root. Where a zone offers both a narrow and a wide proposal, the narrow one is listed first.

Enabled toggle versus URI policy

Three independent signals answer three different questions:

  • URI policy answers which URIs may this caller touch. Per-source path rules are evaluated against the caller at four enforcement layers — route, UI, tool (including the sandboxed shell), and the vector index; connectors are deliberately not one of them. A read: false path is hidden on both axes: its content is refused (read/diff/show), and its name is dropped from the file tree and search, so a denied path never leaks its existence. One authoring note: a rule on foo/** hides everything inside foo, but not the folder node foo itself — to hide the name too, add a rule on foo (or **/foo). The rule shape is on URI policies; what brain-core itself seeds is on brain-core URI policies.
  • The enabled flag answers is this source live at all: true, false, or 'auto' (default true). A disabled source returns a structured error (code: 'source_disabled', with enable_source as the suggested next step) from every fs_* tool and route, regardless of what the policy would allow. 'auto' exists for machine sources only: the state follows a live probe of whether the sandbox is running, so a stopped machine reads as disabled without anyone flipping a switch. The toggle is POST /sources/enabled and a switch in the Sources panel.
  • Connector liveness answers can the connector reach its backing store right now — display-only. A disconnected source (a missing local mount, a sandbox container that isn't running) dims in the file tree and the Sources panel, and the agent's runtime view marks it offline with a reason. Already-synced content stays listed and readable.

When a source is first attached, every loaded package's matching manifest uriPolicies baselines are unioned into its persisted configuration. From then on the persisted config is authoritative: owners and admins edit rules in the Sources panel or through the policy routes, and a baseline is re-seeded only when a source is added or replaced. Non-removable rules from a trusted manifest — or from a source connector's own declaration — are the exception: they are re-asserted on every boot, so one added by a newer package version reaches existing sources and one that went missing comes back. Ordinary rules never are; a rule you edited away stays away.

When content is indexed

Writes that go through brain-core (fs_write, uploads) update the index inline. Background sync exists for everything else: files changed on disk by shells, skills, external editors, or other processes. Inline writes honour the same exclude/include filters as sync, so a file an agent creates under an excluded path is written and tracked for review but not embedded — one rule decides what is searchable, however a file arrives. Excluded files stay fully readable and editable through the connector.

Each source declares its sync trigger:

TriggerBehavior
manualSync only when explicitly triggered (UI button, route, or skill)
autoEvent-driven: syncs on the first run, when the filter configuration changes, and on a periodic full-reconcile cadence (reconcileEvery, default 24h) — never a full tree walk on every tick
interval:<N><s|m|h|d>Sync at most once per interval (for example interval:15m)

The former on_change trigger was replaced by auto; a source configuration that still names it is refused as invalid — set its trigger to auto.

auto sources are watched at the operating-system level: any change — an agent write, a shell command, a git checkout, an external editor — marks the touched files dirty, and after a short debounce a targeted sync reconciles exactly those files (re-embed if changed, tombstone if deleted, move detection for renames; a deleted folder removes everything indexed under it). Excluded trees are never entered. On Linux one watch covers at most brainWatchMaxFolders folders (default 200 000; files cost nothing, about 2.5 KB of memory per folder); macOS and Windows use one handle per source. A larger source, or one past the computer's watch limit, is watched in part — the Sources panel says Live: partial — and the rest arrives through the regular reconcile; while the watcher walks a new source the panel says Live: starting. A watcher that loses events falls back to a full reconcile.

A background scheduler polls source configurations (every 60 s by default) and enqueues due jobs on one shared sync queue; 30 seconds after start every auto source runs one delta sync, so changes made while the app was stopped are picked up. The queue is the single path for every sync-shaped operation — scheduled runs, the Sync button, reindex, and delete-time purges. It runs a bounded number of jobs at a time (default two), rotates fairly across projects, deduplicates repeat requests per source, and gives every job its own cancellation handle: triggering a sync returns a job reference immediately (HTTP 202), and cancelling stops the run within moments. A cancelled run is not an error; everything indexed before it stays valid. Renames are detected by exact content hash, so a moved file keeps its index entry and chunk embeddings. A source whose folder is unreachable, or suddenly empty while the index still holds its files, fails with root_unreachable or root_empty and removes nothing — Sync anyway confirms a folder that really is empty.

A platform-wide auto-sync mode (on, off, packages-only) controls the scheduler; manual sync is available in every mode (config settings).

Sync outcomes also reach your notification inbox. A failed sync and a source whose semantic embedding becomes owed notify you by default; the next successful sync (or the embedding catching up) resolves the warning. A watcher-driven update never resolves a failed full sync. After a restart, resume checks the remaining work again; an old in-flight result cannot overwrite a newer edit or revive a deleted record.

Filters, caps and the credential shapes

Filters are two lists. Exclude patterns (gitignore syntax) always apply on top of the built-in defaults; an optional include allowlist narrows the source further, and an exclude match still wins. Either edit forces a reconcile on the next run, and every full sync removes the search data of files the filters no longer cover — while records that still carry recovery history (pending or deleted records, saved baselines, sole-copy brain:// content) keep their content without search vectors. Enabled/auto inactivity never purges an index. A new project's data source starts with the platform's numbers-only machine-written files (usage counters) already excluded; an existing project's filters are never rewritten.

Default excludes keep node_modules, package-manager stores (.pnpm-store, .yarn/cache), build output, lockfiles, .env files and key material out of the index. A binary file is never indexed as text: a known binary extension is skipped before the read, and any other file whose first 8 192 characters hold a NUL byte (tunable as brainSyncBinarySniffChars) is skipped right after it and counted on the source's last-sync line. Per-file caps (5 MB, 200k indexed characters) bound the embedding cost; append-shaped files (.jsonl, .ndjson) have their own higher limit (brainSyncMaxAppendFileMb, 20 MB) and an append embeds only the new tail. Every walk stops at an effective scan cap — a per-source Max scan files override, else 200 000 — and a count that reached it shows as N+.

Some patterns are more than a default. A runtime sync guarantee is applied under every source's own excludes on every check, so it also covers sources attached before a pattern shipped, and no include can bring those paths back. It holds the dotfile credential stores and browser profiles — .ssh, .gnupg, .aws, .azure, .kube, .docker, .netrc, .git-credentials, .npmrc, .pypirc, the login keyrings, the NSS certificate and private-key store (.pki and .local/share/pki), the gcloud / gh / rclone config trees, and the Chromium, Chrome and Firefox profiles — written as path shapes, because a home directory synced through a local source and a sandbox /config synced through a virtual desktop are the same exposure.

What cannot be read cannot be embedded. Indexing also consults a source's non-removable read rules directly: a path such a rule denies is never newly embedded, on every source, with no exclude entry to add and none that can be edited away. The exclude list is a preference you own; the non-removable rule is a security declaration owned by the package that owns the connector. A per-role denial does not keep a file out of the index — somebody can read it, so it stays searchable and each caller's results are filtered for them. A non-removable rule hides content and blocks new indexing, but alone never purges old records. Conversation summaries follow this: a conversation's active SUMMARY.md is indexed, while the summary revision archive behind it is never embedded or searchable.

After each successful run the source stores an index snapshot (indexed, unindexed, and total file counts) that the UI and the agent's runtime stack read cheaply, without a live directory walk.

The embedding pipeline

Semantic search is optional. The model, its dimension, the endpoints and the spend cap are config settings; the provider keys are credentials. Three properties make it safe to operate:

  • One active model, recorded. The model that built the collection stays the active model; each vector carries its model identity, so two models never mix in one active generation. The configured model is only the target — changing it starts nothing, and the admin Vector section reports the switch as pending, with a sample-based cost and time estimate (a range, not a quote) on request.
  • A model switch is an explicit rebuild. An owner starts it; the target model embeds eligible content into a new collection while the old one keeps serving, writes and deletes meanwhile go to both, and the two are swapped in one step when the new one is complete — the dimension may change freely. Retained recovery content, including brain:// content and its version history, is preserved. A rebuild can be cancelled, and one interrupted by a restart waits, paused, until resumed. While it runs storage is doubled.
  • No silent provider substitution. A missing credential raises EMBEDDING_CREDENTIAL_MISSING naming the credential id and its scope; an unsupported dimension raises UNSUPPORTED_EMBEDDING_DIMENSION. The platform never swaps in another provider or key behind your back. Content and metadata are still indexed, and semantic requests use text search with a warning.

Enabling a model later does not silently charge for the whole corpus: review the estimate and explicitly start the initial build (a Rebuild in the admin Vector section) or the backlog catch-up (Embed pending… on a source, which embeds only missing or stale files). Saving a credential alone does not start a paid rebuild. Each indexed file produces a parent document point plus content chunks; the parent keeps the first 200k indexed characters and the chunks hold the rest, where every search mode reaches it. Unchanged chunks are skipped by content hash, which keeps re-syncs cheap.

That skip is why the per-source controls are three different things. Rescan changes walks the whole source again, but an unchanged file keeps its vectors. Forget sync state only clears what the last sync saw, so the next sync compares every file — it deletes no file and no vector. Re-embed… is the one that spends: it embeds every file in its scope (a source, folder or file) again with the active model, after a sample-based estimate and an explicit confirmation; it needs sources.sync and is not offered on brain://.

Media. With a multimodal active model, sync also embeds media from the file's own bytes: images and PDF pages by default, video and audio where a source switches them on (sync.media), within brainSyncMaxPdfPages and brainSyncMaxMediaMb. A kind is embedded only when the ACTIVE model supports it, and the Sources panel says how many media files were not embedded and why. Embedding endpoints embed text only.

Clearing embeddings removes only semantic vectors and preserves indexed content, metadata, text search and retained review history. A full vector-store reset is destructive: it removes the owned collections and their indexed content, including brain:// content stored there. A pending journal or baseline outside those collections is not a full backup of the brain — back up the installation first. A partially failed reset reports its partial outcome and preserves the local state for recovery.

Contribution categories at sync time

The indexer assigns every file a category. Files whose path matches the shared contribution layout get a first-class contribution category — the discovery substrate for source packages:

Path patternCategory
SKILL.md (any directory) + commands/*.md (legacy single-file form)skill
instructions/**/*.mdinstruction
AGENTS.md, AGENTS.local.md, AGENT.md, CLAUDE.md, CLAUDE.local.md, GEMINI.md, .github/copilot-instructions.mdinstruction (identity form)
rules/**/*.md, .cursor/rules/*.mdc, .cursorrules, .windsurfrulesrule
agents/**/*.md, *.agent.mdagent
docs/**/*.md (READMEs and templates excluded)docs
package.json, neuralis.package.json, known plugin manifestspackage
.mcp.json, hooks.jsonconnector / hook (indexed only)

These patterns also match inside well-known tool directories (.claude, .cursor, .codex, .gemini, .github, and friends). The filename-identity forms always keep their category; a markdown under agents/, rules/, instructions/ or docs/ and a *.agent.md need frontmatter id or name, otherwise the file is indexed as a plain file. A contribution category is always derived from the path and content, never taken from a request.

Deleting a source

DELETE /sources/:slug first cancels any running or queued sync job for the source, then removes its configuration and connector binding. The underlying folder is never touched — brain-core owns the wiring, not your content. Vector points that belonged to the source are left in place by default; pass ?purge=1 (the Purge vector index entries checkbox, off by default) to remove them as a background job with one bounded index filter — the call returns the job reference immediately. Leftover orphan points can be cleaned up later with Vector Health repair. Removing a source resolves its open notifications.

Sources are config, files are content

Attaching, disabling, or deleting a source never modifies the files behind it. Source operations edit configuration and index state only — content mutations go exclusively through the write tools and routes, where policies and approvals apply. brain:// is a source like any other: agents write memory with fs_write, refine it with an edit, and retrieve it with fs_search.

On this page