Repository navigation
Data Sources
Deutsch: Datenquellen
Data sources let the AI answer from your own documents: a file, all files of a folder, or the data of your organization on an ERI server. The AI searches them when a question calls for it, and names the passages it used.
Data sources are a beta feature: almost ready, but expect some bugs. Your e-mails can come in as well, as mailboxes, which are a prototype of their own; see Mailboxes.
- Open "Settings" → "App Options".
- Under "Preview feature visibility", choose "Also show features in beta: these are almost ready for release; expect some bugs".
- Under "Select preview features", choose "RAG: Preview of our RAG implementation where you can refer your files or integrate enterprise data within your company".
The navigation bar then shows "Data sources", and the chat gets a button for choosing data sources.
Files and folders on your computer need an embedding provider. It turns your texts into numbers that AI Studio can search. Set one up in "Settings" → "Configure Embedding Providers"; see Providers and Models.
With an embedding provider in the cloud, your documents are sent to that provider, and embedding can cost money, also repeatedly after changes. AI Studio asks you to confirm this. An embedding provider on your own computer, for example, with ollama, keeps your documents with you.
Data sources on an ERI server need no embedding provider: the server searches its data itself.
Open "Settings" → "Configure Data Sources", or the gear icon on the "Data sources" page, and click "Add Data Source":
- "Local File": one file. Audio and video files are transcribed first, which needs a transcription provider.
- "Local Directory": all files of a folder, and with "Include subfolders?" also those of its subfolders.
- "External Data (ERI-Server v1)": the data of your organization on an ERI server.
- "Mailbox (IMAP)": your e-mails on a mail server. It appears once you switch on the prototype for mailboxes; see Mailboxes.
- "Data Source Name": at most 40 characters.
- "Description": what the data source contains. It helps the AI choose the right data source.
- "Embedding": the embedding provider.
- "Required provider confidence level": only providers at this level or higher may see the content.
- "Where a chat may send data after reading this data source": see Where a chat may send data. The default keeps your content out of web searches.
For a folder, "Files to include" decides which files AI Studio reads, by default all documents and text files. Separate patterns with semicolons, for example, *.pdf;*.docx. The expert settings offer more: hidden files and folders, files and folders to leave out, such as node_modules/, a size limit of 1,024 MB by default, and how AI Studio cuts the documents into pieces.
Enter the "ERI v1 Server Hostname" and the "Port", then click "Test connection & read available metadata". After a successful test, choose the "Authentication Method" that the server offers, enter your credentials, and choose a retrieval process. Under "Your security policy", decide whether the content may go to any provider or only to self-hosted ones.
Your IT department gives you the address and the way to sign in. To build an ERI server yourself, see ERI Server.
AI Studio prepares files and folders in the background. The "Data sources" page shows the progress of each data source: how many files are indexed, which are waiting, and which were skipped and why, for example, because a PDF has no readable text, a file is too large, or a recording needs a transcription provider. Images aren't indexed.
AI Studio notices when files change, and updates the index by itself. You can switch this off with "Automatic local data source refresh" and update a data source with "Refresh" instead. A new folder can be searched once its first indexing is finished.
- In the chat, click "Select the data you want to use here." in the toolbar.
- Switch on "Are data sources enabled?".
- Choose how the AI uses them:
- "AI-based data source selection" on: the AI picks the fitting data sources itself. Off: you choose them under "Available Data Sources".
- "Semantic Search" on, the default: the AI searches your data sources whenever a question calls for it. Off: AI Studio searches them with every message before the AI answers, which is called classic RAG.
Semantic Search needs a model that can use tools. Otherwise, AI Studio uses classic RAG, and the selection says why.
Data sources that the selected provider may not see are greyed out or missing. To start every new chat with certain data sources, set them in the "Chat Options" under "Data Source Selection".
The passages the AI received are listed below its answer, as "Sources provided by the data providers". The eye button shows exactly what the AI received from a source. For documents with pages, the page numbers open the document at that page.
Two settings of a data source decide how much text the AI receives from it:
- "How many matches do you want at most per query?": 10 by default, and at least 10.
- "Token limit": how long a single match may be. By default, a match may be as long as the embedding provider takes at once. The field names this limit, often 8,192 tokens.
For files and folders, both are under "Show Expert Settings". On an ERI server, the server decides how long a match is.
Files and folders return at most this many matches. Requires AI Studio v26.11.1 or newer. Older versions could return up to twice as many, because the search by meaning and the search by words each brought their own matches.
With classic RAG, AI Studio sends the matches of all chosen data sources along with your message. With matches of 8,192 tokens, the default of 10 matches can already hold about 80,000 tokens per data source. That's more than many models take, especially models you run yourself. AI Studio therefore sends only as many matches as fit into half of the model's context window; the rest stays for the chat and the answer. When AI Studio doesn't know the context window, it sends at most 100,000 characters. For a model you run yourself, you can state its window under "Override Model Limits"; see Providers and Models. The data sources take turns: first the best match of each, then the second best, and so on. AI Studio shortens a match longer than the whole limit. Matches that don't fit are left out, and AI Studio tells you so with each answer. Requires AI Studio v26.11.1 or newer. Older versions sent all matches, and the provider turned the message down when they were too many. A long chat with large attachments can still exceed what the model takes; see Troubleshooting.
"AI-based data validation" keeps only the matches that fit your question best. It asks the AI once for each match, though, so many matches mean many requests. It appears in the chat once you switch on "Enable the retrieval context validation agent?" under "Agent: Retrieval Context Validation Options" in the settings.
Semantic Search gets the matches page by page. A single search returns at most 100,000 characters; see Tools in the Chat. Matches that don't fit start the next page, which the AI can ask for, so it reaches every match. AI Studio shortens a match longer than a whole page and tells the AI so. When the AI searches several data sources at once, they share one page; the AI gets the rest by searching a data source alone. Requires AI Studio v26.11.1 or newer. In older versions, matches that didn't fit were left out, and no later page brought them back.
What helps:
- Keep the default of 10 matches.
- Lower the "Token limit". More matches then fit into a message and into a search. AI Studio indexes the data source again afterward, which costs money with an embedding provider in the cloud.
- With classic RAG, switch on "Semantic Search", choose fewer data sources, or choose a model with a larger context window. More matches then reach the AI.
- Tools in the Chat: Semantic Search and the other tools.
- Provider Confidence: keep confidential documents away from some providers.
Describes MindWork AI Studio v26.11.1 · Found a mistake? Open an issue
User guide
Basics
Chat
- Using the Chat
- Workspaces and Chat Search
- Attachments
- Tools in the Chat
- Data Sources
- Mailboxes (prototype)
- Voice Input
Assistants
- Overview
- General
- Business
- Learning
- Software Engineering
- AI Studio Development
More
Enterprise IT
Deployment
Configuring
Topics
- Providers and API Keys
- Confidence and Data Security
- Assistants and Templates
- Tools and Web Access
- Labeling and Usage Statistics
- Managing Backups
Scenarios and help
Developers
Deutsch