20.5Data: datasets, datasources, endpoints and collectors

Bmator organizes external and internal information through four related data objects: datasources, endpoints, collectors, and datasets.

Together, they form an automated pipeline:

• A datasource connects to external data.
• An endpoint specifies a single call made through that connection.
• A collector triggers the endpoint and writes incoming records into a table.
• A dataset stores those records for search, reporting, and workflow tasks.

The four data objects

Datasources

Datasources establish connections to outside services, including third-party APIs, accounting platforms, or file uploads. Each datasource stores its required authentication credentials and can hold an optional synchronization schedule.

Endpoints

Endpoints represent the distinct calls available through a datasource. You can provide input parameters to test a call, view the raw return payload, and save the output directly as knowledge or into a dataset.

Collectors

Collectors execute an endpoint call using specified inputs and append the returned data directly into a dataset. Collectors run without any AI. They can be triggered manually, run on an automated schedule, or execute as a step in a workflow.

Datasets

Datasets are structured tables. You can browse their contents, fill them manually or through automated jobs, download their data, or feed them into automated tasks.

Setting up data collection

To configure an end-to-end data pipeline from an external service into a table:

1. Open Datasources and choose New datasource to enter your provider details and credentials. Select Test connection to verify connectivity, and use Run sync if your datasource uses scheduled syncing.
2. Open Endpoints and select New endpoint to define an API request on that datasource. Select Run to execute the call with your parameters and confirm the response.
3. Open Datasets and select New dataset to create a table to receive incoming records, or import existing records from a CSV, TSV, JSON, or Excel file.
4. Open Collectors and select New collector.
5. Select your endpoint, specify the inputs to pass, and assign the destination table under Target dataset.
6. Select Run now to fetch data immediately, or rely on a schedule or workflow step to execute the collector automatically.

Subscriptions and costs

You can subscribe to public datasets and datasources shared from other workspaces. Subscribed objects function just like local workspace items when referenced in Ask Bmator, workspace search, and automated tasks.

Calls to built-in datasources provided directly by Bmator are metered. The credit or usage cost is calculated and shown before you run the call. Calls made through keyless APIs or connections configured with your own API credentials carry no extra Bmator metering cost.